Subscribe
Accepted Solution

Snapshot timestamp in cm_storage_smsv scheme

I'm working on some customer workflows and leveraging the "Manage snapmirror-snapvault cascade relationships" Pack version 1.0.0 as certified by NetApp. What I'm seeing is an interesting problem with snapshot reservations.  I'm using this pack as it provided a framework for retreiving snapshot information into WFA. (I'm not sure why this functionality is missing from the core product.) I've created a "create snapshot" command, and borrowing heavily from other examples, the reservation insert SQL is:

 

INSERT
INTO
    cm_storage_smsv.snapshot
    (volume, name, cluster, vserver, timestamp)
VALUES
    ('${Volume}', '${Snapshot}', '${Cluster}', '${Vserver}', CURRENT_TIMESTAMP);

I have created a filter that allows me to get snapshots by volume including those created by reservation:

 

SELECT
    snapshot.name as 'name',
    snapshot.volume as 'volume',
    snapshot.vserver as 'vserver',
    snapshot.cluster as 'cluster',
    snapshot.timestamp as 'timestamp',
    unix_timestamp(snapshot.timestamp) as 'unix_timestamp'
FROM
    cm_storage.cluster,
    cm_storage_smsv.snapshot
WHERE
    (
        cluster.name = '${cluster}'
        OR cluster.primary_address = '${cluster}'
    )
    AND snapshot.volume = '${volume}'
    AND snapshot.vserver = '${vserver}'
    AND (
        cluster.primary_address = snapshot.cluster
        OR cluster.name = snapshot.cluster
    )
ORDER BY
    snapshot.timestamp DESC

This works appropriately for all snapshots that are on the volume and picked up by the snapshot data source. However, any new snapshots recorded by reservations do not show the correct timestamp. They report the timestamp of now() and not the actual timestamp of the snapshot. This is causing problems in my testing. I'm unable to see on the WFA database side exactly how the snapshot reservation data is stored, but the filter and reservation SQL works properly for data in the cm_storage_smsv.snapshot table.

 

Has anyone else seen this issue?  Is this a problem with the management pack itself, or something screwy with my SQL code?  For grins I replaced CURRENT_TIMESTAMP in the snapshot command reservation SQL with a static date, and there was no effect.  It is as if the reservation table definition forces the timestamp to now(). 

 

 

 

Re: Snapshot timestamp in cm_storage_smsv scheme

@bjones_ea

 

Can you post your workflow? Desingner issues are difficult to debug without the workflow.. 

 

sinhaa

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

Re: Snapshot timestamp in cm_storage_smsv scheme

Sure thing.  I've attached the workflow DAR file.  This workflow and its components will allow me to create regular snapshots, snapshots with snapmirror labels (for SnapVault), and consistency group snapshots.

Re: Snapshot timestamp in cm_storage_smsv scheme

I've been playing with this quite a bit.  I cloned the cm_storage_smsv dictionary and collection scripts and renamed the database column to 'create_timestamp' as it seemed that 'timestamp' is a reserved word in MySQL.  This had no effect on the problem.  I tried altering the timestamp field to a 20-char string in the dictionary entry.  Also no effect.  In the reservation SQL of the create snapshot command I changed CURRENT_TIMESTAMP to a static string and noticed that immediately the reserved data changed to my static string.  It is as if the reservation SQL of the create snapshot command is being executed and added to the filter data when the filter is executed.  This leads me to believe that anything relying on putting a timestamp into the reservation data will be adversely affected.

 

The problem with this is that as part of our snapshot maintenance I would like to delete snapshots older than the N-th number matching a regex pattern.  If multiple snapshots only exist in the WFA reservation data the timestamps are not accurate to actual age and the wrong snaphosts are deleted.  If the data collection script runs frequently this might not be a problem in a production sense, but it does leave the window open to incorrect snapshot management.

Re: Snapshot timestamp in cm_storage_smsv scheme

Hi:

 

 

Just check if my understanding is correct:

 

  1. When a snapshot create command is run, snapshots are being created with CURRENT_TIMESTAMP is reservation code.
  2. When filter is run, filter returns all with the Snapshots with CURRENT_TIMESTAMP, since for the filter everything is created with CURRENT_TIMESTAMP.
  3. However the creation time of the actual snapshots are different and this is correctly logged in the DB.
  4. So when the Datasource acquisition happens, snapshot.timestamp is updated. If a filter is run on top of it now, the filter results are correct.

If all the workflows are executed at the same time and CURRENT_TIMESTAMP is logged then the issue is a bit tricky to resolve.

However if the workflows are executed in different times and still same timestamp is logged in reservation, then we need to see.

 

Regards

Abhi

Re: Snapshot timestamp in cm_storage_smsv scheme

[ Edited ]

Thanks for looking into this Abhi,

 

To answer your questions:

 

1)  Yes, the snapshot create command is adding reservation code using CURRENT_TIMESTAMP to populate the creation time field of the cm_storage_smsv schema. This should be safe as the creation time of the snapshot on the filer should be within a few seconds of the WFA server's time (assuming the filer and WFA server use the same TZ).

 

2)  When the filter is run with reservation enabled (the default), the filter returns the data in the WFA cache database (cm_storage_smsv) which is collected from the filer, plus the reservation data from recent snapshots.

 

3) The data from the cm_storage_smsv database is correct, but the data from reservations not in the database is incorrect -- it reports the timestamp of the snapshots as now().

 

4)  Yes, that is correct. When the datasource acquisition happens, the cm_storage_smsv data is updated with data from the filer and the reservation data removed. The timestamp of snapshots is then correct. It is only the data stored in reservations that is incorrect. It seems that the reservation SQL code is being executed at the time the filter is run.

 

 

It is unclear whether this issue will cause any problems in production, but it's causing problems with my testing and development, specifically around finding the oldest snapshots to delete. For instance, in my lab I will create 5 snapshots in successsion (perhaps 1 per minute). I am working on code to identify and keep the newest X snapshots and delete the oldest. If X is 3, then 2 of the snapshots should be deleted. Because the datasource acquisition hasn't occurred, and the data returned by the filter shows all 5 snapshots with the same timestamp, the code cannot accurately choose which snapshots to keep and which to delete.

 

My running theory is that WFA does not actually store the reservation data in a database, but rather stores a record of recent transactions in some other format.  When a filter is run against a specific schema, the reservation code from recent commands is then executed and added to the data retreived from the MySQL cache database.  I haven't had time to test this theory, but it should be straightforward to test in a dummy schema.

 

FYI:  I have opened case 2006507321 with NetApp support, but have yet to receive any meaningful feedback from them.  If you are able to assist, I would greatly appreciate it.

 

Regards,

Brian Jones

 

 

Re: Snapshot timestamp in cm_storage_smsv scheme

Thanks Brian for your mail. Another follow up question.

 

1. So when you are creating the snapshots every 1 minute in your dev environment, the current time is always T for all the snapshots?

 

2. If T is the time when the oldest snapshot is taken, then the next few snapshots time is not showing T+1, T+2, T+3 ....

 

though  CURRENT_TIME is specified and you have maintained a gap of 1 minute while taking the snapshots?

 

3. In production, time stamps will be at least few minutes apart, in my opinion. I guess a minimum of 10 minutes apart.

 

 

Regards

Abhi

 

Re: Snapshot timestamp in cm_storage_smsv scheme

Abhi,

 

1)  Yes, the time for all snapshots is reported as the same time T. This is only when the data comes from reservations, and has not been updated from the datasource.

 

2)  The time T reported is not the time any of the snapshots was taken. It is the time T when the filter is executed. Each time the filter is run, the time T is reported as the time when the filter is run. The actual time any of the snapshots was created is not represented.

 

3) In production it is possible that the datasource update would occur frequently enough, or snapshot management run infrequently enough, to make any problem in the reservation data irrelevant.

 

 

Regards,

Brian Jones

Re: Snapshot timestamp in cm_storage_smsv scheme

Able to reproduce the issue.

 

Stay tuned for more updates, most likely tomorrow.

 

Regards

Abhi

Re: Snapshot timestamp in cm_storage_smsv scheme

Hi Brian,

 

In WFA, if a reservation script has dynamic value generation, it cannot work seamlessly.

 

Reservation scripts run at the time of need. If a filter has to include reservation data or a workflow planning has to include reservation data, it runs those reservation SQL statements at that point of time and returns data. In this case, the reservation script had dynamic data i.e “CURRENT_TIMESTAMP” whose value will keep differing every time a filter or planning runs.

Unfortunately, this is a limitation in the WFA platform currently.

 

Now, there are three things I can think of:

 

a. As you probably noticed, making the timestamp value static will work. So, would it be feasible to modify the command you are using to take timestamp as a parameter and use that in the reservation script. In the workflow, you could fill up the parameter using a MVEL expression to get the current timestamp.  That way the timestamp value will be generated during planning and then the same value will be used during every run of that reservation script. This is better than dynamic issue that is happening currently, but there will be a time difference between the actual snapshot creation time and the timestamp value. This difference will be equivalent to planning time versus the time the command actually got executed

 

Note: This is not a great solution per say, it is a workaround to reduce the problem

 

b. You did mention this already. If frequency of data source acquisition can increase or is run at the end of the workflow, then the problem is reduced.

 

c. We will file an enhancement request to support a mode where the reservation script can get values from the execution of a command. If that was possible, then the command execution could update the timestamp value and after that whenever reservation script ran, it could have the accurate value.

 

Hope this helps..

 

 

Thanks,

Shailaja