Subscribe

Slow snapmirror jobs and questionable metrics (CDOT 8.2.3)

Our environment has a history of a few volumes consistently having laggy snapmirror relationships between one site and another. We’re running CDOT 8.2.3 on two filers linked by a fairly slow pipe, and the source filer is about 1500 miles away from the destination.

 

One relationship in particular is as follows:

 

  • The snapmirror job is set to kick off once per day, at 2200 ET
  • The source volume takes daily snapshots at 0210 ET, and weekly snapshots on Sundays at 0215 ET
  • Today's date is Wednesday 08/12/2015 at 1200 ET.
  • The snapmirror job has been in progress for several days. On the destination filer, the current Transfer Snapshot is "snapmirror.foobar.2015-08-09_220100"
  • The Snapshot Progress for the Transfer Snapshot is currently 77.09GB.
  • On the source filer, I verified the name and create-time of the snapshot that's currently in flight. However, the size of this snapshot is 71.32MB, three orders of magnitude smaller.

 

Am I misunderstanding something about how snapshots and snapmirror work? The snapshot that should be snapmirrored is about 71MB, and yet the progress of the snapshot that's in flight is 77GB so far, and growing. What am I missing?

 

Another possibly interesting fact about this snapmirror job:

 

  • When viewing the details of this snapmirror job (snapmirror show -destination-volume foobar -instance), the Total Transfer Time in Seconds is 424397 seconds, which is a little under five days.
  • Five days have not actually elapsed since the start of this snapmirror job. Total elapsed time from the start of the job (08/09/2015 2200 ET) until now (08/12/2015 1200ET) is about 2.5 days.
  • While I've been drafting this post (call it 15-20 minutes), I've checked the snapmirror instance a couple of times, and that value of 424397 seconds has not incremented.
  • The Snapshot Progress has incremented about as expected, from 77.09GB to 77.68GB

 

Shouldn't the Total Transfer Time in Seconds number increment as time progresses?

 

Thanks for the help, everyone!

Re: Slow snapmirror jobs and questionable metrics (CDOT 8.2.3)

Snapshot "size" is amount of changed or deleted data, while snap mirror transfers new data. Did you check Snap delta between two snap mirror snapshots?

I believe, total transfer time refers to the last completed snap mirror transfer, not the one in progress.

Re: Slow snapmirror jobs and questionable metrics (CDOT 8.2.3)

I'm not sure if I know a command to measure the delta between two snapmirror*** snapshots (or any other snapshots for that matter). Is it simply a matter of looking at the size of any two snapshots and doing some arithmetic?

Re: Slow snapmirror jobs and questionable metrics (CDOT 8.2.3)

Hmm ... you are right, seems that cDOT does not have direct analog. Command map suggests "snap delta" from node shell. As long as volumes are still visible fom node shell this should be possible.

Re: Slow snapmirror jobs and questionable metrics (CDOT 8.2.3)

To get more infomration use NetApp API to get all of the data.

        Get-NcSnapmirror

You can get how much has bee transmitted and when it started so you can caclulate how much kb per second.

Did you look at all of you peer relantionships? Do you have one LIF for each node on both cDOT systems (source and destination)?

I would to do a ping from the lif you think that data is being transmitted on, then do a traceroute.