vfiler dr resync failing

[PKKASTURI]

Hi,

I'm trying to perform vfiler dr resync from one filer to other filer. But didn't success, getting below error. I initially did vfiler dr configure on destination filer (netapp-02) and it got successfull. i've added some data and now i'm trying to syncing the data from destination (netapp-02) filer to source (netapp-01) filer.

netapp-01> vfiler status

vfiler0                          running

vf-test                          stopped

netapp-01> vfiler dr resync vf-test@netapp-02

netapp-02's Administrative login: root

netapp-02's Administrative password:

cannot resync path /vol/src_vol: transfer aborted because of network error

Mon Aug 27 19:55:54 GMT [replication.dst.resync.failed:error]: SnapMirror resync of src_vol to netapp-02:src_vol : transfer aborted because of network error.

netapp-01> df -h test_vol

Filesystem               total       used      avail capacity  Mounted on

/vol/test_vol/           240MB       66MB      173MB      28%  /vol/test_vol/

netapp-01> snapmirror status

Snapmirror is on.

Source                Destination           State          Lag        Status

netapp-01:test_vol    netapp-02:sm_vol      Source         -          Idle

netapp-01:src_vol     netapp-02:src_vol     Source         161:51:16  Idle

netapp-01:test_vol    netapp-02:test_vol    Source         161:51:14  Idle

netapp-01>

netapp-02> df -h test_vol

Filesystem               total       used      avail capacity  Mounted on

/vol/test_vol/           240MB       73MB      166MB      31%  /vol/test_vol/

netapp-02> snapmirror status

Snapmirror is on.

Source                Destination           State          Lag        Status

netapp-01:src_vol     netapp-02:src_vol     Broken-off     06:25:12   Idle

netapp-01:test_vol    netapp-02:test_vol    Broken-off     06:25:10   Idle

netapp-02:src_vol     netapp-01:src_vol     Source         -          Idle

netapp-02:test_vol    netapp-01:test_vol    Source         -          Idle

netapp-02>

can anyone tell me what i'm missing?

Thanks,

Phani

[AGUMADAVALLI]

Hi Phani,

Once you do the vfiler dr configure, the vfiler is active on the destination and all its entities are migrated too.

on the source filer you need to clean up the RC file and resync it, if not you will also duplicate IP's or conflicting IP's. Check the RC files on the source and destination, correct them, you should be goood to go.

thank you,

AK G

[scottgelb]

Can you manually "snapmirror resync" one of the volumes?  dr resync does that for all the volumes in the vfiler in the background so that would be a good test for snapmirror.access.  Also, you don't have "-c secure" listed in the syntax so it will use rsh instead of ssh and you need rsh enabled to run the command as you have above.

[bpk30102179]

Hello Phani,

Please share the volume status from the source and destinations filers.

[andrewgeorge]

try doing a snapmirror resyc of the individual volume (and if that works break it again)

I'd be suspecting that src_vol on netapp-01 has got a snapshot that can't be deleted automatically, so a snap list of src_vol on each filer would show the difference.

[PKKASTURI]

Hi,

Tried both vfiler dr resync and snapmirror resync commands, getting the same error mentioned below.

netapp-01>vfiler dr resync -c secure vf-test@netapp-02

netapp-02's Administrative login: root

netapp-02's Administrative password:

cannot resync path /vol/src_vol: transfer aborted because of network error

Mon Aug 27 22:33:22 GMT [replication.dst.resync.failed:error]: SnapMirror resync of src_vol to netapp-02:src_vol : transfer aborted because of network error.

netapp-01> snapmirror resync -S netapp-02:test_vol netapp-01:test_vol

Snapmirror resynchronization of test_vol to netapp-02:test_vol : transfer aborted because of network error

Aborting resync.

Mon Aug 27 22:32:11 GMT [replication.dst.resync.failed:error]: SnapMirror resync of test_vol to netapp-02:test_vol : transfer aborted because of network error.

the following are the log messges taken from snapmirror.log file.

Tue Aug 21 10:46:40 GMT [snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume test_vol to itself, operation not permitted.

Tue Aug 21 11:00:04 GMT [kern.uptime.filer:info]:  11:00am up 22:45 0 NFS ops, 775 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Tue Aug 21 11:03:46 GMT [snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume src_vol to itself, operation not permitted.

Tue Aug 21 11:04:23 GMT [snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume test_vol to itself, operation not permitted.

Tue Aug 21 11:05:35 GMT [snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume src_vol to itself, operation not permitted.

[scottgelb]

options snapmirror.access has permissions on both sides?  It may have been set up one way only..

[PKKASTURI]

No. kept snapmirror.access was set to "all" on both source and destination filers.

netapp-01> options snapmirror

snapmirror.access            all

netapp-02> options snapmirror

snapmirror.access            all

[scottgelb]

Snap list output from both controllers for that specific volume?

Sent from my iPhone 4S

[SHYAMJAIK]

Can you check the value in snapmirror.allow file in etc on the source filer?

[scottgelb]

Access is set to all so allow file is not used. Only when set to legacy. We are all thinking similarly which is good

Sent from my iPhone 4S

[SHYAMJAIK]

Thank you Scott for the tip. Phani, Connectivity checks were all done to the source filer? also DNS, /etc/hosts.?

[spwilkinson]

I'm having the same issue here when doing a vfiler dr resync. I'm using a sim running 8.2 for testing.

sim04> vfiler dr resync -c secure dr@sim03

sim03's Administrative login: root

sim03's Administrative password:

cannot resync path /vol/dr2_root: transfer aborted because of network error

sim04> Fri Mar 28 15:21:01 GMT [sim04:replication.dst.resync.failed:error]: SnapMirror resync of dr2_root to sim03:dr2_root : transfer aborted because of network error.

sim03> Fri Mar 28 15:21:01 GMT [sim03:snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume dr2_root to itself, operation not permitted.

[scottgelb]

Does a manual snapmirror work? Probably a mirror or network issue. If manual mirror fails then we can troubleshoot from there. If it works then a vFiler dr issue to look at.

Sent from my iPhone 5

[spwilkinson]

A manual snapmirror doesn't work either. The above output was from a sim running 8.1.4 not 8.2. I was just testing to make sure it wasn't an issue with 8.2 but it's exactly the same. Full output below from the 8.2 sim.

sim02> vfiler dr activate cifsvfiler@sim01

CIFS local server is running.

Fri Mar 28 17:43:58 GMT [cifsvfiler@sim02:cifs.startup.local.succeeded:info]: CIFS: CIFS local server is running.

Fri Mar 28 17:43:58 GMT [sim02:httpd.config.mime.missing:warning]: /etc/httpd.mimetypes.sample file is missing.

Fri Mar 28 17:43:58 GMT [cifsvfiler@sim02:httpd.config.mime.missing:warning]: /etc/httpd.mimetypes file is missing.

Fri Mar 28 17:43:58 GMT [cifsvfiler@sim02:httpd.config.mime.missing:warning]: /etc/httpd.mimetypes.sample file is missing.

Vfiler cifsvfiler activated.

e0a: flags=0xe48867<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

        inet 192.168.0.21 netmask 0xffffff00 broadcast 192.168.0.255

        ether 00:0c:29:0e:d8:74 (auto-1000t-fd-up) flowcontrol full

sim02> Fri Mar 28 17:43:59 GMT [sim02:cmds.vfiler.dr.activated:info]: Disaster recovery backup vFiler unit: 'cifsvfiler' of the vFiler unit at remote storage system: 'sim01' was activated.

sim02> ifconfig e0a

e0a: flags=0xe48867<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

        inet 192.168.0.21 netmask 0xffffff00 broadcast 192.168.0.255

        inet 192.168.0.120 netmask 0xffffff00 broadcast 192.168.0.255

        ether 00:0c:29:0e:d8:74 (auto-1000t-fd-up) flowcontrol full

sim01> vfiler dr resync -c secure cifsvfiler@sim02

sim02's Administrative login: root

sim02's Administrative password:

cannot resync path /vol/cifsvfiler_root: transfer aborted because of network error

sim01> Fri Mar 28 17:58:44 GMT [sim01:replication.dst.resync.failed:error]: SnapMirror resync of cifsvfiler_root to sim02:cifsvfiler_root : transfer aborted because of network error.

[sim01:snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume cifs to itself, operation not permitted.

[scottgelb]

Check the snapmirror.allow settings in case they got wacked… then if you can ping between source and target and vice-versa… then make sure the VM network didn’t move or disconnect anything.

[spwilkinson]

Yep, checked that and had it was still set to *. I've changed it to the host and can ping between both sims by ip/hostname no problem.

sim02> options snapmirror

snapmirror.access            *

sim01> vfiler dr resync -c secure cifsvfiler@sim02

sim02's Administrative login: root

sim02's Administrative password:

cannot resync path /vol/cifsvfiler_root: transfer aborted because of network error

sim01> Fri Mar 28 18:34:03 GMT [sim01:replication.dst.resync.failed:error]: SnapMirror resync of cifsvfiler_root to sim02:cifsvfiler_root : transfer aborted because of network error.

sim02> Fri Mar 28 18:34:02 GMT [sim02:snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume cifsvfiler_root to itself, operation not permitted.

[scottgelb]

Does snap list have a matching snapshot between the two volumes (probably does and doesn’t even seem to get that far but checking)… if you restrict the target volume does snapmirror manually work?

[spwilkinson]

Snap list below. The snapmirror relationship is sim01-sim02. Does the vfiler dr resync copy any new data back to the original source from the now active dr vfiler?

sim02>

Volume cifsvfiler_root

working...

  %/used       %/total  date          name

----------  ----------  ------------  --------

32% (32%)    0% ( 0%)  Mar 28 17:42  sim02(4079432748)_cifsvfiler_root.2

39% (15%)    0% ( 0%)  Mar 28 17:41  sim02(4079432748)_cifsvfiler_root.1

Volume cifsvfiler_data

working...

  %/used       %/total  date          name

----------  ----------  ------------  --------

36% (36%)    0% ( 0%)  Mar 28 17:42  sim02(4079432748)_cifsvfiler_data.2

54% (38%)    0% ( 0%)  Mar 28 17:41  sim02(4079432748)_cifsvfiler_data.1

Volume cifs

working...

  %/used       %/total  date          name

----------  ----------  ------------  --------

27% (27%)    0% ( 0%)  Mar 28 17:42  sim02(4079432748)_cifs.2

42% (27%)    0% ( 0%)  Mar 28 17:41  sim02(4079432748)_cifs.1

sim01>

Volume cifsvfiler_root

working...

  %/used       %/total  date          name

----------  ----------  ------------  --------

13% (13%)    0% ( 0%)  Mar 28 17:42  sim02(4079432748)_cifsvfiler_root.2 (snapmirror)

Volume cifsvfiler_data

working...

  %/used       %/total  date          name

----------  ----------  ------------  --------

34% (34%)    0% ( 0%)  Mar 28 17:42  sim02(4079432748)_cifsvfiler_data.2 (snapmirror)

Volume cifs

working...

  %/used       %/total  date          name

----------  ----------  ------------  --------

21% (21%)    0% ( 0%)  Mar 28 17:42  sim02(4079432748)_cifs.2 (snapmirror)

[scottgelb]

On the controller where you run dr resync the volumes revert back / snaprestore to the last common snapshot. So you would dr resync back to production. Stop dr. Activate production then dr resync back to dr.

You have common snaps below. Snapmirror resync should work. Not sure what the network error is. Can you create a new mirror going the same direction separate from the vfilers?

Sent from my iPhone 5

[spwilkinson]

Thought so. I can create a new snapmirror relationship and transfer no problem. There is the error below on the source filer from the dr resync.

Transfer Status: resync to self not allowed