topic Re: vfiler dr resync failing in Network and Storage Protocols

vfiler dr resync failing

PKKASTURI — Thu, 30 Aug 2012 12:24:40 GMT

Hi,

I'm trying to perform vfiler dr resync from one filer to other filer. But didn't success, getting below error. I initially did vfiler dr configure on destination filer (netapp-02) and it got successfull. i've added some data and now i'm trying to syncing the data from destination (netapp-02) filer to source (netapp-01) filer.

netapp-01> vfiler status

vfiler0 running

vf-test stopped

netapp-01> vfiler dr resync vf-test@netapp-02

netapp-02's Administrative login: root

netapp-02's Administrative password:

cannot resync path /vol/src_vol: transfer aborted because of network error

Mon Aug 27 19:55:54 GMT [replication.dst.resync.failed:error]: SnapMirror resync of src_vol to netapp-02:src_vol : transfer aborted because of network error.

netapp-01> df -h test_vol

Filesystem total used avail capacity Mounted on

/vol/test_vol/ 240MB 66MB 173MB 28% /vol/test_vol/

netapp-01> snapmirror status

Snapmirror is on.

Source Destination State Lag Status

netapp-01:test_vol netapp-02:sm_vol Source - Idle

netapp-01:src_vol netapp-02:src_vol Source 161:51:16 Idle

netapp-01:test_vol netapp-02:test_vol Source 161:51:14 Idle

netapp-01>

netapp-02> df -h test_vol

Filesystem total used avail capacity Mounted on

/vol/test_vol/ 240MB 73MB 166MB 31% /vol/test_vol/

netapp-02> snapmirror status

Snapmirror is on.

Source Destination State Lag Status

netapp-01:src_vol netapp-02:src_vol Broken-off 06:25:12 Idle

netapp-01:test_vol netapp-02:test_vol Broken-off 06:25:10 Idle

netapp-02:src_vol netapp-01:src_vol Source - Idle

netapp-02:test_vol netapp-01:test_vol Source - Idle

netapp-02>

can anyone tell me what i'm missing?

Thanks,

Phani

Re: vfiler dr resync failing

AGUMADAVALLI — Thu, 30 Aug 2012 14:31:41 GMT

Hi Phani,

Once you do the vfiler dr configure, the vfiler is active on the destination and all its entities are migrated too.

on the source filer you need to clean up the RC file and resync it, if not you will also duplicate IP's or conflicting IP's. Check the RC files on the source and destination, correct them, you should be goood to go.

thank you,

AK G

Re: vfiler dr resync failing

scottgelb — Thu, 30 Aug 2012 15:05:02 GMT

Can you manually "snapmirror resync" one of the volumes? dr resync does that for all the volumes in the vfiler in the background so that would be a good test for snapmirror.access. Also, you don't have "-c secure" listed in the syntax so it will use rsh instead of ssh and you need rsh enabled to run the command as you have above.

Re: vfiler dr resync failing

bpk30102179 — Sat, 01 Sep 2012 05:08:36 GMT

Hello Phani,

Please share the volume status from the source and destinations filers.

Re: vfiler dr resync failing

andrewgeorge — Wed, 05 Sep 2012 06:29:53 GMT

try doing a snapmirror resyc of the individual volume (and if that works break it again)

I'd be suspecting that src_vol on netapp-01 has got a snapshot that can't be deleted automatically, so a snap list of src_vol on each filer would show the difference.

Re: vfiler dr resync failing

PKKASTURI — Wed, 05 Sep 2012 07:20:03 GMT

Hi,

Tried both vfiler dr resync and snapmirror resync commands, getting the same error mentioned below.

netapp-01>vfiler dr resync -c secure vf-test@netapp-02

netapp-02's Administrative login: root

netapp-02's Administrative password:

cannot resync path /vol/src_vol: transfer aborted because of network error

Mon Aug 27 22:33:22 GMT [replication.dst.resync.failed:error]: SnapMirror resync of src_vol to netapp-02:src_vol : transfer aborted because of network error.

netapp-01> snapmirror resync -S netapp-02:test_vol netapp-01:test_vol

Snapmirror resynchronization of test_vol to netapp-02:test_vol : transfer aborted because of network error

Aborting resync.

Mon Aug 27 22:32:11 GMT [replication.dst.resync.failed:error]: SnapMirror resync of test_vol to netapp-02:test_vol : transfer aborted because of network error.

the following are the log messges taken from snapmirror.log file.

Tue Aug 21 10:46:40 GMT [snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume test_vol to itself, operation not permitted.

Tue Aug 21 11:00:04 GMT [kern.uptime.filer:info]: 11:00am up 22:45 0 NFS ops, 775 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops

Tue Aug 21 11:03:46 GMT [snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume src_vol to itself, operation not permitted.

Tue Aug 21 11:04:23 GMT [snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume test_vol to itself, operation not permitted.

Tue Aug 21 11:05:35 GMT [snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume src_vol to itself, operation not permitted.

Re: vfiler dr resync failing

scottgelb — Wed, 05 Sep 2012 12:39:21 GMT

options snapmirror.access has permissions on both sides? It may have been set up one way only..

Re: vfiler dr resync failing

PKKASTURI — Wed, 05 Sep 2012 13:26:05 GMT

No. kept snapmirror.access was set to "all" on both source and destination filers.

netapp-01> options snapmirror

snapmirror.access all

netapp-02> options snapmirror

snapmirror.access all

Re: vfiler dr resync failing

scottgelb — Wed, 05 Sep 2012 13:28:51 GMT

Snap list output from both controllers for that specific volume?

Sent from my iPhone 4S

Re: vfiler dr resync failing

SHYAMJAIK — Wed, 05 Sep 2012 13:37:00 GMT

Can you check the value in snapmirror.allow file in etc on the source filer?

Re: vfiler dr resync failing

scottgelb — Wed, 05 Sep 2012 13:40:16 GMT

Access is set to all so allow file is not used. Only when set to legacy. We are all thinking similarly which is good

Sent from my iPhone 4S

Re: vfiler dr resync failing

SHYAMJAIK — Wed, 05 Sep 2012 14:00:37 GMT

Thank you Scott for the tip. Phani, Connectivity checks were all done to the source filer? also DNS, /etc/hosts.?

Re: vfiler dr resync failing

spwilkinson — Fri, 28 Mar 2014 15:41:19 GMT

I'm having the same issue here when doing a vfiler dr resync. I'm using a sim running 8.2 for testing.

sim04> vfiler dr resync -c secure dr@sim03

sim03's Administrative login: root

sim03's Administrative password:

cannot resync path /vol/dr2_root: transfer aborted because of network error

sim04> Fri Mar 28 15:21:01 GMT [sim04:replication.dst.resync.failed:error]: SnapMirror resync of dr2_root to sim03:dr2_root : transfer aborted because of network error.

sim03> Fri Mar 28 15:21:01 GMT [sim03:snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume dr2_root to itself, operation not permitted.

Re: vfiler dr resync failing

scottgelb — Fri, 28 Mar 2014 15:54:44 GMT

Does a manual snapmirror work? Probably a mirror or network issue. If manual mirror fails then we can troubleshoot from there. If it works then a vFiler dr issue to look at.

Sent from my iPhone 5

Re: vfiler dr resync failing

spwilkinson — Fri, 28 Mar 2014 18:16:26 GMT

A manual snapmirror doesn't work either. The above output was from a sim running 8.1.4 not 8.2. I was just testing to make sure it wasn't an issue with 8.2 but it's exactly the same. Full output below from the 8.2 sim.

sim02> vfiler dr activate cifsvfiler@sim01

CIFS local server is running.

Fri Mar 28 17:43:58 GMT [cifsvfiler@sim02:cifs.startup.local.succeeded:info]: CIFS: CIFS local server is running.

Fri Mar 28 17:43:58 GMT [sim02:httpd.config.mime.missing:warning]: /etc/httpd.mimetypes.sample file is missing.

Fri Mar 28 17:43:58 GMT [cifsvfiler@sim02:httpd.config.mime.missing:warning]: /etc/httpd.mimetypes file is missing.

Fri Mar 28 17:43:58 GMT [cifsvfiler@sim02:httpd.config.mime.missing:warning]: /etc/httpd.mimetypes.sample file is missing.

Vfiler cifsvfiler activated.

e0a: flags=0xe48867<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.0.21 netmask 0xffffff00 broadcast 192.168.0.255

ether 00:0c:29:0e:d8:74 (auto-1000t-fd-up) flowcontrol full

sim02> Fri Mar 28 17:43:59 GMT [sim02:cmds.vfiler.dr.activated:info]: Disaster recovery backup vFiler unit: 'cifsvfiler' of the vFiler unit at remote storage system: 'sim01' was activated.

sim02> ifconfig e0a

e0a: flags=0xe48867<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.0.21 netmask 0xffffff00 broadcast 192.168.0.255

inet 192.168.0.120 netmask 0xffffff00 broadcast 192.168.0.255

ether 00:0c:29:0e:d8:74 (auto-1000t-fd-up) flowcontrol full

sim01> vfiler dr resync -c secure cifsvfiler@sim02

sim02's Administrative login: root

sim02's Administrative password:

cannot resync path /vol/cifsvfiler_root: transfer aborted because of network error

sim01> Fri Mar 28 17:58:44 GMT [sim01:replication.dst.resync.failed:error]: SnapMirror resync of cifsvfiler_root to sim02:cifsvfiler_root : transfer aborted because of network error.

[sim01:snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume cifs to itself, operation not permitted.

Re: vfiler dr resync failing

scottgelb — Fri, 28 Mar 2014 18:19:20 GMT

Check the snapmirror.allow settings in case they got wacked… then if you can ping between source and target and vice-versa… then make sure the VM network didn’t move or disconnect anything.

Re: vfiler dr resync failing

spwilkinson — Fri, 28 Mar 2014 18:43:51 GMT

Yep, checked that and had it was still set to *. I've changed it to the host and can ping between both sims by ip/hostname no problem.

sim02> options snapmirror

snapmirror.access *

sim01> vfiler dr resync -c secure cifsvfiler@sim02

sim02's Administrative login: root

sim02's Administrative password:

cannot resync path /vol/cifsvfiler_root: transfer aborted because of network error

sim01> Fri Mar 28 18:34:03 GMT [sim01:replication.dst.resync.failed:error]: SnapMirror resync of cifsvfiler_root to sim02:cifsvfiler_root : transfer aborted because of network error.

sim02> Fri Mar 28 18:34:02 GMT [sim02:snapmirror.src.resync.toSelf:error]: SnapMirror cannot resync volume cifsvfiler_root to itself, operation not permitted.

Re: vfiler dr resync failing

scottgelb — Fri, 28 Mar 2014 18:46:20 GMT

Does snap list have a matching snapshot between the two volumes (probably does and doesn’t even seem to get that far but checking)… if you restrict the target volume does snapmirror manually work?

Re: vfiler dr resync failing

spwilkinson — Fri, 28 Mar 2014 20:19:22 GMT

Snap list below. The snapmirror relationship is sim01-sim02. Does the vfiler dr resync copy any new data back to the original source from the now active dr vfiler?

sim02>

Volume cifsvfiler_root

working...

%/used %/total date name

---------- ---------- ------------ --------

32% (32%) 0% ( 0%) Mar 28 17:42 sim02(4079432748)_cifsvfiler_root.2

39% (15%) 0% ( 0%) Mar 28 17:41 sim02(4079432748)_cifsvfiler_root.1

Volume cifsvfiler_data

working...

%/used %/total date name

---------- ---------- ------------ --------

36% (36%) 0% ( 0%) Mar 28 17:42 sim02(4079432748)_cifsvfiler_data.2

54% (38%) 0% ( 0%) Mar 28 17:41 sim02(4079432748)_cifsvfiler_data.1

Volume cifs

working...

%/used %/total date name

---------- ---------- ------------ --------

27% (27%) 0% ( 0%) Mar 28 17:42 sim02(4079432748)_cifs.2

42% (27%) 0% ( 0%) Mar 28 17:41 sim02(4079432748)_cifs.1

sim01>

Volume cifsvfiler_root

working...

%/used %/total date name

---------- ---------- ------------ --------

13% (13%) 0% ( 0%) Mar 28 17:42 sim02(4079432748)_cifsvfiler_root.2 (snapmirror)

Volume cifsvfiler_data

working...

%/used %/total date name

---------- ---------- ------------ --------

34% (34%) 0% ( 0%) Mar 28 17:42 sim02(4079432748)_cifsvfiler_data.2 (snapmirror)

Volume cifs

working...

%/used %/total date name

---------- ---------- ------------ --------

21% (21%) 0% ( 0%) Mar 28 17:42 sim02(4079432748)_cifs.2 (snapmirror)

Re: vfiler dr resync failing

scottgelb — Fri, 28 Mar 2014 20:29:32 GMT

On the controller where you run dr resync the volumes revert back / snaprestore to the last common snapshot. So you would dr resync back to production. Stop dr. Activate production then dr resync back to dr.

You have common snaps below. Snapmirror resync should work. Not sure what the network error is. Can you create a new mirror going the same direction separate from the vfilers?

Sent from my iPhone 5