Broken? snapmirrors - how to delete them ?

hardy_maxa · ‎2014-02-03

Hi all

On my filer, I was doing some backup testing, and now have these snapmirrors showing, which I can't delete

aushafs2b> snapmirror status

Snapmirror is off.

Source Destination State Lag Status

aushafs1b:admin aushafs2b:admin Snapmirrored 02:35:19 Idle

aushafs1b:catiax aushafs2b:catiax Snapmirrored 10:35:19 Idle

aushafs1b:pdm aushafs2b:pdm Snapmirrored 10:05:18 Idle

aushafs1b:zen aushafs2b:zen Snapmirrored 09:55:20 Idle

aushafs2b:admin aushafs2b:NDMP_LOCAL_005614 Source - Idle

aushafs2b:images aushafs2b:NDMP_LOCAL_005617 Source - Idle

aushafs2b:domino2_vol aushafs2b:nrst1a Source - Idle

The first 4 entries are fine - they are my mirrors between systems.

But I want to cleanup the last three.

As far as I can tell, there are no backup or smtape sessions still active.

Thanks,

Hardy

aborzenkov · ‎2014-02-03

Use "snapmirror release" to remove old SnapMirror snapshots. You may also need to remove them from snapmirror.conf.

hardy_maxa · ‎2014-02-03

Thanks - This worked on the smtape snapmirror entries, but the not those still displayed in the output (above)

Any other ideas ?

aborzenkov · ‎2014-02-04

Check if you have leftover snapmirror snapshots on destination - manually remove them (they should be unlocked after snapmirror break). Those snapshots are not removed automatically.

YEKYAWTHARMYINT · ‎2014-02-03

Use snapmirror release srcpath dst_filer:dstpath command to release the snapmirror relationship from source filer.

lhoffman1 · ‎2014-02-04

Can you show us the output from "snap list" admin / images and domino2_vol ?

If you have no orphaned records within the "snapmirror.conf" file, you have some relations within old snapshots.

btw. I wonder, your snapmirror is set to off, although your idletime is not so old.

hardy_maxa · ‎2014-02-04

Snap List output:

Volume images
%/used       %/total date          name
---------- ---------- ------------ --------
0% ( 0%)    0% ( 0%) Feb 03 00:00 weekly.0
0% ( 0%)    0% ( 0%) Dec 09 12:03 aushafs2b(1785237204)_images.1109

Volume admin
%/used       %/total date          name
---------- ---------- ------------ --------
0% ( 0%)    0% ( 0%) Feb 05 08:00 aushafs2b(1785237204)_admin.1792
0% ( 0%)    0% ( 0%) Feb 05 00:00 nightly.0
0% ( 0%)    0% ( 0%) Feb 05 00:00 aushafs2b(1785237204)_admin.1791
0% ( 0%)    0% ( 0%) Feb 04 19:00 hourly.0
0% ( 0%)    0% ( 0%) Feb 04 15:00 hourly.1
0% ( 0%)    0% ( 0%) Feb 04 13:00 hourly.2
0% ( 0%)    0% ( 0%) Feb 04 11:00 hourly.3
0% ( 0%)    0% ( 0%) Feb 04 09:00 hourly.4
0% ( 0%)    0% ( 0%) Feb 04 00:00 nightly.1
0% ( 0%)    0% ( 0%) Feb 03 19:00 hourly.5
0% ( 0%)    0% ( 0%) Feb 03 00:00 weekly.0
0% ( 0%)    0% ( 0%) Feb 02 00:00 nightly.2
0% ( 0%)    0% ( 0%) Feb 01 00:00 nightly.3
0% ( 0%)    0% ( 0%) Jan 31 00:00 nightly.4
0% ( 0%)    0% ( 0%) Jan 30 00:00 nightly.5
0% ( 0%)    0% ( 0%) Jan 29 00:00 nightly.6
0% ( 0%)    0% ( 0%) Jan 27 00:00 weekly.1
0% ( 0%)    0% ( 0%) Jan 20 00:00 weekly.2

Volume domino2_vol
working...

No snapshots exist.

I think snapmirror was set off when I noticed the issue.

Now though, it is on.

aushafs2b> snapmirror status

Snapmirror is on.

Source Destination State Lag Status

aushafs1b:admin aushafs2b:admin Snapmirrored 00:19:40 Idle

aushafs1b:catiax aushafs2b:catiax Snapmirrored 08:19:37 Idle

aushafs1b:pdm aushafs2b:pdm Snapmirrored 07:49:37 Idle

aushafs1b:zen aushafs2b:zen Snapmirrored 07:39:39 Idle

aushafs2b:admin aushafs2b:NDMP_LOCAL_005614 Source - Idle

aushafs2b:images aushafs2b:NDMP_LOCAL_005617 Source - Idle

aushafs2b:domino2_vol aushafs2b:nrst1a Source - Idle

aushafs2b>

snapmirror.conf looks fine too

aushafs2b> rdfile /etc/snapmirror.conf

#Regenerated by registry Wed Jan 15 00:39:45 GMT 2014

aushafs1b:catiax aushafs2b:catiax - 0 0,12 * *

aushafs1b:admin aushafs2b:admin - 0 0,8,16 * *

aushafs1b:pdm aushafs2b:pdm - 30 0,12 * *

aushafs1b:zen aushafs2b:zen - 40 0,12 * *

Any other ideas ?

aborzenkov · ‎2014-02-04

For images it should be enough to remove leftover snapshot “aushafs2b(1785237204)_images.1109”. Others two are really interesting. Could you post output of “smtape status -l”?

hardy_maxa · ‎2014-02-04

Not much - it just says "job not found"

MOHIT_FUJITSU · ‎2014-02-05

Can you post the output of "snapmirror status -l" ?

SHARMAMANISH · ‎2014-02-04

Can you send me the output of options snapmirror.enable

hardy_maxa · ‎2014-02-05

Here's all the snapmirror options

aushafs2b> options snapmirror

snapmirror.access legacy

snapmirror.checkip.enable off

snapmirror.delayed_acks.enable on

snapmirror.enable on

snapmirror.log.enable on

snapmirror.vbn_log_enable off (value might be overwritten in takeover)

snapmirror.volume.local_nwk_bypass.enable on

aushafs2b>

lhoffman1 · ‎2014-02-10

again: pls output the command:

"snapmirror status -l aushafs2b:domino2_vol aushafs2b:nrst1a"

thx

hardy_maxa · ‎2014-02-10

aushafs2b> snapmirror status -l aushafs2b:domino2_vol aushafs2b:nrst1a
Snapmirror is on.

Source:                 aushafs2b:domino2_vol
Destination:            aushafs2b:nrst1a
Status:                 Idle
Progress:               -
State:                  Source
Lag:                    -
Mirror Timestamp:       -
Base Snapshot:          -
Current Transfer Type: Store
Current Transfer Error: -
Contents:               -
Last Transfer Type:     -
Last Transfer Size:     -
Last Transfer Duration: -
Last Transfer From:     -
aushafs2b>

Any ideas ?

Thx

ANANDONTAP · ‎2014-02-10

hi all,

go to designation end follow the below steps,

snapmirror status-- check it is snapmirror on status (continue follow steps)

which u want to remove entry in this output ensure.

snapmirror break designation volume

goto ---> rdfile /etc/snapmirror.conf --take this output in notepad

in notepad output--> which line of snapmirror you want to remove ensure and put # comment symbol.

wrfile /etc/snapmirror.conf--> paste the commented output and ctrl+c to save and exit

now u can run snapmirror status all should be clear.

hardy_maxa · ‎2014-02-10

Thanks, but this doesn't help either

aushafs2b> snapmirror break aushafs2b:nrst1a

snapmirror break: nrst1a: destination is offline, is restricted, or does not exist

aushafs2b> rdfile /etc/snapmirror.conf

#Regenerated by registry Wed Jan 15 00:39:45 GMT 2014

aushafs1b:catiax aushafs2b:catiax - 0 0,12 * *

aushafs1b:admin aushafs2b:admin - 0 0,8,16 * *

aushafs1b:pdm aushafs2b:pdm - 30 0,12 * *

aushafs1b:zen aushafs2b:zen - 40 0,12 * *

aushafs2b>

The entries are not in the conf file, but still show in the status

Source Destination State Lag Status

aushafs1b:admin aushafs2b:admin Snapmirrored 05:36:54 Idle

aushafs1b:catiax aushafs2b:catiax Snapmirrored 01:36:58 Idle

aushafs1b:pdm aushafs2b:pdm Snapmirrored 01:06:59 Idle

aushafs1b:zen aushafs2b:zen Snapmirrored 00:56:59 Idle

aushafs2b:admin aushafs2b:NDMP_LOCAL_005614 Source - Idle

aushafs2b:images aushafs2b:NDMP_LOCAL_005617 Source - Idle

aushafs2b:domino2_vol aushafs2b:nrst1a Source - Idle

aushafs2b>

I know its not windows, but what about a reboot ?

YEKYAWTHARMYINT · ‎2014-02-11

Hi Hardy,

Have you already tried snapmirror release from aushafs2b?

hardy_maxa · ‎2014-02-22

Yes I have - Thanks. No good.

lhoffman1 · ‎2014-02-11

No, a reboot doesn't help. I think the problem is on Backup Server side, not filer.

But.. you are set the mirror from aushafs1b to aushafs2b .

What about the snapmirror status from aushafs1b ? and the "snap list admin" by aushafs1b ??

ANANDONTAP · ‎2014-02-11

hi,

aushafs2b:admin aushafs2b:NDMP_LOCAL_005614 Source - Idle

aushafs2b:images aushafs2b:NDMP_LOCAL_005617 Source - Idle

aushafs2b:domino2_vol aushafs2b:nrst1a Source - Idle

this should be source. please follow my steps in designation end.

can please share the below output from dst end.

snapmirror status

df -Vh nrst1a

lun show -l nrst1a

rdfile /etc/snapmirror.conf

lun show unmapped

one more Question this replication within filer or across the filer?

pburcin · ‎2014-03-14

We are having the same issue as Max Hardy. Our problem is related to the use of TSM to create NFS volume backups. TSM is creating and starting the smtape jobs. We currently have more than 220 orphaned smtape jobs on the controllers where the destination is an NDMP_local output like in Max Hardy's original post. If there is no snapmirror destination to "snapmirror release", then it appears that there is no way to remove these orphans in the snapmirror status list. They are not found in the snapmirror.conf file because they are supposed to be temporary associations and should go away when the TSM backup has completed. We currently have a case open because one of our controller pairs is experiencing filer panics, crashing the controllers. We have been told that root cause is smtape exhausting resources. The crash and subsequent reboot did not remove the 220 orphaned entries from the snapmirror status list. I am left believing that the orphaned NDMP_local snapmirrors that cannot be removed, released or otherwise deleted are the root cause of the filer crashes. Perhaps we have all found a bug in the smtape engine. We are still waiting for an answer from support on how to remove the orphans.