ONTAP Discussions

Broken? snapmirrors - how to delete them ?

hardy_maxa

Hi all

On my filer, I was doing some backup testing, and now have these snapmirrors showing, which I can't delete

aushafs2b> snapmirror status

Snapmirror is off.

Source                 Destination                  State          Lag        Status

aushafs1b:admin        aushafs2b:admin              Snapmirrored   02:35:19   Idle

aushafs1b:catiax       aushafs2b:catiax             Snapmirrored   10:35:19   Idle

aushafs1b:pdm          aushafs2b:pdm                Snapmirrored   10:05:18   Idle

aushafs1b:zen          aushafs2b:zen                Snapmirrored   09:55:20   Idle

aushafs2b:admin        aushafs2b:NDMP_LOCAL_005614  Source         -          Idle

aushafs2b:images       aushafs2b:NDMP_LOCAL_005617  Source         -          Idle

aushafs2b:domino2_vol  aushafs2b:nrst1a             Source         -          Idle

The first 4 entries are fine - they are my mirrors between systems.

But I want to cleanup the last three.

As far as I can tell, there are no backup or smtape sessions still active.

Thanks,

Hardy

23 REPLIES 23

SHARMAMANISH

Can you send me the output of  options snapmirror.enable

hardy_maxa

Here's all the snapmirror options

aushafs2b> options snapmirror

snapmirror.access            legacy

snapmirror.checkip.enable    off

snapmirror.delayed_acks.enable on

snapmirror.enable            on

snapmirror.log.enable        on

snapmirror.vbn_log_enable    off        (value might be overwritten in takeover)

snapmirror.volume.local_nwk_bypass.enable on

aushafs2b>

lhoffman1

again:   pls output the command:

"snapmirror status -l aushafs2b:domino2_vol  aushafs2b:nrst1a"

  

thx


hardy_maxa

aushafs2b> snapmirror status -l aushafs2b:domino2_vol  aushafs2b:nrst1a
Snapmirror is on.

Source:                 aushafs2b:domino2_vol
Destination:            aushafs2b:nrst1a
Status:                 Idle
Progress:               -
State:                  Source
Lag:                    -
Mirror Timestamp:       -
Base Snapshot:          -
Current Transfer Type:  Store
Current Transfer Error: -
Contents:               -
Last Transfer Type:     -
Last Transfer Size:     -
Last Transfer Duration: -
Last Transfer From:     -
aushafs2b>

Any ideas ?

Thx

ANANDONTAP

hi all,

go to designation end follow the below steps,

snapmirror status-- check it is snapmirror on status (continue follow steps)

which u want to remove entry in this output ensure.

snapmirror break designation volume

goto ---> rdfile /etc/snapmirror.conf --take this output in notepad

in notepad output--> which line of snapmirror you want to remove ensure and put # comment symbol.

wrfile /etc/snapmirror.conf--> paste the commented output and ctrl+c to save and exit

now u can run snapmirror status all should be clear.

hardy_maxa

Thanks, but this doesn't help either

aushafs2b> snapmirror break aushafs2b:nrst1a

snapmirror break:  nrst1a: destination is offline, is restricted, or does not exist

aushafs2b> rdfile /etc/snapmirror.conf

#Regenerated by registry Wed Jan 15 00:39:45 GMT 2014

aushafs1b:catiax aushafs2b:catiax - 0 0,12 * *

aushafs1b:admin aushafs2b:admin - 0 0,8,16 * *

aushafs1b:pdm aushafs2b:pdm - 30 0,12 * *

aushafs1b:zen aushafs2b:zen - 40 0,12 * *

aushafs2b>

The entries are not in the conf file, but still show in the status

Source                 Destination                  State          Lag        Status

aushafs1b:admin        aushafs2b:admin              Snapmirrored   05:36:54   Idle

aushafs1b:catiax       aushafs2b:catiax             Snapmirrored   01:36:58   Idle

aushafs1b:pdm          aushafs2b:pdm                Snapmirrored   01:06:59   Idle

aushafs1b:zen          aushafs2b:zen                Snapmirrored   00:56:59   Idle

aushafs2b:admin        aushafs2b:NDMP_LOCAL_005614  Source         -          Idle

aushafs2b:images       aushafs2b:NDMP_LOCAL_005617  Source         -          Idle

aushafs2b:domino2_vol  aushafs2b:nrst1a             Source         -          Idle

aushafs2b>

I know its not windows, but what about a reboot ?

ANANDONTAP

hi,

aushafs2b:admin        aushafs2b:NDMP_LOCAL_005614  Source         -          Idle

aushafs2b:images       aushafs2b:NDMP_LOCAL_005617  Source         -          Idle

aushafs2b:domino2_vol  aushafs2b:nrst1a             Source         -          Idle

this should be source. please follow my steps in designation end.

can please share the below output from dst end.

snapmirror status

df -Vh nrst1a

lun show -l nrst1a

rdfile /etc/snapmirror.conf

lun show unmapped

one more Question this replication within filer or across the filer?

pburcin

We are having the same issue as Max Hardy.  Our problem is related to the use of TSM to create NFS volume backups.  TSM is creating and starting the smtape jobs.  We currently have more than 220 orphaned smtape jobs on the controllers where the destination is an NDMP_local output like in Max Hardy's original post.  If there is no snapmirror destination to "snapmirror release", then it appears that there is no way to remove these orphans in the snapmirror status list.  They are not found in the snapmirror.conf file because they are supposed to be temporary associations and should go away when the TSM backup has completed.  We currently have a case open because one of our controller pairs is experiencing filer panics, crashing the controllers.  We have been told that root cause is smtape exhausting resources.  The crash and subsequent reboot did not remove the 220 orphaned entries from the snapmirror status list.   I am left believing that the orphaned NDMP_local snapmirrors that cannot be removed, released or otherwise deleted are the root cause of the filer crashes.  Perhaps we have all found a bug in the smtape engine.  We are still waiting for an answer from support on how to remove the orphans. 

hardy_maxa

Hi pburcin

I look forward to hearing Netapp's reply.  I havn't gone that far yet - I stopped using the smtape backups partly because of this issue.

pburcin

The official response from support is that these "orphaned" smtape jobs that show up in the snapmirror status list can be removed by using "snapmirror release <volname>  <destination controller:NDMP_local_xxxxxx>".  You WILL receive an error that indicates there is "no release-able destination".  However, the orphaned job that you released DOES get removed from the snapmirror status list.  In my case, I have hundreds of these orphaned smtape jobs to remove across numerous controllers running Data ONTAP 8.1.2x.  So the manual removal process will be tedious.  Basically, that was my reply to support.  There's gotta be a better way to fix this permanently!!!  We should not have to watch this and remove the orphans manually on a regular basis.

So this is not a fix.  It is the way to remove the orphaned smtape jobs from the snapmirror status list only.  I received no admission of a bug found or a fix in process.

JNSTEGEMANOEH

"snapmirror release <volname>  <destination controller:NDMP_local_xxxxxx>"    --->>> Worked a charm, cheers.

lhoffman1

No,  a reboot doesn't help.   I think the problem is on Backup Server side,  not filer.

But..  you are set the mirror from aushafs1b  to aushafs2b  .

What about the snapmirror status from  aushafs1b  ?   and  the  "snap list admin"   by aushafs1b  ??


Hi Hardy,

Have you already tried snapmirror release from aushafs2b?

Yes I have - Thanks.  No good.

lhoffman1

Can you show us the output from  "snap list"  admin / images and domino2_vol  ?

If you have no orphaned records within the  "snapmirror.conf" file,  you have some relations within old snapshots.

btw.  I wonder,  your snapmirror is set to off,  although your idletime is not so old.

hardy_maxa

Snap List output:

Volume images
  %/used       %/total  date          name
----------  ----------  ------------  --------
  0% ( 0%)    0% ( 0%)  Feb 03 00:00  weekly.0
  0% ( 0%)    0% ( 0%)  Dec 09 12:03  aushafs2b(1785237204)_images.1109

Volume admin
  %/used       %/total  date          name
----------  ----------  ------------  --------
  0% ( 0%)    0% ( 0%)  Feb 05 08:00  aushafs2b(1785237204)_admin.1792
  0% ( 0%)    0% ( 0%)  Feb 05 00:00  nightly.0
  0% ( 0%)    0% ( 0%)  Feb 05 00:00  aushafs2b(1785237204)_admin.1791
  0% ( 0%)    0% ( 0%)  Feb 04 19:00  hourly.0
  0% ( 0%)    0% ( 0%)  Feb 04 15:00  hourly.1
  0% ( 0%)    0% ( 0%)  Feb 04 13:00  hourly.2
  0% ( 0%)    0% ( 0%)  Feb 04 11:00  hourly.3
  0% ( 0%)    0% ( 0%)  Feb 04 09:00  hourly.4
  0% ( 0%)    0% ( 0%)  Feb 04 00:00  nightly.1
  0% ( 0%)    0% ( 0%)  Feb 03 19:00  hourly.5
  0% ( 0%)    0% ( 0%)  Feb 03 00:00  weekly.0
  0% ( 0%)    0% ( 0%)  Feb 02 00:00  nightly.2
  0% ( 0%)    0% ( 0%)  Feb 01 00:00  nightly.3
  0% ( 0%)    0% ( 0%)  Jan 31 00:00  nightly.4
  0% ( 0%)    0% ( 0%)  Jan 30 00:00  nightly.5
  0% ( 0%)    0% ( 0%)  Jan 29 00:00  nightly.6
  0% ( 0%)    0% ( 0%)  Jan 27 00:00  weekly.1
  0% ( 0%)    0% ( 0%)  Jan 20 00:00  weekly.2

Volume domino2_vol
working...

No snapshots exist.

I think snapmirror was set off when I noticed the issue.

Now though, it is on.

aushafs2b> snapmirror status

Snapmirror is on.

Source                 Destination                  State          Lag        Status

aushafs1b:admin        aushafs2b:admin              Snapmirrored   00:19:40   Idle

aushafs1b:catiax       aushafs2b:catiax             Snapmirrored   08:19:37   Idle

aushafs1b:pdm          aushafs2b:pdm                Snapmirrored   07:49:37   Idle

aushafs1b:zen          aushafs2b:zen                Snapmirrored   07:39:39   Idle

aushafs2b:admin        aushafs2b:NDMP_LOCAL_005614  Source         -          Idle

aushafs2b:images       aushafs2b:NDMP_LOCAL_005617  Source         -          Idle

aushafs2b:domino2_vol  aushafs2b:nrst1a             Source         -          Idle

aushafs2b>

snapmirror.conf looks fine too

aushafs2b> rdfile /etc/snapmirror.conf

#Regenerated by registry Wed Jan 15 00:39:45 GMT 2014

aushafs1b:catiax aushafs2b:catiax - 0 0,12 * *

aushafs1b:admin aushafs2b:admin - 0 0,8,16 * *

aushafs1b:pdm aushafs2b:pdm - 30 0,12 * *

aushafs1b:zen aushafs2b:zen - 40 0,12 * *

Any other ideas ?

aborzenkov

For images it should be enough to remove leftover snapshot “aushafs2b(1785237204)_images.1109”. Others two are really interesting. Could you post output of “smtape status -l”?

hardy_maxa

Not much - it just says "job not found"

MOHIT_FUJITSU

Can you post the output of "snapmirror status -l" ?

YEKYAWTHARMYINT

Use snapmirror release srcpath dst_filer:dstpath command to release the snapmirror relationship from source filer.

Announcements
NetApp on Discord Image

We're on Discord, are you?

Live Chat, Watch Parties, and More!

Explore Banner

Meet Explore, NetApp’s digital sales platform

Engage digitally throughout the sales process, from product discovery to configuration, and handle all your post-purchase needs.

NetApp Insights to Action
I2A Banner
Public