NDMP Backup aborting abruptly, Please help...!!!

dhaneswarsingh — Wed, 04 Jun 2025 18:42:05 GMT

Hi Techies,

We are having a volume: /vol/mir_mhd_user/ on a one of the Vfiler, Where Backup team is taking Weekely backup of it, Which is aborting automatically in the filer itself.

Unfortunately we dont have support from Netapp, Could some one please help us in resolving this issue...!!!

Note :- please find the logs in the attachment.

Backup logs shows:

dmp Sun Oct 2 16:36:58 EDT /vol/mir_mhd_user/(7) Tape_close (ndmp)
dmp Sun Oct 2 16:36:58 EDT /vol/mir_mhd_user/(7) Abort (746684 MB)

dmp Tue Oct 4 19:33:41 EDT /vol/mir_mhd_user/(3) Tape_close (ndmp)
dmp Tue Oct 4 19:33:41 EDT /vol/mir_mhd_user/(3) Abort (1252563 MB)

Filer invoked ABORT signal from hardware (allegedly). Normally it should invoke END signal once the DUMP is done.

Volume Details:-

---------------------

The volume is of about 4160g.

and verified the snap list of the volume it is having a ndmp : snapshot_for_backup.37323 (busy,backup[0],dump)

>>>>> Backup status of the volume is:

ID State Type Device Start Date Level Path
-- ----------- ---- ------ ------------ ----- ---------------
0 ACTIVE NDMP ndmp Oct 09 06:34 0 /vol/mir_mhd_user/

>>>>> Inodes utilization is as below:

Filesystem iused ifree %iused Mounted on
/vol/mir_mhd_user/ 3534931 28341758 11% /vol/mir_mhd_user/

>>>>>> Backup logs are as below:

Search "mir_mhd_user" (42 hits in 1 file)
D:\Users\dhaneswar.s\Desktop\bsk-fas01\bsk-fas01-04-10-2016\backup (42 hits)
Line 950: dmp Sun Oct 2 16:36:58 EDT /vol/mir_mhd_user/(7) Tape_close (ndmp)
Line 951: dmp Sun Oct 2 16:36:58 EDT /vol/mir_mhd_user/(7) Abort (746684 MB)
Line 952: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (reg inodes: 2963986 other inodes: 0 dirs: 378584 nt dirs: 91314 nt inodes: 93196 acls: 3934)
Line 953: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Phase 1 time: 2138196)
Line 954: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Phase 3: directories dumped: 469899)
Line 955: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Phase 3: wafl directory blocks read: 498196)
Line 956: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Phase 3: average wafl directory blocks per inode: 1)
Line 957: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Phase 3: average tape blocks per inode: 3)
Line 958: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Phase 3 throughput (MB sec): read 1 write 0)
Line 959: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Percent of phase3 time spent for: reading inos 5% dumping ino 94%)
Line 960: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Percent of phase3 dump time spent for: convert-wafl-dirs 66% lev0-ra 10%)
Line 961: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Phase 3 averages (usec): wafl load buf time 632 level 0 ra time 217)
Line 962: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Phase 4: inodes dumped: 568439)
Line 963: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Phase 4: wafl data blocks read: 204392938)
Line 964: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Phase 4: average wafl data blocks per inode: 359)
Line 965: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Phase 4: average tape data blocks per inode: 1436)
Line 966: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Percent of phase4 time spent for: reading inos 0% reading file data 0% dumping file data 0% unlocking bufs 0%)
Line 967: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg ( =0 <=64 <=4K <=32K <=64K <=128K <=256K)
Line 968: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg ( 3306 15393 41078 98474 97755 84456 71364)
Line 969: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg ( <=512K <=1M <=512M <=1G <=10G >10G)
Line 970: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg ( 55871 37971 62595 65 107 4)
Line 971: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Total files 568439)
Line 972: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (# buffers of filehistory sent dir: 32030 node: 7268 )
Line 973: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (# times filehistory send was blocked dir: 0 node: 0)
Line 974: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (# filehistory flush operations dir: 2 node: 1)
Line 975: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (# filehistory entries dir: 4099740 node: 930329 )
Line 976: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Dir to FH entry time stats (msec) numEntries: 4099740 min: 0 max: 1 avg: <1 tot: 1250)
Line 977: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Node to FH Entry time stats (msec) numEntries: 930329 min: 0 max: 1 avg: <1 tot: 293)
Line 978: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Dir FH to NDMP Entry Time Stats (msec) numEntries: 32030 min: 1 max: 57 avg: 1 tot: 395407)
Line 979: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Node FH to NDMP Entry Time Stats (msec) numEntries: 7268 min: 1 max: 75 avg: 1 tot: 108921)
Line 980: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Dir FH Choked Stats: numEntries: 32030 min: 0 max: 0 avg: 0 tot: 0)
Line 981: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Node FH Choked Stats: numEntries: 7268 min: 0 max: 0 avg: 0 tot: 0)
Line 982: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Tape write times (msec): average: 0 max: 205232)
Line 983: dmp Sun Oct 2 16:37:44 EDT /vol/mir_mhd_user/(7) Log_msg (Tape changes: 1)
Line 1040: dmp Sun Oct 2 18:59:08 EDT /vol/mir_mhd_user/(3) Start (Level 0, NDMP)
Line 1041: dmp Sun Oct 2 18:59:08 EDT /vol/mir_mhd_user/(3) Options (b=256, u)
Line 1042: dmp Sun Oct 2 18:59:08 EDT /vol/mir_mhd_user/(3) Snapshot (snapshot_for_backup.37076, Sun Oct 2 18:57:26 EDT)
Line 1043: dmp Sun Oct 2 18:59:08 EDT /vol/mir_mhd_user/(3) Tape_open (ndmp)
Line 1044: dmp Sun Oct 2 18:59:08 EDT /vol/mir_mhd_user/(3) Phase_change (I)
Line 1045: dmp Sun Oct 2 20:27:07 EDT /vol/mir_mhd_user/(3) Phase_change (II)
Line 1046: dmp Sun Oct 2 20:29:05 EDT /vol/mir_mhd_user/(3) Phase_change (III)
Line 1047: dmp Sun Oct 2 21:35:42 EDT /vol/mir_mhd_user/(3) Phase_change (IV)

Any suggestions will be highly appreciated...!!!

Re: NDMP Backup aborting abruptly, Please help...!!!

Jeff_Yao — Thu, 20 Oct 2016 10:49:40 GMT

try "dump null" and "dump tape" to see the difference. if dump null is good and dump tape isn't, then probably the tape.

try ndmpcopy to see it works?

Re: NDMP Backup aborting abruptly, Please help...!!!

dhaneswarsingh — Thu, 20 Oct 2016 12:08:08 GMT

Hi yaoguang,

Thanks alot for your response...!!!

i just tried with: dump 0f null /vol/vol0 from the Netapp filer it is as below:

in /etc/log/backup:

dmp Thu Oct 20 07:32:01 EDT /vol/vol0/(0) Start (Level 0)
dmp Thu Oct 20 07:32:01 EDT /vol/vol0/(0) Options (b=63)
dmp Thu Oct 20 07:32:01 EDT /vol/vol0/(0) Snapshot (snapshot_for_backup.37725, Thu Oct 20 07:31:50 EDT)
dmp Thu Oct 20 07:32:01 EDT /vol/vol0/(0) Tape_open (null)
dmp Thu Oct 20 07:32:01 EDT /vol/vol0/(0) Phase_change (I)
dmp Thu Oct 20 07:32:07 EDT /vol/vol0/(0) Phase_change (II)
dmp Thu Oct 20 07:32:09 EDT /vol/vol0/(0) Phase_change (III)
dmp Thu Oct 20 07:32:11 EDT /vol/vol0/(0) Phase_change (IV)
dmp Thu Oct 20 07:33:27 EDT /vol/vol0/(0) Phase_change (V)
dmp Thu Oct 20 07:33:27 EDT /vol/vol0/(0) Tape_close (null)
dmp Thu Oct 20 07:33:27 EDT /vol/vol0/(0) End (3165 MB)

Can you please let us know how to do: dump tape( a syntax will help)

>>> Do i need to perform this on acutal volume or any vol like vol0

>>> Is this ok to do in production hours | any performance issues will report.

the volume is about 4 TB.

Re: NDMP Backup aborting abruptly, Please help...!!!

Jeff_Yao — Thu, 20 Oct 2016 23:41:38 GMT

dump null is a way to check data integraty. so u need to run on ur data which u'd like to backup. and it'll affect ur performance of production. so recommend u do it some time idle?

dump tape is a way to check if there's any issue on tapes. syntax shows in this link:

https://kb.netapp.com/support/s/article/what-is-the-command-syntax-for-dump

below are some links u might think useful

https://kb.netapp.com/support/s/article/troubleshooting-workflow-ndmp-backup-is-aborted?language=en_US

https://kb.netapp.com/support/s/article/top-10-ndmp-issues-and-solutions?language=en_US

hopefully helps

Jeff

topic NDMP Backup aborting abruptly, Please help...!!! in Data Protection

NDMP Backup aborting abruptly, Please help...!!!

Re: NDMP Backup aborting abruptly, Please help...!!!

Re: NDMP Backup aborting abruptly, Please help...!!!

Re: NDMP Backup aborting abruptly, Please help...!!!