Data Backup and Recovery

NDMP backup failure for large file system after 16hrs

saranraj456

Hello Friends,

 

NDMP Backup is failing with netbackup status code 99 for a File system of size 17 TB after 16 hrs from the start of the backup. No Progress in any of the NDMP phases.

 

 

Filer Backup logs:

 

dmp Sun Jan 22 02:00:43 CST 2017 /vol/pool2/(7) Error (job aborted)
dmp Sun Jan 22 02:00:43 CST 2017 /vol/pool2_1/(7) Tape_close (ndmp)
dmp Sun Jan 22 02:00:43 CST 2017 /vol/pool2_1/(7) Abort (0 MB)
dmp Sun Jan 22 02:00:43 CST 2017 /vol/pool2_1/(7) Log_msg (reg inodes: 3232764 other inodes: 22265 dirs: 165388 nt dirs: 0 nt inodes: 0 acls: 0)
dmp Sun Jan 22 02:00:43 CST 2017 /vol/pool2_1/(7) Log_msg (Phase 1 time: 0)
dmp Sun Jan 22 02:00:43 CST 2017 /vol/pool2_1/(7) Log_msg (Phase 3: directories dumped: 0)
dmp Sun Jan 22 02:00:43 CST 2017 /vol/pool2_1/(7) Log_msg (Phase 3: wafl directory blocks read: 0)
dmp Sun Jan 22 02:00:43 CST 2017 /vol/pool2_1/(7) Log_msg (Phase 3 throughput (MB sec): read 0 write 0)
dmp Sun Jan 22 02:00:43 CST 2017 /vol/pool2_1/(7) Log_msg (Percent of phase3 time spent for: reading inos 0% dumping ino 0%)

 

Netbackup logs:

 

01/08/2017 10:45:41 - Error ndmpagent (pid=51001) aborting operation - no mover progress

01/08/2017 10:45:43 - Error ndmpagent (pid=51001) NDMP backup failed, path = /vol/pool2_1/

01/08/2017 10:45:43 - Info ndmpagent (pid=51001)  tapprd01-b: DUMP: job aborted

01/08/2017 10:45:43 - Info ndmpagent (pid=51001)  tapprd01-b: DUMP: DUMP IS ABORTED

 

I have tried a dump with null to this volume , it works fine (cmd used ::"dump -0f null /vol/pool2_1")

 

Not sure on the trace of this issue.

 

Thanks
Saran

 

3 REPLIES 3

saranraj456

We have decided to split the backup with sub folders of multiple policy. Hope it might work well, however still doesn't understand cause of this issue.

Kat
  1. Error 99 is a generic error, used for NDMP backup to disk or tape.

    You may be out of resources because your filer reached the maximum number of threads.
    Because you're backing up to tape, you might want to try the steps in this article:

    https://kb.netapp.com/support/s/article/ka11A00000015mnQAA/troubleshooting-workflow-ndmp-backup-fails-to-start?language=en_US

  2. There are NDMP session limititations for either disk or tape backups.  Take a look at this article:

    https://kb.netapp.com/support/s/article/ka21A0000000j8DQAQ/how-do-ndmp-session-limitations-work-with-data-ontap-8-2-and-later?language=en_US

    Dump and Restore sessions: These are NDMP sessions directly responsible for backing up data to and from disk or tape. They interface with the NDMP data or tape service.

    Control or 'common' sessions: These are bi-directional NDMP control connections responsible for NDMP control commands or messages between the NDMP client (typically a backup application) and the NMDP server (typically a NetApp Storage Controller)

            NDMPsessions.PNG

 

Sergey_Osipov

Unfortunately this doesn't tell us anything except the dump was aborted.

I would recommend you to enable ndmp debug logs prior to the backup start

following this kb

https://kb.netapp.com/support/s/article/how-to-enable-ndmp-debugging-in-data-ontap

then as soon as backup failed collect ndmp and backup logs with mentioning what volume was backed up time of start\fail

then open a case to Netapp with provided collected data

Announcements
Register for Insight 2021 Digital

INSIGHT 2021 Digital: Meet the Specialists 2

On October 20-22, gear up for a fully digital, totally immersive virtual experience with a downright legendary lineup of world-renowned specialists. Tune in for visionary conversations, solution deep dives, technical sessions and more.

NetApp on Discord Image

We're on Discord, are you?

Live Chat, Watch Parties, and More!

Explore Banner

Meet Explore, NetApp’s digital sales platform

Engage digitally throughout the sales process, from product discovery to configuration, and handle all your post-purchase needs.

NetApp Insights to Action
I2A Banner
Public