NDMP snapshot Backup ERROR: BAD ENV OP

[IT_Luke]

Greetings, we have recently encountered this error on a clustered FAS 2552 running Ontap 8.2.4P6 7Mode:

ERROR: DATA: Operation terminated: backup configuration failure (2) on RPC error: ERROR: BAD ENV OP (for /vol/nfs_fas2552_01/.snapshot/smvi__nfs_fas2552_01_recent)

This error (taken from the Netapp NDMP logs folder, details follow) coincides with the reported Netbackup Error. This setup has worked fine for the last couple of years and nothing has been modified in the mean time. The only failure that happened was a Juniper firewall dieing and so the FAS got isolated from the Netbackup machine (with a 48 tape FC Arcvault). The error apparently appeared after the firewall was replaced;  the firewall configuration did not change so it apparently is unrelated. All the other snapshot backups work fine, only these 2 "_recent" ones are failing. From what I recall, after the NDMP backup has initiated immediately a snapshot is created with the postfix "_recent" added to the volume name and .snapshot path. Apparently this is not happening: by looking at the debug log after enabling it everything checks out, MD5 authentication passes, the session is created and connected but something goes wrong (internal error, STAT on filesystem failed). It is as if the snapshot is not being created, hence it doesn't exist and the STAT fails. An excerpt of the debug log follows:

DEBUG: entry='DATA: Operation terminated: backup configuration failure (2) on RPC error: ERROR: BAD ENV OP (for /vol/nfs_fas2552_01/.snapshot/smvi__nfs_fas2552_01__recent)\n'
ERROR: DATA: Operation terminated: backup configuration failure (2) on RPC error: ERROR: BAD ENV OP (for /vol/nfs_fas2552_01/.snapshot/smvi__nfs_fas2552_01__recent)
DEBUG: DATA halt on 'internal error': DATA operation failed: backup configuration failure (2) on RPC error: ERROR: BAD ENV OP (for /vol/nfs_fas2552_01/.snapshot/smvi__nfs_fas2552_01__rec

...

DEBUG: msg (DUMP: ERROR: stat on FILESYSTEM path /vol/nfs_fas2552_01/.snapshot/smvi__nfs_fas2552_01__recent failed.\n)

    Any pointers or heads up are appreciated!

   Luke

[IT_Luke]

Any pointers on this issue anyone?

Thanks!

[donny_lang]

Have you seen this KB article from Veritas (below)? Looks like it might be a good fit for your issue:

https://www.veritas.com/support/en_US/article.100012730

If you look closely at your error output, you can see the differences (note the double underscore in the path name on the second log string).

ERROR: BAD ENV OP (for /vol/nfs_fas2552_01/.snapshot/smvi__nfs_fas2552_01_recent)

DEBUG: msg (DUMP: ERROR: stat on FILESYSTEM path /vol/nfs_fas2552_01/.snapshot/smvi__nfs_fas2552_01__recent failed.\n)

[IT_Luke]

Hi Donny,

   yes I have seen it - unfortunately I tried adding the double underscores already to no avail. You see the thing is that the snapshot backup was working fine with the single underscore, nobody touched it - it just suddenly started acting up. The Veritas KB was the first thing I tried TBH even if I thought it was strange that it had worked like that and suddenly stopped all by itself after the firewall issue. I was thinking it could be related to some incomplete previous "incremental" snapshot (the policy is 1 full per month then one incremental every Friday) but even if I try running the complete one it fails the same way. I also checked the snapshot reserve or space issue but there is plenty of space (2TB vol with 600GB used). The programmed nightly snapshots are regularly taken though.

Thanks anyhow!