After upgrade to 9.7P6 getting below errors

[Garima]

After upgrade to netapp cluster to 9.7P6 getting below err0r.

wafl.snap.create.skip.reason:(Snapshot operation not allowed due to clones backed by snapshots. Try again after sometime.)

Before upgrade, no error was generating whenever backup copy runs from our backup tool (commvault) 

[Ontapforrum]

I doubt this is due to upgrade. As this is happening during CommVault back, it is best to raise a ticket with CommVault.

You might be hitting this issue:

There is a CommVault KB below : Try increasing - snapshot retry interval.

Snapshot operation not allowed due to clones backed by snapshots:
https://documentation.commvault.com/commvault/v11/article?p=34476.htm

[Garima]

Thanks @Ontapforrum  for the document. but its not helping in our issue. Before upgrade these schedules was working fine. We are checking with Netapp too. 

[Ontapforrum]

In order to troubleshoot this issue, we need some logs.  which volume is complaining, does it contain luns ? Have you raised a ticket with CommVault Support ? The backups are triggered via commvault and therefore cvlt Job ID would contain some information regarding the failure.  The error description has a word 'clone' that means we are dealing with LUNs here, for LUNs the backup job initiates 'cloning' via NetApp  and that clone is mounted on the host and the backup is performed. How many jobs are triggered at the same time when this job fails ?  It is much easier for commvault support to analyse the logs and based on their troubleshooting they can suggest, whether there is anything that netapp support needs to look at, such as un-mounting or deleting failed busy snaps etc.

[Garima]

@Ontapforrum Yes, it contains Luns. Backups that are getting failed are Storage level. We have scheduled backup of our storage volume every hour. So whenever, backup runs at Commvault level( backup tool) at that time it skips the backup of the volume that we have scheduled at Netapp.  

And when backup completes at Commvault level after that volume level backups runs fine.

10/12/2020 01:05:00 SA0440001-02 ERROR wafl.snap.create.skip.reason: Volume  7c4eaec3-c8c7-11e4-81ad-00a0987f0e7a: Skipping creation of hourly.2020-10-12_0105 Snapshot copy (Snapshot operation not allowed due to clones backed by snapshots. Try again after sometime.).
10/12/2020 00:10:00 SA0440001-02 ERROR wafl.snap.create.skip.reason: Volume  7c4eaec3-c8c7-11e4-81ad-00a0987f0e7a: Skipping creation of daily.2020-10-12_0010 Snapshot copy (Snapshot operation not allowed due to clones backed by snapshots. Try again after sometime.).

[Ontapforrum]

Ok. I see your point. So, basically whenever CVLT backups co-insides with hourly netapp scheduled snapshots, this situation occurs. I am curious to know how is CVLT doing backups. Could you give the output of these commands.

::> license show

Also, could you give the output of snapshot show command for the concerned volumes.
::> snapshot show -vserver <vserver> -volume <concerned volume>

and this:

::> event log show -node cluster-node  -message-name wafl.snap.create*

[Garima]

@Ontapforrum  we have checked and found these errors are occurring from very long time but its not getting displayed in event log show as  By default, the command displays EMERGENCY, ALERT and ERROR severity level events . And the error was of "NOTICE" severity. After the ontap upgrade to 9.7 netapp changed this alert severity  to ERROR from NOTICE. Thats why its starts to coming in the events. 

[Ontapforrum]

Please give us the output of the requested commands in the last response. If you are unable to collect the output, please open a phone support ticket with NetApp, this will allow engineer to take a look at your setup.