Data Backup and Recovery

Snap Creator 4.1P1 + VIBE plugin and automatically modified global.conf

filipsneppe
4,418 Views

For a reason I don't understand it would appear Snap Creator has modified the global.conf file when simply running a scheduled backup job, and deleted all my cDOT SVM/vserver credentials.

I can only go by the timestamp of the global.conf file, which was modified at 2:30am which corresponds to a time when two backup jobs are run.

The only recent changes on this system are an upgrade of the SC agent on the vcenter server from 4.1.0c to 4.1P1, and the occasional creation, scheduling or modification of backup jobs.

This happened for the first time 3-4 days ago and the timestamp of the file was 1:30am, and because global.conf didn't contain any storage credentials anymore, all backups have been failing since. I've recreated the global.conf today (yesterday actually, technically) and ran a couple of test backups during the day, only to find out that the file has now been modified at 2:30am.

This is the backup log of a job ran at 2:30am. As you can see the credentials for vs1_backup_a70_dr are present at the beginning of the backup, then there are some lines about updating the credentials:

[2014-08-12 02:30:52,685] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:24 2014) Updating NTAP_USERS storage controller/vserver vs1_backup_a70_dr with IP address 172.30.103.78 ...

and then later on Snap Creator failed to find the credentials for the storage SVM/vserver:

[2014-08-12 02:30:53,241] ERROR: Credentials for storage controller vs1_backup_a70_dr not found, please check NTAP_USERS parameter in configuration

What's going on here ?

Here is the full log:

[2014-08-12 02:30:00,518] INFO: Validating policy: daily finished successfully

########## Detecting Data ONTAP mode for vs1_prod_a70_nfs ##########

[2014-08-12 02:30:00,997] INFO: STORAGE-03031: Getting system version details of [vs1_prod_f91_nfs]

[2014-08-12 02:30:00,997] INFO: STORAGE-03032: Getting system version details of [vs1_prod_f91_nfs] finished successfully.

[2014-08-12 02:30:01,469] INFO: STORAGE-03031: Getting system version details of [vs1_prod_a70_nfs]

[2014-08-12 02:30:01,469] INFO: STORAGE-03032: Getting system version details of [vs1_prod_a70_nfs] finished successfully.

[2014-08-12 02:30:01,719] INFO: STORAGE-03031: Getting system version details of [vs1_backup_a70_dr]

[2014-08-12 02:30:01,719] INFO: STORAGE-03032: Getting system version details of [vs1_backup_a70_dr] finished successfully.

[2014-08-12 02:30:02,222] INFO: STORAGE-03031: Getting system version details of [vs3_prod_f91_cifs]

[2014-08-12 02:30:02,222] INFO: STORAGE-03032: Getting system version details of [vs3_prod_f91_cifs] finished successfully.

########## Agent validation ##########

[2014-08-12 02:30:02,334] INFO: Agent validation completed successfully for agent vcenter51:9091

########## Plugin validation ##########

[2014-08-12 02:30:02,348] INFO: Plugin validation completed successfully for plugin vibe

########## Application Discovery ##########

[2014-08-12 02:30:02,451] INFO: Performing discovery on : vibe

[2014-08-12 02:30:52,684] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:15 2014) Checking if VIBE pre-requisites already run (function: discover) ....

[2014-08-12 02:30:52,684] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:15 2014) Executing VIBE configuration.

[2014-08-12 02:30:52,684] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:15 2014) Initializing connectivity to vCenter(s) and storage controller(s)/vserver(s) ...

[2014-08-12 02:30:52,685] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:19 2014) Updating NTAP_USERS storage controller/vserver vs1_prod_f91_nfs with IP address 172.30.103.65 ...

[2014-08-12 02:30:52,685] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:24 2014) Updating NTAP_USERS storage controller/vserver vs1_prod_a70_nfs with IP address 172.30.103.55 ...

[2014-08-12 02:30:52,685] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:24 2014) Updating NTAP_USERS storage controller/vserver vs1_backup_a70_dr with IP address 172.30.103.78 ...

[2014-08-12 02:30:52,685] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:25 2014) Updating NTAP_USERS storage controller/vserver vs3_prod_f91_cifs with IP address 172.30.103.67 ...

[2014-08-12 02:30:52,686] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:27 2014) Collecting list of VirtualMachine objects planned for snapshots ...

[2014-08-12 02:30:52,686] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:46 2014) Updating VirtualMachine objects planned for snapshots ...

[2014-08-12 02:30:52,686] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:46 2014) Collection of VirtualMachine objects successful.

[2014-08-12 02:30:52,686] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:47 2014) Collecting VM information on NFS Datastore prod_a70_nfs_sas_ds02.

[2014-08-12 02:30:52,687] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:48 2014) Setting VOLUMES configuration variable for Snap Creator to "vs1_prod_a70_nfs:vs1_prod_a70_nfs_sas_ds02".

[2014-08-12 02:30:52,687] INFO: [vcenter51:9091 (4.1.1.1)] (Tue Aug 12 02:30:48 2014) Setting SNAPVAULT_VOLUMES configuration variable for Snap Creator to "vs1_prod_a70_nfs:vs1_prod_a70_nfs_sas_ds02".

[2014-08-12 02:30:52,751] INFO: Updating configuration parameter VOLUMES from plugin vibe

[2014-08-12 02:30:52,751] INFO: Updating configuration parameter SNAPVAULT_VOLUMES from plugin vibe

########## Application Discovery finished successfully ##########

########## Running storage discovery ##########

[2014-08-12 02:30:53,240] INFO: STORAGE-02113: Retrieving SnapVault relationships

[2014-08-12 02:30:53,240] INFO: STORAGE-02114: Retrieving SnapVault relationships on controller [vs1_prod_a70_nfs] finished successfully

[2014-08-12 02:30:53,241] ERROR: Credentials for storage controller vs1_backup_a70_dr not found, please check NTAP_USERS parameter in configuration

########## Application cleanup ##########

[2014-08-12 02:30:53,331] INFO: Performing cleanup on : vibe

########## Application cleanup finished successfully ##########

########## Agent Workflow Finalization ##########

[2014-08-12 02:30:58,424] INFO: Agent Workflow Finalization started

[2014-08-12 02:31:03,593] INFO: [vcenter51:9091 (4.1.1.1)] Finalized workflow with id 1880

[2014-08-12 02:31:03,593] INFO: Agent Workflow Finalization finished successfully

########## Snap Creator Framework 4.1P1 failed ##########

[2014-08-12 02:31:03,604] INFO: Pre Exit commands are not defined. Skipping !

5 REPLIES 5

filipsneppe
4,396 Views

The only thing I can add at this point is that it looks like it's related to the use of the VMware plugin in Snap Creator. Doesn't happen when I don't use the VMware plugin ...

marcel_juhnke
4,093 Views

Hi,

 

I observe the same with SC 4.1.1P2. Every time a VIBE backup is run, global.conf is touched, and sometimes loses all credentials, causing all further backups to fail.

 

 

spinks
4,088 Views

Have you filed a case on this?

 

If so can you please share the case number?

You can contact me via email if you prefer not to share it here.

 

Snap Creator should not be editing your global config file, much less removing entries.

 

If there is a case I want to bring it up and be sure it is looked at closely.

 

Thanks,

 

John

marcel_juhnke
4,084 Views

Hi John,

 

I have not opened a case, but if this helps, I can do it of course.

 

Seems like the vibe auto-discovery injects the Filer IP again into the NTAP_USERS field. Unfortunately, it does so in the global.conf file that is being used.

 

The actual problem with the credentials going MIA I suspect is that we are running multiple vibe configs at the same schedule. If they all try to write global.conf at the same time, things only can go wrong.

 

For the time being, we just use normal volume snapshots instead of the vibe plugin, as we don't use VMware snapshots, so it doesn't make much difference.

 

[2015-02-06 10:43:36,249] INFO: Performing discovery on : vibe
[2015-02-06 10:44:06,484] INFO: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:45 2015) Checking if VIBE pre-requisites already run (function: discover) ....
[2015-02-06 10:44:06,484] INFO: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:45 2015) Executing VIBE configuration.
[2015-02-06 10:44:06,484] INFO: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:45 2015) Initializing connectivity to vCenter(s) and storage controller(s)/vserver(s) ...
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:45 2015) Converting vCenter hostname(s) to IP address(es) ...
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:45 2015) Attempting to ping vCenter server 192.168.11.123 ...
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:45 2015) Ping of vCenter server successful.
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:45 2015) Logging into vCenter server 192.168.11.123 ...
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:49 2015) vCenter server login successful.
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:49 2015) Updating vCenter name/object for VM  ...
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:49 2015) Updating vCenter name/object for VM  ...
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:49 2015) Attempting to ping storage controller/vserver FRCSD9101EAM ...
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:49 2015) Ping of storage controller/vserver FRCSD9101EAM successful.
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:49 2015) Attempting to ping storage controller/vserver FRCSD9201EAM ...
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:49 2015) Ping of storage controller/vserver FRCSD9201EAM successful.
[2015-02-06 10:44:06,484] INFO: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:49 2015) Updating NTAP_USERS storage controller/vserver FRCSD9101EAM with IP address 192.168.11.4 ...
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:49 2015) Creating login to storage controller/vserver 192.168.11.4 ...
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:49 2015) Setting transport type for storage controller/vserver 192.168.11.4 to HTTPS.
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:54 2015) Setting transport port for storage controller/vserver 192.168.11.4 to 443.
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:54 2015) 7-Mode ONTAP detected for 192.168.11.4.
[2015-02-06 10:44:06,484] INFO: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:54 2015) Updating NTAP_USERS storage controller/vserver FRCSD9201EAM with IP address 192.168.11.5 ...
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:54 2015) Creating login to storage controller/vserver 192.168.11.5 ...
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:54 2015) Setting transport type for storage controller/vserver 192.168.11.5 to HTTPS.
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:54 2015) Setting transport port for storage controller/vserver 192.168.11.5 to 443.
[2015-02-06 10:44:06,484] DEBUG: [192.168.11.123:9100 (4.1.1.1)] (Fri Feb  6 10:43:54 2015) 7-Mode ONTAP detected for 192.168.11.5.

 

Best regards

Marcel

filipsneppe
4,072 Views
Hi John,

Case 2005279708 is open for this, and before that there was another case for the same issue. It's been dragging on for months. Thanks for having a closer look at this.
Public