Receover an appliance after loss of one EBS volume

[CONDESTORAGEADMIN]

Hi,

We have a AltaVault 2030 hosted on AWS. We faced an issue with the AWS instance yesterday for which we had to stop and start. This took time and AWS investigated that is due to an EBS volume being unavailable. That particular EBS volume is unrecoverable now and AWS has recommended to recover that from a snapshot.

That EBS volume is of 1TB and is part of the datastore. We recovered that volume from an earlier snapshot and attached it to the instance. After that we booted the appliance and we are getting the below mentioned error.

Both megastore guid and serial number don't match.Appliance guid is c99c6fd9-917b-457b-bc1e-5446c9481a98. Megastore guid in cloud is 00000000-0000-0000-0000-000000000000. Appliance serial number is 12A6E5C79212. Serial number in cloud is

Kindly let us know how can a recovery be done in this case.

[Saroj]

Chris/ NetApp Team,

Now we are able to resolve cloud disparity error but now optimiztion service has "initialization error".

Request you to please look into this ASAP.

Sysdump logs are being uploaded, please verify and confirm back.

Thanks

[chriswong]

Hi,

Because of the loss of some of the cache contents (due to the EBS volume loss), your 2030 is no longer consistent with what the cloud data has. This requires that you do a DR. You need to recover the 2030 as if it has gone through a DR scenario sine the cache contents. Login to the CLI to issue the steps for DR (this process is explained in Ch2 of the deployment guide here: https://library.netapp.com/ecm/ecm_download_file/ECMLP2317731). Because your appliance is still the same (you're not recovering to a new appliance), the steps simplify to the following:

amnesiac > enable

amnesiac # configure terminal

amnesiac (config) # no service enable

amnesiac (config) # datastore format local

amnesiac (config) # replication recovery enable

amnesiac (config) # service enable

amnesiac (config) # show service

Once the service is up, you can prepopulate the cache back with the most recent backups (for example). The CLI or GUI can perform prepopulation. The CLI has more options for filtering what content you want to prepop, while the GUI is simplified by a checkbox selection.

Regards,

Christopher

[Saroj]

We tried above steps mentioned by you, but datastore format is not working.

Please advice further.

Regards

SACNPAWS001 (config) # no service enable
Terminating optimization service...
.
SACNPAWS001 (config) # datastore format local
Could not stat Not --- No such file or directory

The device apparently does not exist; did you specify it correctly?
Failed to clean /data.
Failed to format local datastore.
SACNPAWS001 (config) #
SACNPAWS001 (config) #
SACNPAWS001 (config) # replication recovery enable
Enable recovery? (y/N) y
SACNPAWS001 (config) # service enable
Starting optimization service...
...........................
Storage Optimization Service: initialization error
SACNPAWS001 (config) # show service
Storage Optimization Service: initialization error
SACNPAWS001 (config) #

[chriswong]

Saroj,

As mentioned in email sent separately, the datastore format command apparently is not applicable to AMI, and instead you must:

1. Shut down the AMI

2. Take note of the 8 1GB volumes (the volume names, locations)

3. Remove and recreate these 8 1GB volumes (make sure to NOT delete the two 100GB volumes)

4. Restart the AMI

5. Issue the following CLI commands:

en

conf t

no service enable

aws setup data partition

replication recovery enable

service enable

6. Perform prepopulation if applicable to refresh the contents of the cache (and to rebuild the dedupe table) with the last backups. Refer to the deployment guide and CLI guide on options for prepop in the CLI. Manuals are available on the NetApp support website.

https://library.netapp.com/ecm/ecm_download_file/ECMLP2317731

https://library.netapp.com/ecm/ecm_download_file/ECMLP2317736