Subscribe

Data store error running Altavault-v trial in Hyper-v

Posting on behalf of a customer who has been testing the Altavault-v trial VM running on Hyper-v. (2012 R2 version), here is his account of the problem.  Any customer defining data has been altered.

 

I came in this morning, and the services on the altavault were stopped. I looked through the logs and found this:

 

Sep 9 07:43:20 MyAltaVault001 kernel:end_request: I/O error, dev sdb, sector 18849928

Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356225

Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1

Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356226

Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1

Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356227

Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1

Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356228

Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1

Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356229

Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1

Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356230

Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1

Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356231

Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1

Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356232

Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1

Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356233

Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1

Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356234

Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1

Sep 9 07:43:20 MyAltaVault001 kernel:JBD2: Detected IO errors while flushing file data on sdb1-8

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.ERR] (50834) Failed to open database /data/db/replicator: IO error: /data/db/replicator/MANIFEST-000020: Input/output error

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.ERR] (50834) failed to initialize replicator DB: Generic fatal error

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.ERR] (50834) Failed to setup replication db: Generic fatal error

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.ERR] (50834) Mandatory initialization failed: Generic fatal error

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.ERR] (50834) Failed to initialize the Megastore: Generic fatal error

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.ERR] (50834) Error in starting megastore for the volume rfsd_volume

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Shutting down replication manager thread

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Shutting down backend threads

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Shutting down replicator

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Shutting down all replication worker threads

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Processor replicator shutting down 20 threads

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Completed shutdown of replication worker threads

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Replicator shutdown complete

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) ReplicationManager shutdown complete

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [mutable_slab_manager.INFO] (50834) releasing partial slabs

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.INFO] (50834) Transaction Statistics for Transaction Manager rfsd_volume

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.INFO] (50834) Log Records Per Transaction: Samples: 0 Cum. Total: 0 Min: 0 Max: 0 Mean: -nan Median: 0 Median Err: 5

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.INFO] (50834) Payload Bytes Logged Per Transaction: Samples: 0 Cum. Total: 0 Min: 0 Max: 0 Mean: -nan Median: 0 Median Err: 5

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megamount.ERR] (50834) Failed to start megastore: Generic fatal error

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [rfsd.ERR] (50834) Megamount initialization failed.

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [rfsd.INFO] (50832) tearing down RfsContext

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [rfsd.INFO] (50832) Megamount not running

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [rfsd.INFO] (50832) Shutting down backend threads

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [rfsd.INFO] (50832) Shutting down MegastoreGuid threads

Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [mgmt/mgmtd.NOTICE] (50832) rfsd sent event to mgmtd: /rbt/rfsd/events/notready

Sep 9 07:43:20 MyAltaVault001 mgmtd[3667]: [mgmtd.INFO]: EVENT: /rbt/rfsd/events/notready

Sep 9 07:43:20 MyAltaVault001 mgmtd[3667]: [mgmtd.INFO]: in rfsd_notup

Sep 9 07:43:20 MyAltaVault001 mgmtd[3667]: [mgmtd.ERR]: Error no message binding from rfsd.

 

 

 

After rebooting the VM

 

Sep 9 07:49:03 MyAltaVault001 kernel:end_request: I/O error, dev sdb, sector 18854016

Sep 9 07:49:03 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356736

Sep 9 07:49:03 MyAltaVault001 kernel:lost page write due to I/O error on sdb1

Sep 9 07:49:03 MyAltaVault001 kernel:JBD2: Detected IO errors while flushing file data on sdb1-8

 

 

 

I checked out the hyper-v server, and I see something with the VM failed last night, and it rebooted.   I also see this message on the hyper-v servers:

A storage device in 'NetappAltaVault' loaded but has a different version from the server.  Server version 6.0  Client version 4.2 (Virtual machine ID XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX). The device will work, but this is an unsupported configuration. This means that technical support will not be provided until this problem is resolved. To fix this problem, upgrade the integration services. To upgrade, connect to the virtual machine and select Insert Integration Services Setup Disk from the Action menu.

 

Can anyone assist in getting this back up and running?

 

Re: Data store error running Altavault-v trial in Hyper-v

Please advise why there is an issue with hyperV

Re: Data store error running Altavault-v trial in Hyper-v

Tony, There is no similarity between your comment and the above discussion, I've replied to you separately via email. In the future, please open new discussion threads to ensure that your issue is not incorrectly diagnosed as someone else's problem. Thanks!

 

Regards,

Christopher