Posting on behalf of a customer who has been testing the Altavault-v trial VM running on Hyper-v. (2012 R2 version), here is his account of the problem. Any customer defining data has been altered.
I came in this morning, and the services on the altavault were stopped. I looked through the logs and found this:
Sep 9 07:43:20 MyAltaVault001 kernel:end_request: I/O error, dev sdb, sector 18849928
Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356225
Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1
Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356226
Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1
Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356227
Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1
Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356228
Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1
Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356229
Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1
Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356230
Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1
Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356231
Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1
Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356232
Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1
Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356233
Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1
Sep 9 07:43:20 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356234
Sep 9 07:43:20 MyAltaVault001 kernel:lost page write due to I/O error on sdb1
Sep 9 07:43:20 MyAltaVault001 kernel:JBD2: Detected IO errors while flushing file data on sdb1-8
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.ERR] (50834) Failed to open database /data/db/replicator: IO error: /data/db/replicator/MANIFEST-000020: Input/output error
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.ERR] (50834) failed to initialize replicator DB: Generic fatal error
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.ERR] (50834) Failed to setup replication db: Generic fatal error
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.ERR] (50834) Mandatory initialization failed: Generic fatal error
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.ERR] (50834) Failed to initialize the Megastore: Generic fatal error
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.ERR] (50834) Error in starting megastore for the volume rfsd_volume
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Shutting down replication manager thread
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Shutting down backend threads
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Shutting down replicator
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Shutting down all replication worker threads
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Processor replicator shutting down 20 threads
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Completed shutdown of replication worker threads
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) Replicator shutdown complete
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [replicator.INFO] (50834) ReplicationManager shutdown complete
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [mutable_slab_manager.INFO] (50834) releasing partial slabs
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.INFO] (50834) Transaction Statistics for Transaction Manager rfsd_volume
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.INFO] (50834) Log Records Per Transaction: Samples: 0 Cum. Total: 0 Min: 0 Max: 0 Mean: -nan Median: 0 Median Err: 5
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megastore.INFO] (50834) Payload Bytes Logged Per Transaction: Samples: 0 Cum. Total: 0 Min: 0 Max: 0 Mean: -nan Median: 0 Median Err: 5
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [megamount.ERR] (50834) Failed to start megastore: Generic fatal error
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [rfsd.ERR] (50834) Megamount initialization failed.
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [rfsd.INFO] (50832) tearing down RfsContext
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [rfsd.INFO] (50832) Megamount not running
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [rfsd.INFO] (50832) Shutting down backend threads
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [rfsd.INFO] (50832) Shutting down MegastoreGuid threads
Sep 9 07:43:20 MyAltaVault001 rfsd[50832]: [mgmt/mgmtd.NOTICE] (50832) rfsd sent event to mgmtd: /rbt/rfsd/events/notready
Sep 9 07:43:20 MyAltaVault001 mgmtd[3667]: [mgmtd.INFO]: EVENT: /rbt/rfsd/events/notready
Sep 9 07:43:20 MyAltaVault001 mgmtd[3667]: [mgmtd.INFO]: in rfsd_notup
Sep 9 07:43:20 MyAltaVault001 mgmtd[3667]: [mgmtd.ERR]: Error no message binding from rfsd.
After rebooting the VM
Sep 9 07:49:03 MyAltaVault001 kernel:end_request: I/O error, dev sdb, sector 18854016
Sep 9 07:49:03 MyAltaVault001 kernel:Buffer I/O error on device sdb1, logical block 2356736
Sep 9 07:49:03 MyAltaVault001 kernel:lost page write due to I/O error on sdb1
Sep 9 07:49:03 MyAltaVault001 kernel:JBD2: Detected IO errors while flushing file data on sdb1-8
I checked out the hyper-v server, and I see something with the VM failed last night, and it rebooted. I also see this message on the hyper-v servers:
A storage device in 'NetappAltaVault' loaded but has a different version from the server. Server version 6.0 Client version 4.2 (Virtual machine ID XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX). The device will work, but this is an unsupported configuration. This means that technical support will not be provided until this problem is resolved. To fix this problem, upgrade the integration services. To upgrade, connect to the virtual machine and select Insert Integration Services Setup Disk from the Action menu.
Can anyone assist in getting this back up and running?
