Steelstore Storage Optimization Service Initialization Error

[Sandy13]

We recently shut down and physically moved SteelStore running 3.2.3. The storage optimization service started, but failed to run. The error shows that it is not ready due to an initialization error. I've tried restarting the service and rebooting the appliance. Could anyone comment on this or point me to documentation that might help resolve the issue? Thanks.   

[chriswong]

Hi,

Can you tell me if the physical move also included any change in IP addresses that SteelStore uses? The service initialization process attempts to connect to the cloud (via the primary interface IP address by default) to confirm that it can re-connect to its' cloud provider/bucket. If this fails, then the service fails (which is possibly what is happening here). Confirm the cloud replication interface being used (Configure > Storage > Cloud Settings, then the bandwidth tab in the page that appears), and that this interface is IP'd properly and can reach out to the Internet.

You can check the system log (Reports > System Logs) to see the cause of the error during a service start attempt. The messages are marked in red typically, with supporting text about the error in black possibly above or below it. You can paste it in your next response.

Thanks,

Christopher

[Sandy13]

There were no changes to the IP addresses. Storage optimization is running, but it is not ready. I eventually connect to the cloud reusing an existing connection, but it still is not ready. The errors are below. I'm hoping it's just something simple I've overlooked. Thanks for your help.

Jan 14 15:48:07 WhiteWater mgmtd[18961]: [mgmtd.INFO]: EVENT: /pm/events/proc/terminate

Jan 14 15:48:07 WhiteWater rfsd[21285]: [replicator.ERR] (21617) Failed to get rbt_oids.dat from the cloud bucket Cloud

Jan 14 15:48:07 WhiteWater rfsd[21285]: [replicator.ERR] (21617) Verification of rbt_oids.dat failed between local and cloud copies

Jan 14 15:48:07 WhiteWater rfsd[21285]: [rfsd.ERR] (21617) Cloud test failed: Server is shutting down

Jan 14 15:48:07 WhiteWater rfsd[21285]: [rfsd.ERR] (21617) Cloud test failed

Jan 14 15:48:07 WhiteWater rfsd[21285]: [mgmt/mgmtd.NOTICE] (21285) rfsd sent event to mgmtd: /rbt/rfsd/events/notready

Jan 14 15:48:07 WhiteWater rfsd[21285]: [rfsd/graphite.INFO] (21620) Graphite thread exiting

Jan 14 15:48:07 WhiteWater mgmtd[18961]: [mgmtd.INFO]: EVENT: /rbt/rfsd/events/notready

Jan 14 15:48:07 WhiteWater mgmtd[18961]: [mgmtd.INFO]: in rfsd_notup

Jan 14 15:48:07 WhiteWater mgmtd[18961]: [mgmtd.ERR]: Error no message binding from rfsd.

Jan 14 15:48:07 WhiteWater rfsd[21285]: [megastore.WARN] (21285) Attempt to shut down uninitialized Megastore

Jan 14 15:48:07 WhiteWater rfsd[21285]: [rfsd.NOTICE] (21285) rfsd shut down cleanly.

Jan 14 15:48:09 WhiteWater mgmtd[18961]: [mgmtd.INFO]: EVENT: /pm/events/proc/restart

Jan 14 15:48:09 WhiteWater rfsd[10257]: [encoder.INFO] (10257) Segment size max: 32768 avg: 8192 min: 512

Jan 14 15:48:09 WhiteWater rfsd[10257]: [ctl.INFO] (10257) /etc/rfsd.conf set curl.timeout=10800

Jan 14 15:48:09 WhiteWater rfsd[10257]: [ctl.INFO] (10257) /etc/rfsd.conf set encoder.labels_per_slab_cutoff=0

Jan 14 15:48:09 WhiteWater rfsd[10257]: [ctl.INFO] (10257) /etc/rfsd.conf set encoder.max_load_anchor_slabs=12

Jan 14 15:48:09 WhiteWater rfsd[10257]: [ctl.INFO] (10257) /etc/rfsd.conf set encoder.max_pinned_anchor_slabs=12

Jan 14 15:48:09 WhiteWater rfsd[10257]: [ctl.INFO] (10257) /etc/rfsd.conf set encoder.max_slabs_for_ref=24

Jan 14 15:48:09 WhiteWater rfsd[10257]: [ctl.ERR] (10257) no such node evicter.maxpctused

Jan 14 15:48:09 WhiteWater rfsd[10257]: [ctl.INFO] (10257) delaying setting evicter.maxpctused=90: not available

Jan 14 15:48:09 WhiteWater rfsd[10257]: [ctl.INFO] (10257) /etc/rfsd.conf set gc.mode=1

Jan 14 15:48:09 WhiteWater rfsd[10257]: [ctl.ERR] (10257) no such node prepop.restore_percent_limit_per_hour

Jan 14 15:48:09 WhiteWater rfsd[10257]: [ctl.INFO] (10257) delaying setting prepop.restore_percent_limit_per_hour=5: not available

Jan 14 15:48:09 WhiteWater rfsd[10257]: [ctl.ERR] (10257) no such node replicator.paused

Jan 14 15:48:09 WhiteWater rfsd[10257]: [ctl.INFO] (10257) delaying setting replicator.paused=false: not available

[chriswong]

Hi,

Thanks for providing the logs. It appears that for some reason, the file rbt_oids.dat has become different locally than in the cloud (they should be the same). If you could email me at christopher.wong@netapp.com, I can email you the specific steps to resolve this error (assuming that you are correctly stating the only thing done to this AltaVault is a physical move and that's it).

Regards,

Christopher