Suggestions about how to use SnapCreator to backup instances in an OpenStack cloud?

[pierrek]

Dear All,

We have a customer that would like to schedule snapshot based backups of VM (instances) running on NFS volumes exported from a vserver. The setup is quite simple, with 1 NFS volume used for storing the nova instances running on KVM compute nodes and 1 NFS volume storing Cinder block volumes for those instances.

The idea is to start with a very simple approach, without application integration (so no SC agent needed inside instances), just a SC server scheduling the NetApp snapshots on the vserver. But we would like to synchronize the execution of this NetApp snapshot with the execution of a libvirt/KVM snapshot, probably by executing a script (via the SC agent installed somewhere in the infra) that will take those snapshots. Can someone help me here with suggestions or examples for this?

Thanks a lot in advance

Pierre

[tlichten]

Pierre,

in last week's 2-day Snap Creator Innovation challenge I did a proof-of-concept integration of OpenStack and Snap Creator. In the previous link you will find a screencast under Team2 called SnapStack which demos the integration which may be a helpful suggestion to you.

The setup was slightly different but also similar to yours. It was using an OpenStack installation with VMs running within libvirt/KVM. On the VMs an agent was running that quiesced a database to show application integration, then an agent with the proof-of-concept SnapStack plug-in on the OpenStack server was suspending the VM and taking OpenStack instance volume snapshots to demo snapshot based backups of VMs. The Cinder block volumes were stored on a VServer using the NetApp iSCSI driver. Each OpenStack volume and OpenStack snapshot was mapped as a LUN path under one VServer volume. Finally, Snap Creator was used to schedule and perform quiescing the application, taking OpenStack snapshots, and snapshotting the VServer volume. Within the 2-day time frame I could only get a very basic proof-of-concept restore running: Using the Snap Creator's single file restore capability, I used the VServer volume snapshot, selected the OpenStack instance volume snapshot to be restored, and restored it to the OpenStack instance volume. The agent on the OpenStack server was shutting down the VM before restoring the instance volume.

So, all in all, I just needed to write an agent plug-in that was running on the OpenStack server and which interfaces with OpenStack's block storage and compute APIs. I could then use Snap Creator to schedule backups and make use of the restore capability.

I hope this can give you a suggestion of a possible integration. Please note, this project is NOT scheduled for inclusion in a specific release. It was part of the aforementioned innovation challenge and is simply an example of a proof-of-concept integration that was developed in a short period of time and is meant to highlight the capabilities of the Snap Creator Framework.

I'm happy to answer any questions you may have.

Regards,

Thomas

[pierrek]

Hi Thomas,

Thanks for this. So far for some reason I failed viewing the full video, I will watch it later. But so you say that you made a plugin that interacts with the OpenStack controller/dashboard and automates the creation of instance snapshots? I understand that an agent is installed inside the instance for application consistency of the data on the cinder volume (vda disk). But let's say that this requirement is not needed, how would you proceed? My simple view of the process:

SC server starts the execution of a job:

--> an agent running on the OpenStack controller calls the nova command that takes the snapshot of a set of Instances

first question: this is a qcow2 2 snapshot I guess? Where is it stored? Does it make sense to have a snapshot of the ephemeral storage of a running instance?

second question: does this snapshot also work for the vda disk provided by cinder?

--> When the nova snapshot is made, we freeze the state of the instances in a NetApp snapshot (on both the cinder NFS volume and the volume used to store instance ephemeral storage).

--> Finally the nova snapshot is removed using a command executed by the SC agent on the OpenStack controller.

This approach is kind of inspired by what the VSC does in a VMWare context, but is it valid? Even conceptually only? What do you think?

Regards

Pierre

[tlichten]

Hi Pierre

pierrek wrote:
--> an agent running on the OpenStack controller calls the nova command that takes the snapshot of a set of Instances
first question: this is a qcow2 2 snapshot I guess? Where is it stored? Does it make sense to have a snapshot of the ephemeral storage of a running instance?

For the OpenStack installation I used devstack and went mostly with the default configuration to get running quickly (2 day time limit for the challenge). I only configured cinder with the iSCSI driver to use a NetApp volume for cinder volume storage. IIRC per this configuration the instance snapshots were stored within the file system the installation was running on, from what I remember under /opt/openstack/nova/instances/. IIRC, those snapshots were in qcow2 format. I'm not very knowledgeable about the correct way to consistently backup VMs, so I can't comment whether it makes sense to snapshot the ephemeral storage. In my plug-in and scenario I didn't take instance snapshots since in my setup they were not stored on the NetApp storage. But I don't see any reason why this shouldn't work, it should mostly be another yet similar API call.

pierrek wrote:
second question: does this snapshot also work for the vda disk provided by cinder?

This is actually what I did. I suspended the VMs in OpenStack which had a cinder volume attached. I then took cinder volume snapshots of all those attached and also not attached volumes. I don't claim this is the right way to do snapshot based VM backups, there is probably more to it like sync'ing and freezing the file system, but in my controlled environment this approach was good enough; remember: 2 day time limit for the challenge.

pierrek wrote:
--> When the nova snapshot is made, we freeze the state of the instances in a NetApp snapshot (on both the cinder NFS volume and the volume used to store instance ephemeral storage).
--> Finally the nova snapshot is removed using a command executed by the SC agent on the OpenStack controller.
This approach is kind of inspired by what the VSC does in a VMWare context, but is it valid? Even conceptually only? What do you think?

Again, I can't comment on whether this is a valid approach for consistent snapshotting, but from what I learned while doing the proof-of-concept it should be reasonably straight forward to create an agent plug-in that orchestrates the needed API calls required for the outlined approach. How would a restore be orchestrated?

If you plan to work on this I would be very interested to collaborate.

Thanks,

Thomas

[pierrek]

Thomas,

On Monday I will meet the customer and will have a better view of what exactly are his expectations. For sure if you allow me I'll come back to you with probably more questions .

Thanks a lot for your time so far.

Kind regards

Pierre

[tlichten]

Sure, pleasure.

[OLIVER_LIENHARD]

Hello

Is there any feature development on this or on another snap plugin for openstack?

Thank you

[steiner]

It's still being discussed, but overall there are no plans to add this directly to SnapCreator in its current form. 

I was looking back at the thread, and it looks like you were thinking backups in the context of the boot LUNs. Is that correct, or are you thinking of backups of other data types in an OpenStack cloud?

[OLIVER_LIENHARD]

Hello
Thank you.
We run openstack on NFS shares.
We have enabled default snapshots on the filer.
This is almost okay, but for some database and app servers it isn't consistent.
I was looking more for a storage driver like plugin to integrate it to the openstack dashboard.
So customers can do there own backup shedules, maybe in conjunction with snap manager.

[steiner]

I've been talking to the OpenStack team about this for a few months now, and it seems to me like OpenStack isn't quite ready for this, but it's going to need to be resolved somehow. It wouldn't be too diffcult to add a tab to OpenStack for backups and then figure out a way to invoke a script to do a backup, but that would mean a customized approach just for one customer. 

What really needs to happen is OpenStack needs to have a backup API where you can tell a Cinder, Nova, or Manila storage resource to "backup yourself up" and then it calls an outside progam. That might be SnapCreator, SnapManager for Oracle, or even something like CommVault. I'm sure it will happen eventually, but I'd guess it's a year out before standards are really established.

In the interim, the best plan is probably to configure something like SnapCreator to work alongside OpenStack. The wouldn't integrate, but they wouldn't interfere with one another either.