Vibe error

[itinfrastructure]

Hi all,

we are using vibe to snapshot the virtual devices on a ESX 3.5 host. at the beginning it worked fine, but now i get some errors i dont know what that means.

following errors i get.

WARNING: ZAPI call snapshot-create failed: Snapshot operation failed: No space left on device.

Checking if snapshot Citrix"C"partiton.7 exists on XXXXX on storage appliance XXX.XXX.XXX.XXX ...
[08:11:36] Snapshot Citrix"C"partiton.7 on XXXXXXX not found.

Exiting with return code: 1

Does anyone of you what i have to do ?

Thanks for your support

[erichpopp]

Hi,

please check following topics:

- volume size on netapp side

- volume fill rate

- snapshot reserve

- lun size (if iscsi or fcp)

with VIBE there will be netapp volume snapshots created and you you need some free blocks on your volume (while creating) and some additional blocks (while snapshot is available -> depends on your change rate).

similar things apply for VMs on VMFS3:

with VIBE there will be created a vmware snapshot for each disk (not in independent mode). each disk will start with a 16 MB sized delta file and each change area above 16 MB on that disk will increase the corresponding delta file with additional 16 MB. So, you need some free space on a VMFS3 file system.

I believe that you do not have free space for netapp volume snapshots

regards

Erich

[itinfrastructure]

Hi Erich,

unfortunatly i cant finde any low space. on the volume there is only 53% taken space. on the datastore i have 86.1 GB free space. bellow is the error i get. Hmmm strange thing...

Thanks

Running ZAPI snapshot-create 'volume' 'vc_citrix_os' 'snapshot' 'Citrix"C"partiton.recent' ...
WARNING: ZAPI call snapshot-create failed: Snapshot operation failed: No space left on device.
Removing oldest snapshot with ZAPI ...

Snapshot Citrix"C"partiton.7 on vc_citrix_os not found.
Oldest snapshot Citrix"C"partiton.7 does not exist on vc_citrix_os.
Rotating snapshots with ZAPI ...
Rotating snapshots named Citrix"C"partiton.# on vc_citrix_os ...
Checking if snapshot Citrix"C"partiton.6 exists on vc_citrix_os on storage appliance xxx.xxx.xxx .
..
Snapshot Citrix"C"partiton.6 on vc_citrix_os not found.
Checking if snapshot Citrix"C"partiton.5 exists on vc_citrix_os on storage appliance xxx.xxx.xxx
..
Snapshot Citrix"C"partiton.5 on vc_citrix_os not found.
Checking if snapshot Citrix"C"partiton.4 exists on vc_citrix_os on storage appliance xxx.xxx.xxx .
..
Snapshot Citrix"C"partiton.4 on vc_citrix_os not found.
Checking if snapshot Citrix"C"partiton.3 exists on vc_citrix_os on storage appliance xxx.xxx.xxx .
..
Snapshot Citrix"C"partiton.3 on vc_citrix_os not found.
Checking if snapshot Citrix"C"partiton.2 exists on vc_citrix_os on storage appliance xxx.xxx.xxx .
..
Snapshot Citrix"C"partiton.2 on vc_citrix_os not found.
Checking if snapshot Citrix"C"partiton.1 exists on vc_citrix_os on storage appliance xxx.xxx.xxx .
..
Snapshot Citrix"C"partiton.1 on vc_citrix_os not found.
Checking if snapshot Citrix"C"partiton.0 exists on vc_citrix_os on storage appliance xxx.xxx.xxx .
..
Snapshot Citrix"C"partiton.0 on vc_citrix_os not found.
Removing oldest snapshot with ZAPI ...
Checking if snapshot Citrix"C"partiton.7 exists on vc_citrix_os on storage appliance xxx.xxx.xxx .
..
Snapshot Citrix"C"partiton.7 on vc_citrix_os not found.
Oldest snapshot Citrix"C"partiton.7 does not exist on vc_citrix_os.
Renaming most recent snapshot with ZAPI ...
Renaming snapshot Citrix"C"partiton.recent to Citrix"C"partiton.0 for vc_citrix_os on storage applia
nce xxx.xxx.xxx ...
Checking if snapshot Citrix"C"partiton.recent exists on vc_citrix_os on storage appliance xxx.xxx.xxx

WARNING: Recent snapshot Citrix"C"partiton.recent does not exist!
...

[rmatt]

I think the problem in this case is the datastore name. Using the double-quotes in the datastore name may be causing the conflict. Does this work with other datastore names?

--Matt

[rmatt]

There are two things:

1) Check your /etc/messages logs on the NetApp storage controller to verify that snapshots for the volume aren't failing;

2) Test snapshots with 'snap create <volname> test' to see if you can take a snapshot.

If (2) fails, it's the same thing that VIBE is coming back with.

The ZAPI call you see below is essentially the same as creating a manual snapshot, and if you can't do that, that's the problem. It's almost certain to be the issue. If you use 'vol autosize' or a similar function it would allow the volume container to grow in advance of the snapshot.

One other point; if you are consuming a lot of snapshot space, you might want to make sure you aren't swapping within your VMs, which is going to blow up the amount of space needed for snapshots.

Thanks,

--Matt