Solved: Can't attach a Volume to a VM

pravinp · ‎2014-05-26

Have configured the c-mode with drivers for VM.

Ubuntu with Icehouse

From GUI and Command i am able see the volume attach working without error when i check from the vm i do not see any attached volume to the vms.

I believe this is nothing to do with cinder as this api is more related nova, just wanted to know if anyone is aware of this kida problem.

root@controller:~# nova volume-attach 4c9207c9-7888-43b6-b724-4ca45b33adef 81d3c673-a82b-432f-954b-8dd36e4ed73b

+----------+--------------------------------------+

| Property | Value |

+----------+--------------------------------------+

| device | /dev/vdb |

| id | 81d3c673-a82b-432f-954b-8dd36e4ed73b |

| serverId | 4c9207c9-7888-43b6-b724-4ca45b33adef |

| volumeId | 81d3c673-a82b-432f-954b-8dd36e4ed73b |

+----------+--------------------------------------+

pravinp · ‎2014-05-27

Fixed.

I am using separate nodes for all compute an block. I had to install nfs-common on the compute node which i did not.

The logs are as below on Compute node with while attaching the volume:

This gave me a clue why it was failing

Stderr: 'mount: wrong fs type, bad option, bad superblock on 10.238.229.43:/open_cinder,\n

missing codepage or helper program, or other error\n

(for several filesystems (e.g. nfs, cifs) you might\n

need a /sbin/mount.<type> helper program)\n

In some cases useful info is found in syslog - try\n

dmesg | tail or so\n\n'

3a931-b10f-4465-9bdb-b0a3893bd1a6] Attaching volume 377cccd6-b1ef-451d-909d-6dee4f47756b to /dev/vdb

2014-05-27 13:51:33.771 27773 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): controller

2014-05-27 13:51:34.028 27773 WARNING nova.virt.libvirt.utils [req-0aa458ca-9471-4ead-b333-4bd5d906ecf6 4d2a100c10b04586bd682508f18c4fce bd5d1cb84a0546bb84cb3aad69b9700d] systool is not installed

2014-05-27 13:51:34.098 27773 WARNING nova.virt.libvirt.utils [req-0aa458ca-9471-4ead-b333-4bd5d906ecf6 4d2a100c10b04586bd682508f18c4fce bd5d1cb84a0546bb84cb3aad69b9700d] systool is not installed

2014-05-27 13:51:34.101 27773 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): controller

2014-05-27 13:51:34.342 27773 ERROR nova.virt.block_device [req-0aa458ca-9471-4ead-b333-4bd5d906ecf6 4d2a100c10b04586bd682508f18c4fce bd5d1cb84a0546bb84cb3aad69b9700d] [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] Driver failed to attach volume 377cccd6-b1ef-451d-909d-6dee4f47756b at /dev/vdb

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] Traceback (most recent call last):

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 239, in attach

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] device_type=self['device_type'], encryption=encryption)

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1224, in attach_volume

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] disk_info)

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1183, in volume_driver_method

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] return method(connection_info, *args, **kwargs)

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 660, in connect_volume

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] path = self._ensure_mounted(connection_info['data']['export'], options)

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 689, in _ensure_mounted

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] self._mount_nfs(mount_path, nfs_export, options, ensure=True)

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 705, in _mount_nfs

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] utils.execute(*nfs_cmd, run_as_root=True)

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 164, in execute

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] return processutils.execute(*cmd, **kwargs)

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py", line 193, in execute

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] cmd=' '.join(cmd))

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] ProcessExecutionError: Unexpected error while running command.

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount -t nfs 10.238.229.43:/open_cinder /var/lib/nova/mnt/4a31c2a3745485c719e0a63c6c1ab5f7

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] Exit code: 32

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] Stdout: ''

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] Stderr: 'mount: wrong fs type, bad option, bad superblock on 10.238.229.43:/open_cinder,\n missing codepage or helper program, or other error\n (for several filesystems (e.g. nfs, cifs) you might\n need a /sbin/mount.<type> helper program)\n In some cases useful info is found in syslog - try\n dmesg | tail or so\n\n'

2014-05-27 13:51:34.342 27773 TRACE nova.virt.block_device [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6]

2014-05-27 13:51:34.346 27773 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): controller

2014-05-27 13:51:34.472 27773 ERROR nova.compute.manager [req-0aa458ca-9471-4ead-b333-4bd5d906ecf6 4d2a100c10b04586bd682508f18c4fce bd5d1cb84a0546bb84cb3aad69b9700d] [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] Failed to attach 377cccd6-b1ef-451d-909d-6dee4f47756b at /dev/vdb

2014-05-27 13:51:34.472 27773 TRACE nova.compute.manager [instance: 5273a931-b10f-4465-9bdb-b0a3893bd1a6] Traceback (most recent call last):

View solution in original post

akerr · ‎2014-05-26

Hi pravinp,

I have seen this type of behavior before. For me, I had my data lif set to accept the wrong protocol (I was using the iSCSI driver, but my lif only allowed nfs). Unfortunately, an unsuccessful attach doesn't usually leave any user-facing traces, other than the volume still being in "available" state, and nova not showing the attachment. You should check the cinder volume and nova logs for errors during the attach process to try and figure out what has gone wrong.

pravinp · ‎2014-05-27