NetApp OpenStack Deployment and Operations Guide 3.0 (Icehouse)

by rcallawa Former NetApp Employee on ‎2014-06-03 07:56 AM

The NetApp OpenStack Deployment and Operations Guide version 3.0.  This version focuses on OpenStack Block Storage (Cinder), OpenStack Image (Glance), OpenStack Object Storage (Swift) and aspects of OpenStack Compute (Nova) deployment. As NetApp's File Share Service (the OpenStack Manila development project) is still under development and not yet integrated into OpenStack it is not covered in this version of the guide.  This version applies to the Icehouse release of OpenStack and augments and expands upon the community documentation available at:

http://docs.openstack.org/icehouse/config-reference/content/netapp-volume-driver.html

Also note that an HTML version of this page is available at http://netapp.github.io/openstack-deploy-ops-guide/

Note: Please refer to prior versions of this document for Havana or prior.

Comments
takeshik Former NetApp Employee

I tried deploying OpenStack by this document.

I found some error.

1, P45,46

It does not exist "--name". I think this option is "--display-name".

2,

When I tried cinder volumes with iSCSI on cDOT,  it did not work finn by the following command.

# cinder type-key iscsi set storage_protocol=iSCSI

In this case, iSCSI LUN was not created in cDOT , a cinder volume created as LVM volume.

I changed the command option as below;

# cinder type-key iscsi set volume_backend_name=cdot-iscsi

It worked fine. I looked a cinder volume and it's snapshot created.

Please give me comment.

---------------

My Environment:

HP ProLiant DL160

Icehouse RDO , allinone

Fedora 20

---------------

Best Regards,

Takeshi.K

Cool Guide

Henry

lgreg Former NetApp Employee

Hi Takeshi.K

Thanks for your time in working thru the document. We will review the document along with your comments , and edit accordingly soon. If you have any other comments, please let us know. Again, thanks for your time in reviewing and the use of the document.

Regards,

Greg

takeshik Former NetApp Employee

Hello Greg,

I have an additional comment.

When I used Fedora20, it could mount by NFSv4.x  but I could not " cinder create " by WARNING.

Therefore, I set disabled for NFS v4 , v4.1 and pNFs in Page 43.

(This means instance mounts these volume by using nfs v3)

Regards,

Takeshi

rcallawa Former NetApp Employee

1) re --name: You are correct; I will update the document to reflect --display-name

2) Given the example sets the value of enabled_backends to not include an LVM backend, I would think that your environment should be updated to exclude an LVM backend.

3) Unless you hardcoded the mount options in the nfs_mount_options Cinder.conf parameter - Cinder should have attempted a mount with NFS v4.1, v4, and then successfully mounted using v3. If that is not the case then a defect should be opened.

takeshik Former NetApp Employee

Thank you.

About 2).

I succeeded by setting below cinder.conf

[DEFAULT]

# volume_backend_name=DEFAULT

enabled_backends=cdot-nfs,cdot-iscsi

3)

I did not hardcod the nfs_mount_options.

It has been commented out as below.

#nfs_mount_options=<None>

I can try again by v4 and v4.1 then I can send to you a log too.

I would like to mount v4 or v4.1, because I cannot test a copy offload capability by NetApp Copy Offload tool.

Regards,

Takeshi.K

takeshik Former NetApp Employee

About mount with nfs v4.

Fedora 20(RDO All in one host)  could not can mount by manual operation.

Error message:

[root@tkstack cinder]# mount -t nfs 10.130.208.55:/vol2_dedup /a -v

mount.nfs: timeout set for Fri Aug  1 17:54:43 2014

mount.nfs: trying text-based options 'vers=4,addr=10.130.208.55,clientaddr=10.130.205.95'

mount.nfs: mount(2): Invalid argument

mount.nfs: an incorrect mount option was specified

Also, the other host (ex, Ubuntu Server)  could mount it by nfsv4. Therefore I think cDOT is no problem.

I guess that a fedora 20 has a problem. I am investigating but I have not resolved this yet...

takeshik Former NetApp Employee

Additional comment.

I edited cinder.conf.

[cdot-nfs]

nfs_mount_options = sec=sys

When I restarts cinder-volume service, they could mount by nfsv4.

Is it correct operation?

lgreg Former NetApp Employee

It's possible that your Fedora defaults to a krb5 type of security. Typically, should be sec=sys on most flavors of Linux by default. Using sec=sys will use local UUID's,etc. I suspect that without Kerberos setup, not desired in this case, it would provide some errors that you see in the mounting..

takeshik Former NetApp Employee

About volume type;

When I run " cinder create --display-name myNFS --volume-type nfs 1" , this volume was created under the vol2_dedup.

According to this document I think it should be created under the vol5_plain. Which should I check?

lgreg Former NetApp Employee

You look thru the cinder scheduler log as well as cinder volume logs - hold see some details on why the scheduler selected that backend. Can you also share your extra specs, a df -kh, shares.conf, and cinder.conf please?

rcallawa Former NetApp Employee

As noted in the doc:

The nfs type provisions Cinder volumes onto any backend that uses the NFS storage protocol (in this example, that would be [cdot-nfs]).

Given that the content of your /etc/cinder/nfs.shares looks something like this:

10.63.40.153:/vol2_dedup
10.63.40.153:/vol3_compressed
10.63.40.153:/vol4_mirrored
10.63.40.153:/vol5_plain

The Cinder volume could end up in any of these locations - as the extra spec criteria for the nfs type would have been met; The FlexVol volume that will be chosen should be the one with the most available space.

takeshik Former NetApp Employee

I attached log when I did recreate.

[root@tkstack ~(keystone_demo)]#  cinder create --display-name myNFS --volume-type nfs 1

+---------------------+--------------------------------------+

|       Property      |                Value                 |

+---------------------+--------------------------------------+

|     attachments     |                  []                  |

|  availability_zone  |                 nova                 |

|       bootable      |                false                 |

|      created_at     |      2014-08-01T12:34:25.168415      |

| display_description |                 None                 |

|     display_name    |                myNFS                 |

|      encrypted      |                False                 |

|          id         | 93c4e059-f279-4690-9804-1fb07accb7b2 |

|       metadata      |                  {}                  |

|         size        |                  1                   |

|     snapshot_id     |                 None                 |

|     source_volid    |                 None                 |

|        status       |               creating               |

|     volume_type     |                 nfs                  |

+---------------------+--------------------------------------+

[volume.log]

2014-08-01 21:34:18.061 17652 INFO cinder.volume.manager [-] Updating volume status

2014-08-01 21:34:25.299 17650 WARNING cinder.context [-] Arguments dropped when creating context: {'user': u'5548c424732047ca99b29a6fd0aca7df', 'tenant': u'63bba5d3297c4b38af6699e7a06a0468', 'user_identity': u'5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -'}

2014-08-01 21:34:25.391 17650 INFO cinder.volume.flows.manager.create_volume [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] Volume 93c4e059-f279-4690-9804-1fb07accb7b2: being created using CreateVolumeFromSpecTask._create_raw_volume with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': u'volume-93c4e059-f279-4690-9804-1fb07accb7b2'}

2014-08-01 21:34:25.403 17650 INFO cinder.brick.remotefs.remotefs [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] Already mounted: /var/lib/cinder/mnt/7a2ad54f8ce5ac4c4f092e6cb1f1b39b

2014-08-01 21:34:25.416 17650 INFO cinder.brick.remotefs.remotefs [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] Already mounted: /var/lib/cinder/mnt/ea9ebfcad8827c2df164a5792194d80e

2014-08-01 21:34:25.425 17650 INFO cinder.brick.remotefs.remotefs [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] Already mounted: /var/lib/cinder/mnt/980e55e4b6bcbba7ff1f0adc1016e52e

2014-08-01 21:34:25.435 17650 INFO cinder.brick.remotefs.remotefs [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] Already mounted: /var/lib/cinder/mnt/5fa24de47b0ced340706db7fde94ab64

2014-08-01 21:34:26.085 17650 INFO cinder.volume.drivers.netapp.nfs [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] casted to 10.130.208.55:/vol2_dedup

2014-08-01 21:34:26.287 17650 INFO cinder.volume.flows.manager.create_volume [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] Volume volume-93c4e059-f279-4690-9804-1fb07accb7b2 (93c4e059-f279-4690-9804-1fb07accb7b2): created successfully

[schedule.log]

2014-08-01 21:34:18.062 9876 WARNING cinder.context [-] Arguments dropped when creating context: {'user': None, 'tenant': None, 'user_identity': u'- - - - -'}

2014-08-01 21:34:25.237 9876 WARNING cinder.context [-] Arguments dropped when creating context: {'user': u'5548c424732047ca99b29a6fd0aca7df', 'tenant': u'63bba5d3297c4b38af6699e7a06a0468', 'user_identity': u'5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -'}

2014-08-01 21:34:25.253 9876 WARNING cinder.scheduler.host_manager [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] volume service is down or disabled. (host: tkstack.sample.net)

2014-08-01 21:34:25.254 9876 WARNING cinder.scheduler.host_manager [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] volume service is down or disabled. (host: tkstack.sample.net@lvmdriver)

[root@tkstack ~(keystone_admin)]# cinder extra-specs-list

+--------------------------------------+-----------+--------------------------------------------+

|                  ID                  |    Name   |                extra_specs                 |

+--------------------------------------+-----------+--------------------------------------------+

| 2fb0a3b6-a341-44a6-800d-8daba4706838 |    lvm    |   {u'volume_backend_name': u'lvmdriver'}   |

| 3f615755-fef0-46d4-b3d7-aa7d87977209 |    nfs    |       {u'storage_protocol': u'nfs'}        |

| 55f541cf-b7bd-4912-8c0d-a7b7ef3aa706 |   iscsi   |      {u'storage_protocol': u'iSCSI'}       |

| 577d372e-9e6c-4468-ba8b-a4b0e11939f7 |   silver  |         {u'netapp_dedup': u'true'}         |

| 8ff32cfd-5fb2-4d5a-a9f9-fa429969e603 | analytics | {u'volume_backend_name': u'eseries-iscsi'} |

| 9081d1c7-5812-4e94-9ce9-fc1aa9118d5b |    gold   |       {u'netapp_mirrored': u'true'}        |

| f6b87b16-0a08-4270-a5f0-1707ffcd68c0 |   bronze  |      {u'netapp_compression': u'true'}      |

+--------------------------------------+-----------+--------------------------------------------+

[root@tkstack ~(keystone_admin)]#

[root@tkstack ~(keystone_admin)]# df -kh

ファイルシス                            サイズ  使用  残り 使用% マウント位置

/dev/mapper/fedora_tkstack-root            50G  8.2G   39G   18% /

devtmpfs                                  7.9G     0  7.9G    0% /dev

tmpfs                                     7.9G  4.0K  7.9G    1% /dev/shm

tmpfs                                     7.9G  636K  7.9G    1% /run

tmpfs                                     7.9G     0  7.9G    0% /sys/fs/cgroup

tmpfs                                     7.9G  116K  7.9G    1% /tmp

/dev/loop0                                1.9G  3.0M  1.7G    1% /srv/node/swift_loopback

/dev/sda1                                 477M   87M  361M   20% /boot

10.130.208.230:/vol/vol_takeshik_images   100G  550M  100G    1% /glance/images

tmpfs                                     7.9G  636K  7.9G    1% /run/netns

10.130.208.55:/vol4_mirrored               50G  256K   50G    1% /var/lib/cinder/mnt/7a2ad54f8ce5ac4c4f092e6cb1f1b39b

10.130.208.55:/vol2_dedup                 100G  192K  100G    1% /var/lib/cinder/mnt/ea9ebfcad8827c2df164a5792194d80e

10.130.208.55:/vol5_plain                 100G  128K  100G    1% /var/lib/cinder/mnt/980e55e4b6bcbba7ff1f0adc1016e52e

10.130.208.55:/vol3_compressed            100G  192K  100G    1% /var/lib/cinder/mnt/5fa24de47b0ced340706db7fde94ab64

[root@tkstack ~(keystone_admin)]#

[root@tkstack cinder(keystone_admin)]# more nfs.shares

10.130.208.55:/vol2_dedup

10.130.208.55:/vol3_compressed

10.130.208.55:/vol4_mirrored

10.130.208.55:/vol5_plain

[root@tkstack cinder(keystone_admin)]# cat cinder.conf

[DEFAULT]

amqp_durable_queues=False

rabbit_host=10.130.205.95

rabbit_port=5672

rabbit_hosts=10.130.205.95:5672

rabbit_userid=guest

rabbit_password=guest

rabbit_virtual_host=/

rabbit_ha_queues=False

notification_driver=cinder.openstack.common.notifier.rpc_notifier

rpc_backend=cinder.openstack.common.rpc.impl_kombu

control_exchange=openstack

osapi_volume_listen=0.0.0.0

backup_swift_url=http://10.130.205.95:8080/v1/AUTH_

backup_swift_container=volumes_backup

backup_swift_object_size=52428800

backup_swift_retry_attempts=3

backup_swift_retry_backoff=2

backup_driver=cinder.backup.drivers.swift

api_paste_config=/etc/cinder/api-paste.ini

glance_host=10.130.205.95

backup_topic=cinder-backup

backup_manager=cinder.backup.manager.BackupManager

backup_api_class=cinder.backup.api.API

auth_strategy=keystone

backup_name_template=backup-%s

debug=false

verbose=true

log_dir=/var/log/cinder

use_syslog=False

iscsi_ip_address=10.130.205.95

iscsi_helper=tgtadm

#default_volume_type=lvm

#volume_group=cinder-volumes

enabled_backends=cdot-nfs,cdot-iscsi

[BRCD_FABRIC_EXAMPLE]

[database]

connection=mysql://cinder:0955707b425a435b@10.130.205.95/cinder

idle_timeout=3600

[fc-zone-manager]

[keymgr]

[keystone_authtoken]

[matchmaker_ring]

[ssl]

#[lvmdriver]

#volume_group=cinder-volumes

#volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver

#volume_backend_name=lvmdriver

[cdot-nfs]

volume_backend_name=cdot-nfs

volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver

netapp_server_hostname=10.130.202.180

netapp_server_port=80

netapp_storage_protocol=nfs

netapp_storage_family=ontap_cluster

netapp_login=openstack

netapp_password=eisaku35

netapp_vserver=demo-nfs-svm

nfs_shares_config=/etc/cinder/nfs.shares

nfs_mount_options = sec=sys

[cdot-iscsi]

volume_backend_name=cdot-iscsi

volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver

netapp_server_hostname=10.130.202.180

netapp_server_port=80

netapp_storage_protocol=iscsi

netapp_storage_family=ontap_cluster

netapp_login=openstack

netapp_password=eisaku35

netapp_vserver=demo-iscsi-svm

takeshik Former NetApp Employee

Thank you.

I can understand that I should setup Kerb5 before I use NFSv4.

takeshik Former NetApp Employee

Thank you.

I'll try to test several times.

takeshik Former NetApp Employee

Hi All,

I found the error in this document.

https://communities.netapp.com/docs/DOC-34042

I created a openstack role as P36 in this document.

In this case I saw that a Cinder Volume(iSCSI LUN) failed to connect to an instance.

2014-09-01 10:44:33.045 25164 WARNING cinder.volume.drivers.netapp.iscsi [req-c397f4f1-3f0b-44a9-b78b-363fa5c371ed 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -]

Error unmapping lun. Code :13003, Message:Insufficient privileges: user 'openstack' does not have write access to this resource

Therefore I modified a role form "readonly" to "all" for "lun mapped" command then I run the same oparation again. It was successful.

cluster01-cdot::> security login role modify -role openstack -cmddirname "lun mapped" -access all

I think it should set "lun mapped" to "all" privileges.

Regards,

lgreg Former NetApp Employee

I'll look at this today. Thanks for pointing this out.

Greg

Warning!

This NetApp Community is public and open website that is indexed by search engines such as Google. Participation in the NetApp Community is voluntary. All content posted on the NetApp Community is publicly viewable and available. This includes the rich text editor which is not encrypted for https.

In accordance to our Code of Conduct and Community Terms of Use DO NOT post or attach the following:

  • Software files (compressed or uncompressed)
  • Files that require an End User License Agreement (EULA)
  • Confidential information
  • Personal data you do not want publicly available
  • Another’s personally identifiable information
  • Copyrighted materials without the permission of the copyright owner

Files and content that do not abide by the Community Terms of Use or Code of Conduct will be removed. Continued non-compliance may result in NetApp Community account restrictions or termination.