NetApp OpenStack Deployment and Operations Guide 3.0 (Icehouse)

[rcallawa]

The NetApp OpenStack Deployment and Operations Guide version 3.0.  This version focuses on OpenStack Block Storage (Cinder), OpenStack Image (Glance), OpenStack Object Storage (Swift) and aspects of OpenStack Compute (Nova) deployment. As NetApp's File Share Service (the OpenStack Manila development project) is still under development and not yet integrated into OpenStack it is not covered in this version of the guide.  This version applies to the Icehouse release of OpenStack and augments and expands upon the community documentation available at:

http://docs.openstack.org/icehouse/config-reference/content/netapp-volume-driver.html

Also note that an HTML version of this page is available at http://netapp.github.io/openstack-deploy-ops-guide/

Note: Please refer to prior versions of this document for Havana or prior.

[takeshik]

I tried deploying OpenStack by this document.

I found some error.

1, P45,46

It does not exist "--name". I think this option is "--display-name".

2,

When I tried cinder volumes with iSCSI on cDOT,  it did not work finn by the following command.

# cinder type-key iscsi set storage_protocol=iSCSI

In this case, iSCSI LUN was not created in cDOT , a cinder volume created as LVM volume.

I changed the command option as below;

# cinder type-key iscsi set volume_backend_name=cdot-iscsi

It worked fine. I looked a cinder volume and it's snapshot created.

Please give me comment.

---------------

My Environment:

HP ProLiant DL160

Icehouse RDO , allinone

Fedora 20

---------------

Best Regards,

Takeshi.K

[HENRYPAN2]

Cool Guide

Henry

[lgreg]

Hi Takeshi.K

Thanks for your time in working thru the document. We will review the document along with your comments , and edit accordingly soon. If you have any other comments, please let us know. Again, thanks for your time in reviewing and the use of the document.

Regards,

Greg

[takeshik]

Hello Greg,

I have an additional comment.

When I used Fedora20, it could mount by NFSv4.x  but I could not " cinder create " by WARNING.

Therefore, I set disabled for NFS v4 , v4.1 and pNFs in Page 43.

(This means instance mounts these volume by using nfs v3)

Regards,

Takeshi

[rcallawa]

1) re --name: You are correct; I will update the document to reflect --display-name

2) Given the example sets the value of enabled_backends to not include an LVM backend, I would think that your environment should be updated to exclude an LVM backend.

3) Unless you hardcoded the mount options in the nfs_mount_options Cinder.conf parameter - Cinder should have attempted a mount with NFS v4.1, v4, and then successfully mounted using v3. If that is not the case then a defect should be opened.

[takeshik]

Thank you.

About 2).

I succeeded by setting below cinder.conf

[DEFAULT]

# volume_backend_name=DEFAULT

enabled_backends=cdot-nfs,cdot-iscsi

3)

I did not hardcod the nfs_mount_options.

It has been commented out as below.

#nfs_mount_options=<None>

I can try again by v4 and v4.1 then I can send to you a log too.

I would like to mount v4 or v4.1, because I cannot test a copy offload capability by NetApp Copy Offload tool.

Regards,

Takeshi.K

[takeshik]

About mount with nfs v4.

Fedora 20(RDO All in one host)  could not can mount by manual operation.

Error message:

[root@tkstack cinder]# mount -t nfs 10.130.208.55:/vol2_dedup /a -v

mount.nfs: timeout set for Fri Aug  1 17:54:43 2014

mount.nfs: trying text-based options 'vers=4,addr=10.130.208.55,clientaddr=10.130.205.95'

mount.nfs: mount(2): Invalid argument

mount.nfs: an incorrect mount option was specified

Also, the other host (ex, Ubuntu Server)  could mount it by nfsv4. Therefore I think cDOT is no problem.

I guess that a fedora 20 has a problem. I am investigating but I have not resolved this yet...

[takeshik]

Additional comment.

I edited cinder.conf.

[cdot-nfs]

nfs_mount_options = sec=sys

When I restarts cinder-volume service, they could mount by nfsv4.

Is it correct operation?

[lgreg]

It's possible that your Fedora defaults to a krb5 type of security. Typically, should be sec=sys on most flavors of Linux by default. Using sec=sys will use local UUID's,etc. I suspect that without Kerberos setup, not desired in this case, it would provide some errors that you see in the mounting..

[takeshik]

About volume type;

When I run " cinder create --display-name myNFS --volume-type nfs 1" , this volume was created under the vol2_dedup.

According to this document I think it should be created under the vol5_plain. Which should I check?

[lgreg]

You look thru the cinder scheduler log as well as cinder volume logs - hold see some details on why the scheduler selected that backend. Can you also share your extra specs, a df -kh, shares.conf, and cinder.conf please?

[rcallawa]

As noted in the doc:

The nfs type provisions Cinder volumes onto any backend that uses the NFS storage protocol (in this example, that would be [cdot-nfs]).

Given that the content of your /etc/cinder/nfs.shares looks something like this:

10.63.40.153:/vol2_dedup 
10.63.40.153:/vol3_compressed 
10.63.40.153:/vol4_mirrored 
10.63.40.153:/vol5_plain 

The Cinder volume could end up in any of these locations - as the extra spec criteria for the nfs type would have been met; The FlexVol volume that will be chosen should be the one with the most available space.

[takeshik]

I attached log when I did recreate.

[root@tkstack ~(keystone_demo)]#  cinder create --display-name myNFS --volume-type nfs 1
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2014-08-01T12:34:25.168415      |
| display_description |                 None                 |
|     display_name    |                myNFS                 |
|      encrypted      |                False                 |
|          id         | 93c4e059-f279-4690-9804-1fb07accb7b2 |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 nfs                  |
+---------------------+--------------------------------------+

[volume.log]
2014-08-01 21:34:18.061 17652 INFO cinder.volume.manager [-] Updating volume status
2014-08-01 21:34:25.299 17650 WARNING cinder.context [-] Arguments dropped when creating context: {'user': u'5548c424732047ca99b29a6fd0aca7df', 'tenant': u'63bba5d3297c4b38af6699e7a06a0468', 'user_identity': u'5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -'}
2014-08-01 21:34:25.391 17650 INFO cinder.volume.flows.manager.create_volume [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] Volume 93c4e059-f279-4690-9804-1fb07accb7b2: being created using CreateVolumeFromSpecTask._create_raw_volume with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': u'volume-93c4e059-f279-4690-9804-1fb07accb7b2'}
2014-08-01 21:34:25.403 17650 INFO cinder.brick.remotefs.remotefs [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] Already mounted: /var/lib/cinder/mnt/7a2ad54f8ce5ac4c4f092e6cb1f1b39b
2014-08-01 21:34:25.416 17650 INFO cinder.brick.remotefs.remotefs [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] Already mounted: /var/lib/cinder/mnt/ea9ebfcad8827c2df164a5792194d80e
2014-08-01 21:34:25.425 17650 INFO cinder.brick.remotefs.remotefs [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] Already mounted: /var/lib/cinder/mnt/980e55e4b6bcbba7ff1f0adc1016e52e
2014-08-01 21:34:25.435 17650 INFO cinder.brick.remotefs.remotefs [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] Already mounted: /var/lib/cinder/mnt/5fa24de47b0ced340706db7fde94ab64
2014-08-01 21:34:26.085 17650 INFO cinder.volume.drivers.netapp.nfs [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] casted to 10.130.208.55:/vol2_dedup
2014-08-01 21:34:26.287 17650 INFO cinder.volume.flows.manager.create_volume [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] Volume volume-93c4e059-f279-4690-9804-1fb07accb7b2 (93c4e059-f279-4690-9804-1fb07accb7b2): created successfully

[schedule.log]
2014-08-01 21:34:18.062 9876 WARNING cinder.context [-] Arguments dropped when creating context: {'user': None, 'tenant': None, 'user_identity': u'- - - - -'}
2014-08-01 21:34:25.237 9876 WARNING cinder.context [-] Arguments dropped when creating context: {'user': u'5548c424732047ca99b29a6fd0aca7df', 'tenant': u'63bba5d3297c4b38af6699e7a06a0468', 'user_identity': u'5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -'}
2014-08-01 21:34:25.253 9876 WARNING cinder.scheduler.host_manager [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] volume service is down or disabled. (host: tkstack.sample.net)
2014-08-01 21:34:25.254 9876 WARNING cinder.scheduler.host_manager [req-3ead17e5-3385-4a9f-8d8d-ec844c6adfb0 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] volume service is down or disabled. (host: tkstack.sample.net@lvmdriver)

[root@tkstack ~(keystone_admin)]# cinder extra-specs-list
+--------------------------------------+-----------+--------------------------------------------+
|                  ID                  |    Name   |                extra_specs                 |
+--------------------------------------+-----------+--------------------------------------------+
| 2fb0a3b6-a341-44a6-800d-8daba4706838 |    lvm    |   {u'volume_backend_name': u'lvmdriver'}   |
| 3f615755-fef0-46d4-b3d7-aa7d87977209 |    nfs    |       {u'storage_protocol': u'nfs'}        |
| 55f541cf-b7bd-4912-8c0d-a7b7ef3aa706 |   iscsi   |      {u'storage_protocol': u'iSCSI'}       |
| 577d372e-9e6c-4468-ba8b-a4b0e11939f7 |   silver  |         {u'netapp_dedup': u'true'}         |
| 8ff32cfd-5fb2-4d5a-a9f9-fa429969e603 | analytics | {u'volume_backend_name': u'eseries-iscsi'} |
| 9081d1c7-5812-4e94-9ce9-fc1aa9118d5b |    gold   |       {u'netapp_mirrored': u'true'}        |
| f6b87b16-0a08-4270-a5f0-1707ffcd68c0 |   bronze  |      {u'netapp_compression': u'true'}      |
+--------------------------------------+-----------+--------------------------------------------+
[root@tkstack ~(keystone_admin)]#

[root@tkstack ~(keystone_admin)]# df -kh
ファイルシス                            サイズ  使用  残り 使用% マウント位置
/dev/mapper/fedora_tkstack-root            50G  8.2G   39G   18% /
devtmpfs                                  7.9G     0  7.9G    0% /dev
tmpfs                                     7.9G  4.0K  7.9G    1% /dev/shm
tmpfs                                     7.9G  636K  7.9G    1% /run
tmpfs                                     7.9G     0  7.9G    0% /sys/fs/cgroup
tmpfs                                     7.9G  116K  7.9G    1% /tmp
/dev/loop0                                1.9G  3.0M  1.7G    1% /srv/node/swift_loopback
/dev/sda1                                 477M   87M  361M   20% /boot
10.130.208.230:/vol/vol_takeshik_images   100G  550M  100G    1% /glance/images
tmpfs                                     7.9G  636K  7.9G    1% /run/netns
10.130.208.55:/vol4_mirrored               50G  256K   50G    1% /var/lib/cinder/mnt/7a2ad54f8ce5ac4c4f092e6cb1f1b39b
10.130.208.55:/vol2_dedup                 100G  192K  100G    1% /var/lib/cinder/mnt/ea9ebfcad8827c2df164a5792194d80e
10.130.208.55:/vol5_plain                 100G  128K  100G    1% /var/lib/cinder/mnt/980e55e4b6bcbba7ff1f0adc1016e52e
10.130.208.55:/vol3_compressed            100G  192K  100G    1% /var/lib/cinder/mnt/5fa24de47b0ced340706db7fde94ab64
[root@tkstack ~(keystone_admin)]#

[root@tkstack cinder(keystone_admin)]# more nfs.shares
10.130.208.55:/vol2_dedup
10.130.208.55:/vol3_compressed
10.130.208.55:/vol4_mirrored
10.130.208.55:/vol5_plain
[root@tkstack cinder(keystone_admin)]# cat cinder.conf
[DEFAULT]
amqp_durable_queues=False
rabbit_host=10.130.205.95
rabbit_port=5672
rabbit_hosts=10.130.205.95:5672
rabbit_userid=guest
rabbit_password=guest
rabbit_virtual_host=/
rabbit_ha_queues=False
notification_driver=cinder.openstack.common.notifier.rpc_notifier
rpc_backend=cinder.openstack.common.rpc.impl_kombu
control_exchange=openstack
osapi_volume_listen=0.0.0.0
backup_swift_url=http://10.130.205.95:8080/v1/AUTH_
backup_swift_container=volumes_backup
backup_swift_object_size=52428800
backup_swift_retry_attempts=3
backup_swift_retry_backoff=2
backup_driver=cinder.backup.drivers.swift
api_paste_config=/etc/cinder/api-paste.ini
glance_host=10.130.205.95
backup_topic=cinder-backup
backup_manager=cinder.backup.manager.BackupManager
backup_api_class=cinder.backup.api.API
auth_strategy=keystone
backup_name_template=backup-%s
debug=false
verbose=true
log_dir=/var/log/cinder
use_syslog=False
iscsi_ip_address=10.130.205.95
iscsi_helper=tgtadm
#default_volume_type=lvm
#volume_group=cinder-volumes
enabled_backends=cdot-nfs,cdot-iscsi
[BRCD_FABRIC_EXAMPLE]
[database]
connection=mysql://cinder:0955707b425a435b@10.130.205.95/cinder
idle_timeout=3600
[fc-zone-manager]
[keymgr]
[keystone_authtoken]
[matchmaker_ring]
[ssl]
#[lvmdriver]
#volume_group=cinder-volumes
#volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
#volume_backend_name=lvmdriver
[cdot-nfs]
volume_backend_name=cdot-nfs
volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
netapp_server_hostname=10.130.202.180
netapp_server_port=80
netapp_storage_protocol=nfs
netapp_storage_family=ontap_cluster
netapp_login=openstack
netapp_password=eisaku35
netapp_vserver=demo-nfs-svm
nfs_shares_config=/etc/cinder/nfs.shares
nfs_mount_options = sec=sys
[cdot-iscsi]
volume_backend_name=cdot-iscsi
volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
netapp_server_hostname=10.130.202.180
netapp_server_port=80
netapp_storage_protocol=iscsi
netapp_storage_family=ontap_cluster
netapp_login=openstack
netapp_password=eisaku35
netapp_vserver=demo-iscsi-svm

[takeshik]

Thank you.

I can understand that I should setup Kerb5 before I use NFSv4.

[takeshik]

Thank you.

I'll try to test several times.

[takeshik]

Hi All,

I found the error in this document.

https://communities.netapp.com/docs/DOC-34042

I created a openstack role as P36 in this document.

In this case I saw that a Cinder Volume(iSCSI LUN) failed to connect to an instance.

2014-09-01 10:44:33.045 25164 WARNING cinder.volume.drivers.netapp.iscsi [req-c397f4f1-3f0b-44a9-b78b-363fa5c371ed 5548c424732047ca99b29a6fd0aca7df 63bba5d3297c4b38af6699e7a06a0468 - - -] 
 Error unmapping lun. Code :13003, Message:Insufficient privileges: user 'openstack' does not have write access to this resource

Therefore I modified a role form "readonly" to "all" for "lun mapped" command then I run the same oparation again. It was successful.

cluster01-cdot::> security login role modify -role openstack -cmddirname "lun mapped" -access all

I think it should set "lun mapped" to "all" privileges.

Regards,

[lgreg]

I'll look at this today. Thanks for pointing this out.

Greg