Good morning, this is my first post to this forum. I'm tasked with building a large OpenStack Queens private cloud at NASA Goddard Space Flight Center. We purchased a FAS 2650 ontap cmode 9.3P4 cluster with about 70TB of usable SSD storage. OpenStack is running well, and I'm just trying to establish tighter integration between Cinder/Glance and the NFS backend, to take advantage of features like Copy Offload. I'm having limited success there, though
Basically, I can get it to work, but only when Glance & Cinder are hosted from the same flexvol. So, it's working right now, but this is not how I wanted to lay out my storage. I wanted Glance & Cinder at least in separate flexvols, and preferably balanced with one on one filer head node and the other on the second filer head node.
Current situation is this:
filer2:/images (contains both Glance, Cinder images -- copy offload works great)
Situations where copy offload fails:
1) filer1:/openstack/glance and filer2:/openstack/cinder (where 'openstack' is just a junction point in the export namespace).
This fails badly. First off, the driver doesn't seem to understand junction paths, and it ends up thinking that 'openstack' is a flexvol, when in fact it's just a directory (a mountpoint) in the namespace. You get stuff like this:
NaApiError: NetApp API failed. Reason - 13040:Volume "openstack" does not exist in Vserver "openstack". Reason: entry doesn't exist.
2018-08-29 09:37:27.965 32295 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode Traceback (most recent call last):
But regardless, the driver cannot find the source file (meaning the glance image):
Discover file retries exhausted.
2018-08-29 09:23:29.326 32295 INFO
cinder.volume.drivers.netapp.dataontap.nfs_base
[req-4c82cd34-bc97-4820-be4c-dd134fcc727f
5cc84caf09f814620d952c9aa399c0ecaf8dbfa538c11143a6ed24bd06ece0f3
6e6d8ff081014c679f18ad4b818ffd4c - - -] Image cloning unsuccessful for
image 4b8371ec-e7ef-4081-ba36-629eb5a18957. Message: NetApp API failed.
Reason - 14935:Clone operation failed to start: Source file does not exist..
2) filer2:/images/glance and filer2:/images/cinder, where 'images' is a flexvol and 'glance' and 'cinder' are qtrees in that same flexvol. Fails with cinder volume agent saying it can't find the volume, or the volume doesn't exist, because the driver doesn't appear to understand what a qtree is.
3) filer1:/glance and filer2:/cinder, where 'glance' and 'cinder' are just flexvols, but hosted on different filer heads of the same ontap cluster. Same problem, driver cannot find source file glance image, like:
Discover file retries exhausted.
2018-08-29 09:23:29.326 32295 INFO
cinder.volume.drivers.netapp.dataontap.nfs_base
[req-4c82cd34-bc97-4820-be4c-dd134fcc727f
5cc84caf09f814620d952c9aa399c0ecaf8dbfa538c11143a6ed24bd06ece0f3
6e6d8ff081014c679f18ad4b818ffd4c - - -] Image cloning unsuccessful for
image 4b8371ec-e7ef-4081-ba36-629eb5a18957. Message: NetApp API failed.
Reason - 14935:Clone operation failed to start: Source file does not exist..
4) filer2:/glance and filer2:/cinder, where 'glance' and 'cinder' are separate flexvols in the same aggr on the same filer head. This kinda sorta works, but the API still fails, and it falls back to doing what looks like a literal unix 'mv' command. it also takes a long time because there's a timeout waiting on the API failure. It looks like this:
cinder.volume.drivers.netapp.dataontap.nfs_base
[req-03f1f945-cd36-417a-a378-4669b40e43c4 5cc84caf09f814620d9
52c9aa399c0ecaf8dbfa538c11143a6ed24bd06ece0f3
6e6d8ff081014c679f18ad4b818ffd4c - - -] Discover file retries exhausted.
2018-08-29 12:42:53.993 23332 WARNING
cinder.volume.drivers.netapp.dataontap.nfs_base
[req-03f1f945-cd36-417a-a378-4669b40e43c4 5cc84caf09f814620d9
52c9aa399c0ecaf8dbfa538c11143a6ed24bd06ece0f3
6e6d8ff081014c679f18ad4b818ffd4c - - -] Exception moving file
/var/lib/cinder/mnt/5905fe460e578283c84
96e21d8b3a6da/25b9beee-3812-4941-946c-bb7f753c85f5. Message - Unexpected
error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mv
/var/lib/cinder/mnt/5905fe460e578283c8496e21d8b3a6da/25b9beee-3812-4941-946c-bb7f753c85f5
/var/lib/cinder/mnt/5905fe460e578283c8496e21d8b3a6da/img-cache-58aaa550-ff19-4372-bb93-634af3715dd5
Exit code: 1
Stdout: u''
Stderr: '/bin/mv: failed to access
\xe2\x80\x98/var/lib/cinder/mnt/5905fe460e578283c8496e21d8b3a6da/img-cache-58aaa550-ff19-4372-bb93-634af3715dd5\xe2\x80\x99:
Not a directory\n': ProcessExecutionError: Unexpected error while
running command.
2018-08-29 12:42:53.995 23332 INFO
cinder.volume.drivers.netapp.dataontap.nfs_base
[req-03f1f945-cd36-417a-a378-4669b40e43c4
5cc84caf09f814620d952c9aa399c0ecaf8dbfa538c11143a6ed24bd06ece0f3
6e6d8ff081014c679f18ad4b818ffd4c - - -] Performing post clone for
volume-5fe2827f-e865-49f4-bf64-d32419cfc2fc
In all cases, Glance has been configured to expose multiple location metadata as per this post by D. Cain:
https://community.netapp.com/t5/OpenStack-Discussions/Copy-offload-unsuccessful-Source-host-details-not-found/m-p/108795#M240.
And here is my ontap backend config section from cinder.conf:
[gpc-xxxxx]
volume_backend_name = gpc-xxxxx
volume_driver = cinder.volume.drivers.netapp.common.NetAppDriver
trace_flags = method,api
netapp_api_trace_pattern = ^(?!(perf)).*$
netapp_storage_family = ontap_cluster
netapp_copyoffload_tool_path = /etc/cinder/na_copyoffload_64
nas_secure_file_operations = True
nas_secure_file_permissions = True
netapp_server_hostname = openstack
netapp_vserver = openstack
expiry_thres_minutes = 1
thres_avl_size_perc_stop = 100
thres_avl_size_perc_start = 100
netapp_server_port = 443
netapp_storage_protocol = nfs
nfs_shares_config = /etc/cinder/nfs_shares
nfs_mount_attempts = 3
nfs_mount_options =
'rw,noatime,vers=4.0,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,sec=sys,lookupcache=pos'
netapp_login = cinder
netapp_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
netapp_transport_type = https
image_volume_cache_enabled = False
There are no unix permissions issues, no nfs export issues, and no API auth issues. The API and/or copy offload utility just fails to "find" the glance image unless those images are inside the same flexvol as the Cinder volume backend that the driver is pointed at.
Any assistance, or tips, would be hugely appreciated!
Cheers,
Jonathan Mills
