Clone file does not exist error occurs when creating volume.

[shuya]

Hello. We are using cDOT8.3P2 and OpenStack Queens on RHEL7.5.

When we create volume, "NaApiError: NetApp API failed. Reason - 14970:Clone operation failed to start: Clone file does not exist.." error occurred.

Error message is shown in the cinder-volume.log as below.

2018-07-04 15:16:36.158 61 DEBUG cinder.volume.drivers.netapp.dataontap.nfs_cmode [req-b258c973-7618-4289-a600-8d043e5571fc 015111ec881e4182a77ae3f46a640b60 3099a7d711cb46ab8a22b0cc651f927c - default default]	Trying copy from cache using cloning. _copy_from_cache /usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/nfs_cmode.py:481
2018-07-04 15:16:36.328 61 DEBUG cinder.volume.drivers.netapp.dataontap.client.client_cmode [req-b258c973-7618-4289-a600-8d043e5571fc 015111ec881e4182a77ae3f46a640b60 3099a7d711cb46ab8a22b0cc651f927c - default default]	Cloning with params volume cinder_system, src img-cache-e38c5845-d167-46bd-ba73-2103f5cd7db5, dest volume-34027a4a-fd3a-4f25-b0b6-a7d624c27571, vserver <vserver name>,source_snapshot None clone_file /usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_cmode.py:713
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode [req-b258c973-7618-4289-a600-8d043e5571fc 015111ec881e4182a77ae3f46a640b60 3099a7d711cb46ab8a22b0cc651f927c - default default]	Error in workflow copy from cache.: NaApiError: NetApp API failed. Reason - 14970:Clone operation failed to start: Clone file does not exist..
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	Traceback (most recent call last):
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/nfs_cmode.py", line 484, in _copy_from_cache
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	     nfs_share, file_name, volume['name'], dest_exists=True)
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 849, in trace_method_logging_wrapper
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	     return f(*args, **kwargs)
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 849, in trace_method_logging_wrapper
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	     return f(*args, **kwargs)
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/nfs_cmode.py", line 560, in _clone_file_dst_exists
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	     dest_exists=dest_exists)
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 849, in trace_method_logging_wrapper
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	     return f(*args, **kwargs)
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 849, in trace_method_logging_wrapper
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	     return f(*args, **kwargs)
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_cmode.py", line 728, in clone_file
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	     self._invoke_vserver_api(clone_create, vserver)
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 849, in trace_method_logging_wrapper
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	     return f(*args, **kwargs)
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 849, in trace_method_logging_wrapper
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	     return f(*args, **kwargs)
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_cmode.py", line 86, in _invoke_vserver_api
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	     result = server.invoke_successfully(na_element, True)
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 222, in invoke_successfully
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	     raise NaApiError(code, msg)
2018-07-04 15:16:36.396 61 ERROR cinder.volume.drivers.netapp.dataontap.nfs_cmode	 NaApiError: NetApp API failed. Reason - 14970:Clone operation failed to start: Clone file does not exist..

When calling volume clone create command from NetApp driver of Cinder volume with "<destination-exists>true</destination-exists>" parameter, Occurred Clone file does not exist error.

We tried calling volume clone create command without "<destination-exists>true</destination-exists>" parameter, it succeed.

Are there any bugs in NetApp driver or OS(cDOT8.3P2)?

Best Regards.

[L4rS]

Were you able to solve that problem? I ran into the same issue.

[L4rS]

I found the reason. The netapp storage throws an error if you specify -overwrite-destination (same option on cli as xml child destination-exists) if the destination file does not exist:

mynetapp::> volume file clone create -vserver test02 -volume vol1 -source-path /img-cache-28d4de59-71fb-4537-865c-02e0c7c9b401 -destination-path /test-clone -overwrite-destination

Warning: You have specified "-overwrite-destination". If the destination exists, it will be overwritten.
Do you want to continue? {y|n}: y

Error: command failed: Clone start failed: Clone operation failed to start: Clone file does not exist.

And this is exactly what happens in the netapp cinder driver: Destination volume does not exist yet and the drivers tries to create it with the overwrite (destination-exists) option.

This looks like a netapp bug to me, because if overwrite is set it should not throw an error if the volume does not exist. Could anyone please have a look into this?

[shuya]

Hi L4rS,

We have not solved it yet. Also no reply from this community yet.

[L4rS · Accepted solution]

Hi, check out this openstack cinder bug I opened recently: https://bugs.launchpad.net/cinder/+bug/1799214

It's a netapp ontap bug and is fixed in the latest release.

[shuya]

Thank you for sharing the information.
We also just found the same cause yesterday.
We would like to update Netapp ontap and try again.