Solved: Re: I try CopyOffload

takeshik · ‎2014-08-01

I can set CopyOffload.

However, I do not know how to check if they could be set correctly.

What should I check or test?

Regards,

Takeshi.K

rcallawa · ‎2014-08-02

It doesn't appear that Red Hat has created a new RPM for RDO on Fedora 21 yet:

https://repos.fedorapeople.org/repos/openstack/openstack-icehouse/fedora-20/

You could manually update the file /usr/lib/python2.7/site-packages/cinder/image/glance.py - line 434, to look like:

output[attr] = getattr(image, attr, None)

to see if this resolves your issue.

View solution in original post

rcallawa · ‎2014-08-01

If the copy offload operation succeeds, you should see an info-level log message in Cinder logs stating "Copied image %(img)s to volume %(vol)s using copy offload workflow"

If the copy offload operation fails, you should see a debug level message in Cinder logs stating "Copy offload either not configured or unsupported." or an exception "Copy offload workflow unsuccessful".

takeshik · ‎2014-08-01

I cloud not find these messages in any the cinder log.

By the way, Please tell me some example operation to see the copy offload operation.

Regards,

Takeshi.K

rcallawa · ‎2014-08-01

If you use the cinder command line tool to issue the following command (replacing 01234567890 with the id of an image in your glance repository)

cinder create --image-id 01234567890 20

This will trigger the creation of a new Cinder volume whose source content comes from Glance.

Be sure that Glance is configured to use an NFS export from a different FlexVol volume on the same cluster where the Cinder backend is defined.

takeshik · ‎2014-08-01

Thank you.

I tried a command.

[root@tkstack glance(keystone_admin)]# cinder create --image-id 17a544b2-1bf8-4b1c-954a-c3a405b51fbd 20
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-02de21a7-02ab-48f6-a670-4097910ea322)

There may be some problems..

rcallawa · ‎2014-08-02

Without logs from Cinder, I can't help much more.

Please provide relevant logs from Cinder, and the output from these two commands:

glance image-list

cinder service-list

takeshik · ‎2014-08-02

Thank you.

Exactly,

When I will retry , I would like to provide some logs and outputs from commands.

takeshik · ‎2014-08-02

When I run this command , volume.log and schedule.log were not output anything. api.log was output.

cinder create --image-id 17a544b2-1bf8-4b1c-954a-c3a405b51fbd 20

2014-08-02 20:51:20.492 7776 INFO eventlet.wsgi.server [-] (7776) accepted ('10.130.205.95', 57096)

2014-08-02 20:51:20.496 7776 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.130.205.95

2014-08-02 20:51:20.551 7776 INFO cinder.api.openstack.wsgi [req-60e69d94-25b0-40dc-b0c2-0547084a790f 13d3305af9e94e888b451e3775dd09ac 51907191e78543afa754de6b69624205 - - -] POST http://10.130.205.95:8776/v1/51907191e78543afa754de6b69624205/volumes

2014-08-02 20:51:20.552 7776 AUDIT cinder.api.v1.volumes [req-60e69d94-25b0-40dc-b0c2-0547084a790f 13d3305af9e94e888b451e3775dd09ac 51907191e78543afa754de6b69624205 - - -] Create volume of 20 GB

2014-08-02 20:51:20.720 7776 ERROR cinder.api.middleware.fault [req-60e69d94-25b0-40dc-b0c2-0547084a790f 13d3305af9e94e888b451e3775dd09ac 51907191e78543afa754de6b69624205 - - -] Caught error: deleted

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault Traceback (most recent call last):

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/middleware/fault.py", line 75, in __call__

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault return req.get_response(self.application)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault application, catch_exc_info=False)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault app_iter = application(self.environ, start_response)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault return resp(environ, start_response)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py", line 679, in __call__

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault return self.app(env, start_response)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault return resp(environ, start_response)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault return resp(environ, start_response)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault response = self.app(environ, start_response)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault return resp(environ, start_response)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault resp = self.call_func(req, *args, **self.kwargs)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault return self.func(req, *args, **kwargs)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 895, in __call__

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault content_type, body, accept)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 943, in _process_stack

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault action_result = self.dispatch(meth, request, action_args)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 1019, in dispatch

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault return method(req=request, **action_args)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/v1/volumes.py", line 432, in create

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault **kwargs)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/volume/api.py", line 189, in create

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault flow_engine.run()

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/taskflow/utils/lock_utils.py", line 51, in wrapper

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault return f(*args, **kwargs)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 118, in run

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault self._run()

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 128, in _run

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault self._revert(misc.Failure())

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 81, in _revert

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault misc.Failure.reraise_if_any(failures.values())

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/taskflow/utils/misc.py", line 487, in reraise_if_any

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault failures[0].reraise()

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/taskflow/utils/misc.py", line 494, in reraise

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault six.reraise(*self._exc_info)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 36, in _execute_task

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault result = task.execute(**arguments)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/volume/flows/api/create_volume.py", line 341, in execute

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault self._check_image_metadata(context, image_id, size)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/volume/flows/api/create_volume.py", line 180, in _check_image_metadata

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault image_meta = self.image_service.show(context, image_id)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/image/glance.py", line 228, in show

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault base_image_meta = self._translate_from_glance(image)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/image/glance.py", line 336, in _translate_from_glance

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault image_meta = _extract_attributes(image)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/image/glance.py", line 434, in _extract_attributes

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault output[attr] = getattr(image, attr)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/warlock/model.py", line 69, in __getattr__

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault raise AttributeError(key)

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault AttributeError: deleted

2014-08-02 20:51:20.720 7776 TRACE cinder.api.middleware.fault

2014-08-02 20:51:20.723 7776 INFO cinder.api.middleware.fault [req-60e69d94-25b0-40dc-b0c2-0547084a790f 13d3305af9e94e888b451e3775dd09ac 51907191e78543afa754de6b69624205 - - -] http://10.130.205.95:8776/v1/51907191e78543afa754de6b69624205/volumes returned with HTTP 500

2014-08-02 20:51:20.726 7776 INFO eventlet.wsgi.server [req-60e69d94-25b0-40dc-b0c2-0547084a790f 13d3305af9e94e888b451e3775dd09ac 51907191e78543afa754de6b69624205 - - -] 10.130.205.95 - - [02/Aug/2014 20:51:20] "POST /v1/51907191e78543afa754de6b69624205/volumes HTTP/1.1" 500 401 0.233147

Two command

[root@tkstack ~(keystone_admin)]# glance image-list
+--------------------------------------+-------------------+-------------+------------------+-----------+--------+
| ID                                   | Name              | Disk Format | Container Format | Size      | Status |
+--------------------------------------+-------------------+-------------+------------------+-----------+--------+
| 309e930a-7eba-483c-9bc8-d8880b392d45 | CentOS-6.5_x86_64 | qcow2       | bare             | 344457216 | active |
| 17a544b2-1bf8-4b1c-954a-c3a405b51fbd | cirros            | qcow2       | bare             | 13147648  | active |
| 156e8961-aedc-4879-8b42-903236f2e8d8 | Fedora 20 x86_64  | qcow2       | bare             | 210829312 | active |
+--------------------------------------+-------------------+-------------+------------------+-----------+--------+
[root@tkstack ~(keystone_admin)]# cinder service-list
+------------------+-------------------------------+------+---------+-------+----------------------------+-----------------+
|      Binary      |              Host             | Zone |  Status | State |         Updated_at         | Disabled Reason |
+------------------+-------------------------------+------+---------+-------+----------------------------+-----------------+
|  cinder-backup   |       tkstack.sample.net      | nova | enabled |   up  | 2014-08-02T11:50:04.000000 |       None      |
| cinder-scheduler |       tkstack.sample.net      | nova | enabled |   up  | 2014-08-02T11:50:07.000000 |       None      |
|  cinder-volume   |       tkstack.sample.net      | nova | enabled |  down | 2014-07-28T07:00:45.000000 |       None      |
|  cinder-volume   | tkstack.sample.net@cdot-iscsi | nova | enabled |   up  | 2014-08-02T11:50:08.000000 |       None      |
|  cinder-volume   |  tkstack.sample.net@cdot-nfs  | nova | enabled |   up  | 2014-08-02T11:50:08.000000 |       None      |
|  cinder-volume   |  tkstack.sample.net@lvmdriver | nova | enabled |  down | 2014-07-31T15:19:05.000000 |       None      |
+------------------+-------------------------------+------+---------+-------+----------------------------+-----------------+
[root@tkstack ~(keystone_admin)]#

takeshik · ‎2014-08-02

Additional Information.

When I edited from "glance_api_version=2" to "glance_api_version=1" (or comment out this parameter) in cinder.conf,

this command succeeded.

[root@tkstack cinder(keystone_admin)]# cinder create --image-id 17a544b2-1bf8-4b1c-954a-c3a405b51fbd 20
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2014-08-02T12:02:29.168455      |
| display_description |                 None                 |
|     display_name    |                 None                 |
|      encrypted      |                False                 |
|          id         | 0d8e77c5-75ed-4725-9aed-3e5609a0fe87 |
|       image_id      | 17a544b2-1bf8-4b1c-954a-c3a405b51fbd |
|       metadata      |                  {}                  |
|         size        |                  20                  |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+
[root@tkstack cinder(keystone_admin)]#

Also , I could find "Copy Offload" by netapp driver in volume.log.

2014-08-02 21:02:31.382 15478 INFO cinder.volume.drivers.netapp.nfs [req-c1cdec7c-fb37-4bb8-9845-ed8781fefde2 13d3305af9e94e888b451e3775dd09ac 51907191e78543afa754de6b69624205 - - -] Copied image to volume volume-0d8e77c5-75ed-4725-9aed-3e5609a0fe87 using regular download.

2014-08-02 21:02:31.382 15478 INFO cinder.volume.drivers.netapp.nfs [req-c1cdec7c-fb37-4bb8-9845-ed8781fefde2 13d3305af9e94e888b451e3775dd09ac 51907191e78543afa754de6b69624205 - - -] Registering image in cache img-cache-17a544b2-1bf8-4b1c-954a-c3a405b51fbd

2014-08-02 21:02:31.384 15478 INFO cinder.volume.drivers.netapp.nfs [req-c1cdec7c-fb37-4bb8-9845-ed8781fefde2 13d3305af9e94e888b451e3775dd09ac 51907191e78543afa754de6b69624205 - - -] Cloning from cache to destination img-cache-17a544b2-1bf8-4b1c-954a-c3a405b51fbd

Is this correct?

If this configuration is correct , Should I use "1" for an api version of glance in cinder.conf ?

Thanks,

rcallawa · ‎2014-08-02

Yes, 2 is the correct value for api version of Glance.

Please see the upstream documentation for all the settings/configuration that is required for copy offload:

http://docs.openstack.org/icehouse/config-reference/content/ontap-cluster-nfs.html#d6e2969

takeshik · ‎2014-08-02

I confused.

I have already read below document.

http://docs.openstack.org/icehouse/config-reference/content/ontap-cluster-nfs.html#d6e2969

When a glance_api_version = 2 , the command did not succeed.

When a glance_api_version = 1 , the command succeed.

Should I use "version 2" to use NetApp unified driver for copy offload?

However I seems it works fine with "version 1" too.

rcallawa · ‎2014-08-02

You have hit a bug in Cinder:

https://bugs.launchpad.net/cinder/+bug/1308058

This appears to be fixed in the first service release of Icehouse, 2014.1.1 - which distribution of OpenStack are you using?

takeshik · ‎2014-08-02

Thank you

I use RDO Icehouse release.

https://github.com/redhat-openstack/rdo-release

[root@tkstack glance(keystone_admin)]# rpm -qi openstack-cinder-2014.1-2.fc21.noarch

Name : openstack-cinder

Version : 2014.1

Release : 2.fc21

Architecture: noarch

Install Date: 2014年07月23日 16時36分06秒

Group : Applications/System

Size : 134169

License : ASL 2.0

Signature : RSA/SHA1, 2014年04月23日 23時08分35秒, Key ID e50be6ab0e4fbd28

Source RPM : openstack-cinder-2014.1-2.fc21.src.rpm

Build Date : 2014年04月22日 04時48分42秒

Build Host : arm02-builder11.arm.fedoraproject.org

Relocations : (not relocatable)

Packager : Fedora Project

Vendor : Fedora Project

URL : http://www.openstack.org/software/openstack-storage/

Summary : OpenStack Volume service

Description :

OpenStack Volume (codename Cinder) provides services to manage and

access block storage volumes for use by Virtual Machine instances.

[root@tkstack glance(keystone_admin)]#

rcallawa · ‎2014-08-02

It doesn't appear that Red Hat has created a new RPM for RDO on Fedora 21 yet:

https://repos.fedorapeople.org/repos/openstack/openstack-icehouse/fedora-20/

You could manually update the file /usr/lib/python2.7/site-packages/cinder/image/glance.py - line 434, to look like:

output[attr] = getattr(image, attr, None)

to see if this resolves your issue.

takeshik · ‎2014-08-02

Thanks,

After I edited a glancy.py , I tried again.

It was works fine.

By the way , What should I check in cDOT about this copy offload?

For example, clone status , snapshot show or lun show or any messages.

rcallawa · ‎2014-08-02

You can go into diag mode and look at the value of the copy_reqs counter - it should be increasing by 1 every time a copy offload operation is successful.

login: admin

Password:

rcallawacluster-1::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.

Do you want to continue? {y|n}: y

rcallawacluster-1::*> statistics start -object copy_manager

Statistics collection is being started for Sample-id: sample_12

rcallawacluster-1::*> statistics show -object copy_manager -instance rcallawa-nfs-vserver -counter copy_reqs

Object: copy_manager

Instance: rcallawa-nfs-vserver

Start-time: 8/2/2014 14:43:56

End-time: 8/2/2014 14:44:34

Cluster: rcallawacluster-1

Number of Constituents: 2 (complete_aggregation)

Counter Value

-------------------------------- --------------------------------

copy_reqs 0

takeshik · ‎2014-08-03

Thank you

I will try it again .