Icehouse Cluster mode Driver can't create Volume (No valid host was found)

[GERNETAPP]

Icehouse Cluster mode Driver can't create Volume (No valid host was found)

The cinder.conf file is the following:

[DEFAULT]

enabled_backends=iscsi_emc_vmax_1,3250direct1

...

[3250direct1]

volume_backend_name=3250direct1

volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver

netapp_server_hostname=xx.xx.xx.xx -> cluster mgmt ip addr

netapp_server_port=80

netapp_storage_protocol=iscsi

netapp_storage_family=ontap_cluster

netapp_login=admin    

netapp_password=xxxxxx

netapp_vserver=openstackVS1

When i try to create a volume on backend named "3250direct1", i received the following error message on cinder-scheduler.log file:

The command issued to create the volume is "cinder create --volume-type 3250direct1 --display-name test 1"

...

2014-06-06 11:55:24.481 24590 ERROR cinder.scheduler.flows.create_volume [req-52c129c0-8f1c-468e-9566-8ef287b3e31b 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Failed to schedule_create_volume: No valid host was found.

2014-06-06 11:55:24.484 24590 DEBUG cinder.volume.flows.common [req-52c129c0-8f1c-468e-9566-8ef287b3e31b 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Updating volume: 1461f265-ca41-43e6-9eb4-5cf958d8c421 with {'status': 'error'} due to: No valid host was found.  error_out_volume /usr/lib/python2.7/dist-packages/cinder/volume/flows/common.py:87

...

Also I noticed that when trying to update a volume it also display an error msg:

...

3ab47e19d21c4672973e57e7a8c85c83 - - -] Updating volume: 22f86318-5668-466e-b80a-9452314fa5f6 with {'status': 'error'} due to: No valid host was found.  error_out_volume /usr/lib/python2.7/dist-packages/cinder/volume/flows/common.py:87

...

On cinder-volume.log we can see that the driver is loaded successfully. Anyone had this kind of problems before? It's really painful since we already try to use the LIF mgmt port of the Vserver to connect and we are getting the same result, also I forgot to mention that every time that we make any change on the cinder.conf file, we restart all the services (api,scheduler and volume).

Also another strange thing is that we have running this cluster-mode array with the same settings on a Grizzly Cinder environment.

Thanks in advance,

German

[rcallawa]

How have you defined the volume type? As an admin, could you paste in a reply the output of executing the following CLI command:

cinder extra-specs-list

[GERNETAPP]

Hi how are you? Thanks a lot for the quick response. Sure, here's the output:

root@icehouse-cinder-dev01:~# cinder extra-specs-list

+--------------------------------------+------------+------------------------------------------+

|                  ID                  |    Name    |               extra_specs                |

+--------------------------------------+------------+------------------------------------------+

| 18eb3e72-4441-4f59-8201-04d810c19e99 |    emc     | {u'volume_backend': u'iscsi_emc_vmax_1'} |

| b016ff8c-47de-4a1b-94d5-80a639d4871d | netapp3250 |   {u'volume_backend': u'3250direct1'}    |

+--------------------------------------+------------+------------------------------------------+

[rcallawa]

The name of your volume type is incorrect in your "cinder create" command. You had

cinder create --volume-type 3250direct1 --display-name test 1

but your volume type is named 'netapp3250'... try this

cinder create --volume-type netapp3250 --display-name test 1

Let me know if that is successful.

[GERNETAPP]

Oh sorry i forgot to update the first part of the question, we are using the correct parameter.

...

[3250direct1]

volume_backend_name=netapp3250

volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver

netapp_server_hostname=xx.xx.xx.xx -> cluster mgmt ip addr

netapp_server_port=80

netapp_storage_protocol=iscsi

netapp_storage_family=ontap_cluster

netapp_login=admin    

netapp_password=xxxxxx

netapp_vserver=openstackVS1

But we are still receiving the same error message.

[akerr]

The volume type is defined incorrectly:

| b016ff8c-47de-4a1b-94d5-80a639d4871d | netapp3250 |   {u'volume_backend': u'3250direct1'}    |

The extra spec "volume_backend" should match the same line as in your cinder.conf file:

volume_backend_name=netapp3250

So you'll need to redefine the volume type with "cinder type-key b016ff8c-47de-4a1b-94d5-80a639d4871d set volume_backend=netapp3250"

After that the output should look like this:

| b016ff8c-47de-4a1b-94d5-80a639d4871d | netapp3250 |   {u'volume_backend': u'netapp3250'}    |

You can then do as Bob mentioned and create the volume with "cinder create --volume-type netapp3250 --display-name test 1"

[rcallawa]

If Andrew's latest suggestion still doesn't fix the problem, please paste in a full excerpt from the cinder-scheduler logs.

[GERNETAPP]

It didn't work, here's the complete process:

mysql drop cinder; mysql create cinder;

cinder-manage db sync

cinder type-create 3250direct1

cinder type-create emc

cinder type-key 3250direct1 set volume_backend=3250direct1

cinder type-key emc set volume_backend=iscsi_emc_vmax_1

root@icehouse-cinder-dev01:~# cinder type-list

+--------------------------------------+-------------+

|                  ID                  |     Name    |

+--------------------------------------+-------------+

| c4f674cb-c532-4e29-92f8-55ca735729d6 | 3250direct1 |

| f0a853fd-a4de-4d9e-989f-91718e205bd8 |     emc     |

+--------------------------------------+-------------+

root@icehouse-cinder-dev01:~# cinder extra-specs-list

+--------------------------------------+-------------+------------------------------------------+

|                  ID                  |     Name    |               extra_specs                |

+--------------------------------------+-------------+------------------------------------------+

| c4f674cb-c532-4e29-92f8-55ca735729d6 | 3250direct1 |   {u'volume_backend': u'3250direct1'}    |

| f0a853fd-a4de-4d9e-989f-91718e205bd8 |     emc     | {u'volume_backend': u'iscsi_emc_vmax_1'} |

+--------------------------------------+-------------+------------------------------------------+

root@icehouse-cinder-dev01:~# cinder create --volume-type 3250direct1  --display-name test 1

And the output from the log files:

# cinder-api.log

2014-06-06 17:05:15.915 28244 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:15.919 28244 DEBUG oslo.messaging._drivers.amqp [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] UNIQUE_ID is 1ef091b9fbf244f6b4a48fb1ab65edbb. _add_unique_id /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqp.py:338

2014-06-06 17:05:15.921 28244 DEBUG oslo.messaging._drivers.amqp [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Pool creating new connection create /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqp.py:70

2014-06-06 17:05:15.969 28244 INFO oslo.messaging._drivers.impl_rabbit [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Connected to AMQP server on 10.111.81.31:5672

2014-06-06 17:05:15.983 28244 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:15.985 28244 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:15.987 28244 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_flow_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:15.990 28244 AUDIT cinder.api.v1.volumes [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] vol={'migration_status': None, 'availability_zone': 'nova', 'terminated_at': None, 'reservations': ['7e8a0945-5c88-40f0-bd2f-e387b321bb6b', 'd3068fbc-be22-4797-91b5-957b6f6232e4', '6ad34cc5-07c7-4e4a-82ba-2374e511909f', '2c413e62-e002-4ba7-bcff-1ab9d240db21'], 'updated_at': None, 'provider_geometry': None, 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': 'd6913cf6-2192-4d1e-a3c3-9346a318f7b9', 'size': 1, 'user_id': u'912af7e8976448df80894348d8158d39', 'attach_time': None, 'attached_host': None, 'display_description': None, 'volume_admin_metadata': [], 'encryption_key_id': None, 'project_id': u'3ab47e19d21c4672973e57e7a8c85c83', 'launched_at': None, 'scheduled_at': None, 'status': 'creating', 'volume_type_id': u'c4f674cb-c532-4e29-92f8-55ca735729d6', 'deleted': False, 'provider_location': None, 'host': None, 'source_volid': None, 'provider_auth': None, 'display_name': u'test', 'instance_uuid': None, 'bootable': False, 'created_at': datetime.datetime(2014, 6, 6, 21, 5, 15, 764214), 'attach_status': 'detached', 'volume_type': <cinder.db.sqlalchemy.models.VolumeTypes object at 0x7f0c60e94210>, '_name_id': None, 'volume_metadata': [], 'metadata': {}}

2014-06-06 17:05:15.994 28244 INFO cinder.api.openstack.wsgi [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] http://10.111.81.30:8776/v1/3ab47e19d21c4672973e57e7a8c85c83/volumes returned with HTTP 200

2014-06-06 17:05:15.997 28244 INFO eventlet.wsgi.server [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] 10.111.81.30 - - [06/Jun/2014 17:05:15] "POST /v1/3ab47e19d21c4672973e57e7a8c85c83/volumes HTTP/1.1" 200 605 1.973848

cinder-scheduler.log

2014-06-06 17:05:15.982 28215 DEBUG oslo.messaging._drivers.amqpdriver [-] received {u'_context_domain': None, u'_context_request_id': u'req-c0b96cd4-1a69-4623-8018-e0ebf819c23f', u'_context_quota_class': None, u'_context_service_catalog': [{u'endpoints': [{u'adminURL': u'http://10.111.81.31:8774/v2/3ab47e19d21c4672973e57e7a8c85c83', u'region': u'regionOne', u'internalURL': u'http://10.111.81.31:8774/v2/3ab47e19d21c4672973e57e7a8c85c83', u'id': u'7df3235de660451db6c9c047ffa89ab2', u'publicURL': u'http://10.111.81.31:8774/v2/3ab47e19d21c4672973e57e7a8c85c83'}], u'endpoints_links': [], u'type': u'compute', u'name': u'nova'}], u'_context_auth_token': '<SANITIZED>', u'_context_user': u'912af7e8976448df80894348d8158d39', u'_context_user_id': u'912af7e8976448df80894348d8158d39', u'_context_is_admin': True, u'version': u'1.2', u'_context_project_domain': None, u'_context_timestamp': u'2014-06-06T21:05:14.809729', u'method': u'create_volume', u'_context_remote_address': u'10.111.81.30', u'_context_roles': [u'_member_', u'admin'], u'args': {u'request_spec': {u'volume_properties': {u'status': u'creating', u'volume_type_id': u'c4f674cb-c532-4e29-92f8-55ca735729d6', u'user_id': u'912af7e8976448df80894348d8158d39', u'availability_zone': u'nova', u'reservations': [u'7e8a0945-5c88-40f0-bd2f-e387b321bb6b', u'd3068fbc-be22-4797-91b5-957b6f6232e4', u'6ad34cc5-07c7-4e4a-82ba-2374e511909f', u'2c413e62-e002-4ba7-bcff-1ab9d240db21'], u'snapshot_id': None, u'attach_status': u'detached', u'display_description': None, u'volume_metadata': [], u'encryption_key_id': None, u'source_volid': None, u'volume_admin_metadata': [], u'display_name': u'test', u'project_id': u'3ab47e19d21c4672973e57e7a8c85c83', u'id': u'd6913cf6-2192-4d1e-a3c3-9346a318f7b9', u'size': 1, u'metadata': {}}, u'volume_type': {u'name': u'3250direct1', u'qos_specs_id': None, u'deleted': False, u'created_at': u'2014-06-06T21:03:28.000000', u'updated_at': None, u'extra_specs': {u'volume_backend': u'3250direct1'}, u'deleted_at': None, u'id': u'c4f674cb-c532-4e29-92f8-55ca735729d6'}, u'image_id': None, u'snapshot_id': None, u'source_volid': None, u'volume_id': u'd6913cf6-2192-4d1e-a3c3-9346a318f7b9'}, u'volume_id': u'd6913cf6-2192-4d1e-a3c3-9346a318f7b9', u'filter_properties': {}, u'topic': u'cinder-volume', u'image_id': None, u'snapshot_id': None}, u'_unique_id': u'1ef091b9fbf244f6b4a48fb1ab65edbb', u'_context_project_name': u'admin', u'_context_read_deleted': u'no', u'_context_user_identity': u'912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -', u'_context_tenant': u'3ab47e19d21c4672973e57e7a8c85c83', u'_context_project_id': u'3ab47e19d21c4672973e57e7a8c85c83', u'_context_user_domain': None} _safe_log /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py:290

2014-06-06 17:05:15.985 28215 DEBUG oslo.messaging._drivers.amqp [-] unpacked context: {'tenant': u'3ab47e19d21c4672973e57e7a8c85c83', 'domain': None, 'project_name': u'admin', 'user_id': u'912af7e8976448df80894348d8158d39', 'roles': [u'_member_', u'admin'], 'user_identity': u'912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -', 'project_domain': None, 'timestamp': u'2014-06-06T21:05:14.809729', 'auth_token': '<SANITIZED>', 'remote_address': u'10.111.81.30', 'quota_class': None, 'user_domain': None, 'is_admin': True, 'user': u'912af7e8976448df80894348d8158d39', 'service_catalog': [{u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://10.111.81.31:8774/v2/3ab47e19d21c4672973e57e7a8c85c83', u'region': u'regionOne', u'publicURL': u'http://10.111.81.31:8774/v2/3ab47e19d21c4672973e57e7a8c85c83', u'internalURL': u'http://10.111.81.31:8774/v2/3ab47e19d21c4672973e57e7a8c85c83', u'id': u'7df3235de660451db6c9c047ffa89ab2'}], u'type': u'compute', u'name': u'nova'}], 'request_id': u'req-c0b96cd4-1a69-4623-8018-e0ebf819c23f', 'project_id': u'3ab47e19d21c4672973e57e7a8c85c83', 'read_deleted': u'no'} _safe_log /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py:290

2014-06-06 17:05:15.988 28215 WARNING cinder.context [-] Arguments dropped when creating context: {'user': u'912af7e8976448df80894348d8158d39', 'tenant': u'3ab47e19d21c4672973e57e7a8c85c83', 'user_identity': u'912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -'}

2014-06-06 17:05:15.991 28215 DEBUG stevedore.extension [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] found extension EntryPoint.parse('default = taskflow.engines.action_engine.engine:SingleThreadedActionEngine') _load_plugins /usr/lib/python2.7/dist-packages/stevedore/extension.py:156

2014-06-06 17:05:16.179 28215 DEBUG stevedore.extension [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] found extension EntryPoint.parse('serial = taskflow.engines.action_engine.engine:SingleThreadedActionEngine') _load_plugins /usr/lib/python2.7/dist-packages/stevedore/extension.py:156

2014-06-06 17:05:16.181 28215 DEBUG stevedore.extension [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] found extension EntryPoint.parse('parallel = taskflow.engines.action_engine.engine:MultiThreadedActionEngine') _load_plugins /usr/lib/python2.7/dist-packages/stevedore/extension.py:156

2014-06-06 17:05:16.183 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.save_both' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.185 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.187 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Starting to flatten 'taskflow.patterns.linear_flow.Flow: volume_create_scheduler; 2' _pre_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:154

2014-06-06 17:05:16.188 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Starting to flatten 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create==1.0' _pre_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:154

2014-06-06 17:05:16.190 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Finished flattening 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create==1.0' _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:159

2014-06-06 17:05:16.191 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Translated 'cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create==1.0' into a graph: _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:164

2014-06-06 17:05:16.193 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Name: cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.194 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Type: DiGraph _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.196 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Frozen: False _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.197 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Nodes: 1 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.198 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]    - cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create==1.0 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.199 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Edges: 0 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.200 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Density: 0.000 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.201 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Cycles: 0 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.203 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Starting to flatten 'cinder.scheduler.flows.create_volume.schedule_create_volume==1.0' _pre_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:154

2014-06-06 17:05:16.205 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Finished flattening 'cinder.scheduler.flows.create_volume.schedule_create_volume==1.0' _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:159

2014-06-06 17:05:16.206 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Translated 'cinder.scheduler.flows.create_volume.schedule_create_volume==1.0' into a graph: _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:164

2014-06-06 17:05:16.208 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Name: cinder.scheduler.flows.create_volume.schedule_create_volume _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.209 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Type: DiGraph _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.210 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Frozen: False _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.211 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Nodes: 1 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.213 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]    - cinder.scheduler.flows.create_volume.schedule_create_volume==1.0 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.214 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Edges: 0 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.215 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Density: 0.000 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.216 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Cycles: 0 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.217 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Finished flattening 'taskflow.patterns.linear_flow.Flow: volume_create_scheduler; 2' _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:159

2014-06-06 17:05:16.219 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Translated 'taskflow.patterns.linear_flow.Flow: volume_create_scheduler; 2' into a graph: _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:164

2014-06-06 17:05:16.220 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Name: volume_create_scheduler _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.222 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Type: DiGraph _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.223 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Frozen: False _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.224 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Nodes: 2 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.225 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]    - cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create==1.0 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.226 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]    - cinder.scheduler.flows.create_volume.schedule_create_volume==1.0 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.227 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Edges: 1 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.228 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]    cinder.scheduler.flows.create_volume.ExtractSchedulerSpecTask;volume:create==1.0 -> cinder.scheduler.flows.create_volume.schedule_create_volume==1.0 ({'flatten': True}) _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.230 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Density: 0.500 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.231 28215 DEBUG taskflow.utils.flow_utils [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -]  Cycles: 0 _post_item_flatten /usr/lib/python2.7/dist-packages/taskflow/utils/flow_utils.py:167

2014-06-06 17:05:16.233 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.save_both' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.234 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.save_both' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.236 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_flow_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.238 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.239 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.241 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.243 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.244 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.246 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.262 28215 DEBUG cinder.openstack.common.scheduler.filters.capabilities_filter [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] host 'icehouse-cinder-dev01@iscsi_emc_vmax_1': free_capacity_gb: unknown fails resource_type extra_specs requirements host_passes /usr/lib/python2.7/dist-packages/cinder/openstack/common/scheduler/filters/capabilities_filter.py:68

2014-06-06 17:05:16.264 28215 DEBUG cinder.openstack.common.scheduler.filters.capabilities_filter [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] host 'icehouse-cinder-dev01@3250direct1': free_capacity_gb: 102 fails resource_type extra_specs requirements host_passes /usr/lib/python2.7/dist-packages/cinder/openstack/common/scheduler/filters/capabilities_filter.py:68

2014-06-06 17:05:16.268 28215 DEBUG oslo.messaging._drivers.amqp [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] UNIQUE_ID is 31a8a045bbef4ade925fe259ec186bf9. _add_unique_id /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqp.py:338

2014-06-06 17:05:16.287 28215 ERROR cinder.scheduler.flows.create_volume [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Failed to schedule_create_volume: No valid host was found.

2014-06-06 17:05:16.289 28215 DEBUG cinder.volume.flows.common [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] Updating volume: d6913cf6-2192-4d1e-a3c3-9346a318f7b9 with {'status': 'error'} due to: No valid host was found.  error_out_volume /usr/lib/python2.7/dist-packages/cinder/volume/flows/common.py:87

2014-06-06 17:05:16.398 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.399 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.401 28215 DEBUG taskflow.storage [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_flow_detail' _with_connection /usr/lib/python2.7/dist-packages/taskflow/storage.py:88

2014-06-06 17:05:16.403 28215 DEBUG oslo.messaging._drivers.amqp [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] UNIQUE_ID is f135053d90ce465b8a9aff6af1b8d67e. _add_unique_id /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqp.py:338

2014-06-06 17:05:16.406 28215 DEBUG oslo.messaging._drivers.amqp [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] UNIQUE_ID is 6871a63b3836431eb01074e622158ae4. _add_unique_id /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqp.py:338

[rcallawa · Accepted solution]

Ah, I see a typo in your extra specs.

cinder type-key 3250direct1 set volume_backend=3250direct1

cinder type-key emc set volume_backend=iscsi_emc_vmax_1

should be

cinder type-key 3250direct1 set volume_backend_name=3250direct1

cinder type-key emc set volume_backend_name=iscsi_emc_vmax_1

These lines from cinder-scheduler.log denote that it tried to match the volume type against both backends, but failed:

2014-06-06 17:05:16.262 28215 DEBUG cinder.openstack.common.scheduler.filters.capabilities_filter [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] host 'icehouse-cinder-dev01@iscsi_emc_vmax_1': free_capacity_gb: unknown fails resource_type extra_specs requirements host_passes /usr/lib/python2.7/dist-packages/cinder/openstack/common/scheduler/filters/capabilities_filter.py:68

2014-06-06 17:05:16.264 28215 DEBUG cinder.openstack.common.scheduler.filters.capabilities_filter [req-c0b96cd4-1a69-4623-8018-e0ebf819c23f 912af7e8976448df80894348d8158d39 3ab47e19d21c4672973e57e7a8c85c83 - - -] host 'icehouse-cinder-dev01@3250direct1': free_capacity_gb: 102 fails resource_type extra_specs requirements host_passes /usr/lib/python2.7/dist-packages/cinder/openstack/common/scheduler/filters/capabilities_filter.py:68

[GERNETAPP]

Thanks a lot!!!! It was that! as soon as changed it, it started to work perfect! Best regards!!!