Re: cinder iscsi driver initialization failed

MarginHu2017 · ‎2017-05-31

Hi Guys,

I'm using Openstack community upstream Ocata with Netapp Storage FAS8040, and found cinder iscsi driver initialization failed, the following is configuration and log,

you can check detail info from the attachement.

cinder.conf

[NetappIscsiBackend]
volume_backend_name = NetappIscsiBackend
volume_driver = cinder.volume.drivers.netapp.common.NetAppDriver
netapp_server_hostname = cloud-sz-netapp-b14-02.sz.cloud.genomics.cn
netapp_transport_type = https
netapp_server_port = 443
netapp_storage_protocol = iscsi
netapp_storage_family = ontap_cluster
netapp_login = admin
netapp_password = netapp123
netapp_vserver = svm_iscsi
#netapp_vserver = ntap01
max_oversubscription_ratio = 1.0
reserved_percentage = 5
use_multipath_for_image_xfer = True

cinder-volume.log

ntap01::> security certificate show
Vserver    Serial Number   Common Name                            Type
---------- --------------- -------------------------------------- ------------
ntap01     01              cloud-sz-netapp-b14-02.sz.cloud.genomics.cn
                                                                  server
    Certificate Authority: cloud-sz-netapp-b14-02.sz.cloud.genomics.cn
          Expiration Date: Fri May 28 08:34:44 2027

ntap01     01              cloud-sz-netapp-b14-02.sz.cloud.genomics.cn
                                                                  server-chain
    Certificate Authority: cloud-sz-netapp-b14-02.sz.cloud.genomics.cn
          Expiration Date: -

svm_iscsi 01              cloud-sz-netapp-b14-02.sz.cloud.genomics.cn
                                                                  server
    Certificate Authority: cloud-sz-netapp-b14-02.sz.cloud.genomics.cn
          Expiration Date: Fri May 28 08:34:44 2027

svm_iscsi 01              cloud-sz-netapp-b14-02.sz.cloud.genomics.cn
                                                                  server-chain
    Certificate Authority: cloud-sz-netapp-b14-02.sz.cloud.genomics.cn
          Expiration Date: -

svm_nfs    054F4C7779F6B2 svm_nfs                                server
    Certificate Authority: svm_nfs
          Expiration Date: Sat May 12 12:52:04 2018

5 entries were displayed.

ntap01::> security ssl show
          Serial                                         Server Client
Vserver   Number Common Name                             Enabled Enabled
--------- ------ --------------------------------------- ------- -------
ntap01    01     cloud-sz-netapp-b14-02.sz.cloud.genomics.cn
                                                         true    false
Certificate Authority: cloud-sz-netapp-b14-02.sz.cloud.genomics.cn

svm_iscsi 01     cloud-sz-netapp-b14-02.sz.cloud.genomics.cn
                                                         true    false
Certificate Authority: cloud-sz-netapp-b14-02.sz.cloud.genomics.cn

svm_nfs   054F4C7779F6B2
                 svm_nfs                                 true    false
Certificate Authority: svm_nfs

3 entries were displayed.

ntap01::>

MarginHu2017 · ‎2017-05-31

2017-05-31 22:26:52.364 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] rate_limit_interval = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.364 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] replication_device = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api [req-d095a8d4-ee51-4ce0-96b3-db061026b6aa - - - - -] Error communicating with NetApp filer.
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api Traceback (most recent call last):
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 190, in send_http_request
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api response = self._opener.open(request)
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api File "/usr/lib64/python2.7/urllib2.py", line 431, in open
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api response = self._open(req, data)
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api '_open', req)
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api result = func(*args)
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api context=self._context, check_hostname=self._check_hostname)
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api raise URLError(err)
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api

MarginHu2017 · ‎2017-05-31

2017-05-31 22:26:52.364 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] rate_limit_interval            = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.364 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] replication_device             = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api [req-d095a8d4-ee51-4ce0-96b3-db061026b6aa - - - - -] Error communicating with NetApp filer.
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api Traceback (most recent call last):
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 190, in send_http_request
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api     response = self._opener.open(request)
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api   File "/usr/lib64/python2.7/urllib2.py", line 431, in open
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api     response = self._open(req, data)
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api   File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api     '_open', req)
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api   File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api     result = func(*args)
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api   File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api     context=self._context, check_hostname=self._check_hostname)
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api   File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api     raise URLError(err)
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>
2017-05-31 22:26:52.363 31 ERROR cinder.volume.drivers.netapp.dataontap.client.api
2017-05-31 22:26:52.364 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] report_discard_supported       = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.365 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] report_interval                = 10 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.365 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] reservation_expire             = 86400 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.365 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] reserved_percentage            = 0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.365 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] rootwrap_config                = /etc/cinder/rootwrap.conf log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.366 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] rpc_backend                    = rabbit log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.366 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] rpc_response_timeout           = 60 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.366 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] run_external_periodic_tasks    = True log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.367 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] scheduler_manager              = cinder.scheduler.manager.SchedulerManager log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.367 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] scst_target_driver             = iscsi log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.367 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] scst_target_iqn_name           = None log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.367 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] service_down_time              = 60 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.368 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] snapshot_name_template         = snapshot-%s log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.368 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] ssh_hosts_key_file             = /var/lib/cinder/ssh_known_hosts log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.368 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] state_path                     = /var/lib/cinder log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.369 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] storage_availability_zone      = nova log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.369 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] storage_protocol               = iscsi log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager [req-d095a8d4-ee51-4ce0-96b3-db061026b6aa - - - - -] Failed to initialize driver.
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager Traceback (most recent call last):
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 431, in init_host
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     self.driver.do_setup(ctxt)
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/iscsi_cmode.py", line 41, in do_setup
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     self.library.do_setup(context)
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     return f(*args, **kwargs)
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     return f(*args, **kwargs)
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/block_cmode.py", line 73, in do_setup
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     self.failed_over_backend_name or self.backend_name)
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/utils/utils.py", line 75, in get_client_for_backend
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     trace=utils.TRACE_API)
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     return f(*args, **kwargs)
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     return f(*args, **kwargs)
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_cmode.py", line 52, in __init__
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     (major, minor) = self.get_ontapi_version(cached=False)
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     return f(*args, **kwargs)
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_base.py", line 70, in get_ontapi_version
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     res = self.connection.invoke_successfully(ontapi_version, False)
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 210, in invoke_successfully
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     result = self.send_http_request(na_element, enable_tunneling)
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 837, in trace_api_logging_wrapper
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     return f(*args, **kwargs)
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 195, in send_http_request
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager     raise NaApiError('Unexpected error')
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager NaApiError: NetApp API failed. Reason - Unexpected error:unknown
2017-05-31 22:26:52.367 31 ERROR cinder.volume.manager
2017-05-31 22:26:52.369 6 DEBUG oslo_service.service [req-86278db3-b37e-4fd6-a4ee-6914a4c7d62c - - - - -] strict_ssh_host_key_policy     = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2740

diez · ‎2017-06-01

Have you tried the steps from the following document?

http://netapp.io/2017/02/15/use-certificate-verification-netapp-ontap-openstack-cinder-driver/

MarginHu2017 · ‎2017-06-06

yes, I have tried but the issue still appeared, maybe I do something wrong because my os is centos7 not Ubuntu.

Last , I enable http protocol on the netapp storage , and use http instead of https , this is only workaround.

I hope netapp offer a detail solution for it.

MarginHu2017 · ‎2017-06-10

Hi Netapper :

any progress ?