Getting 13003:Insufficient privileges after more than a year running with no issues

gasparuben · ‎2015-11-02

Hello,

I am trying to debug this situation: our openstack people deployed kilo a while ago. Since apparently last Friday that they noticed that after resizing a lun they couldnt mount/attach it back to same VM.

In production I am running kilo on a clustered ONTAP 8.2.2P1, we get this error while trying to reattach:

2015-11-02 09:34:36.493 21340 ERROR oslo_messaging._drivers.common [req-62d7c63f-efea-4417-893f-b81c7cc5c8cd bukowiec 4d679467-f828-41bc-90fa-ef8633594a6f - - -] Returning exception Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: NetApp API failed. Reason - 13003:Insufficient privileges: user 'cinder_admin' does not have read access to this resource to caller

cinder_admin have following privilages as per doc:http://netapp.github.io/openstack-deploy-ops-guide/kilo/openstack-deployment-ops-guide.pdf

rac51::*> security login role show -vserver rac51 -role cinder_cluster
Role Command/ Access
Vserver Name Directory Query Level
---------- ------------- --------- ----------------------------------- --------
rac51 cinder_cluster
DEFAULT none
rac51 cinder_cluster
event all
rac51 cinder_cluster
lun readonly
rac51 cinder_cluster
lun create all
rac51 cinder_cluster
lun delete all
rac51 cinder_cluster
lun igroup readonly
rac51 cinder_cluster
lun igroup add all
rac51 cinder_cluster
lun igroup create all
rac51 cinder_cluster
lun igroup modify all
rac51 cinder_cluster
lun igroup show all
rac51 cinder_cluster
lun map all

Role Command/ Access
Vserver Name Directory Query Level
---------- ------------- --------- ----------------------------------- --------
rac51 cinder_cluster
lun mapped readonly
rac51 cinder_cluster
lun modify all
rac51 cinder_cluster
lun resize all
rac51 cinder_cluster
lun show all
rac51 cinder_cluster
lun unmap all
rac51 cinder_cluster
security readonly
rac51 cinder_cluster
snapmirror readonly
rac51 cinder_cluster
storage aggregate readonly
rac51 cinder_cluster
storage disk readonly
rac51 cinder_cluster
version all

Role Command/ Access
Vserver Name Directory Query Level
---------- ------------- --------- ----------------------------------- --------
rac51 cinder_cluster
volume readonly
rac51 cinder_cluster
volume efficiency readonly
rac51 cinder_cluster
volume file clone create all
rac51 cinder_cluster
vserver readonly
rac51 cinder_cluster
vserver iscsi readonly
rac51 cinder_cluster
vserver iscsi interface readonly
27 entries were displayed.

Sorry for the formating.

One of my colleagues has done a test setup of openstack and I have done the same on a test ONTAP cluster, this one running ONTAP 8.3.1. I get similar errors:

2015-11-02 16:53:56.222 17313 WARNING cinder.volume.drivers.netapp.dataontap.client.client_base [req-eec18ed9-147e-4a8d-8943-9ad895f69497 9ad073f097c347509aee2414c0021f27 81b42e48d34446da9688d691078cbcd6 - - -] Error mapping LUN. Code :13003, Message:Insufficient privileges: user 'ois_admin' does not have write access to this resource
2015-11-02 16:53:56.459 17313 ERROR cinder.volume.manager [req-eec18ed9-147e-4a8d-8943-9ad895f69497 9ad073f097c347509aee2414c0021f27 81b42e48d34446da9688d691078cbcd6 - - -] Unable to fetch connection information from backend: NetApp API failed. Reason - 13003:Insufficient privileges: user 'ois_admin' does not have write access to this resource
2015-11-02 16:53:56.461 17313 ERROR oslo_messaging.rpc.dispatcher [req-eec18ed9-147e-4a8d-8943-9ad895f69497 9ad073f097c347509aee2414c0021f27 81b42e48d34446da9688d691078cbcd6 - - -] Exception during message handling: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: NetApp API failed. Reason - 13003:Insufficient privileges: user 'ois_admin' does not have write access to this resource

Again while trying to attach the volume to a vm:

The security is as per doc again:

c02::*> security login role show -vserver c02 -role openstach
Role Command/ Access
Vserver Name Directory Query Level
---------- ------------- --------- ----------------------------------- --------
c02 openstach DEFAULT none
c02 openstach event all
c02 openstach lun readonly
c02 openstach lun create all
c02 openstach lun delete all
c02 openstach lun igroup readonly
c02 openstach lun igroup add all
c02 openstach lun igroup create all
c02 openstach lun igroup modify all
c02 openstach lun igroup show all
c02 openstach lun map all
c02 openstach lun mapped readonly
c02 openstach lun modify all
c02 openstach lun resize all
c02 openstach lun show all
c02 openstach lun unmap all
c02 openstach security readonly
c02 openstach snapmirror readonly
c02 openstach storage aggregate readonly
c02 openstach storage disk readonly
c02 openstach version all
c02 openstach volume readonly
c02 openstach volume efficiency readonly
c02 openstach volume file clone create all
c02 openstach vserver readonly
c02 openstach vserver iscsi readonly
c02 openstach vserver iscsi interface readonly
27 entries were displayed.

Please could you let me know what do you think, which permissions I am missing. Still astonished this started to fail now, this operations has been done plenty of times in the past.

THank you

gasparuben · ‎2015-11-03

Just to add more information.
Both ois_admin and cinder_admin are cluster accounts like:

security login create -username cinder_admin -application ontapi -authmethod password -role cinder_cluster

This as defined in the documentation: http://netapp.github.io/openstack-deploy-ops-guide/kilo/openstack-deployment-ops-guide.pdf

Please let us know if you need further evidence.
Thank you

gasparuben · ‎2015-11-04

Further analysis of my colleages in Openstack has found that the call that is not working is:

'<netapp xmlns="http://www.netapp.com/filer/admin" version="1.31" vfiler="vsiscsi"><lun-map><path>/vol/openstack_vol01/volume-babd5700-2ebb-48c6-ae27-667ba167b209</path><initiator-group>openstack-573d9b2b-ae32-457f-8227-119707531793</initiator-group></lun-map></netapp>'

The response that we get is: NetApp API failed. Reason - 13003:Insufficient privileges .

Please could you advice. As you can see above the account should have that privilege by means of a role.

Thank you

dcain · ‎2015-11-04

Can you try the following to see if it makes a difference?

c02::*> security login role modify -role openstach -vserver c02 -cmddirname "lun mapped" -access all

It would probably be best to open a support case via http://mysupport.netapp.com to ensure that this is driven in a timely manner, should the above command not help. I cannot explain why this worked for so long and then suddenly stopped.

gasparuben · ‎2015-11-05

Thanks for your reply.

I still get the same error. I opened in the past a case on mysupport and I was redirected to the NetApp community. I can always try for sure.

If you need further evidencies please let me know.

I have open case 2005973370.

cschnidr · ‎2015-11-06

Hi Ruben

I have no real openstack knowhow, but the call looks like a 7-Mode call to me and you are saying you run on cDOT. Anything you can change there?

<netapp xmlns="http://www.netapp.com/filer/admin" version="1.31" vfiler="vsiscsi"><lun-map><path>/vol/openstack_vol01/volume-babd5700-2ebb-48c6-ae27-667ba167b209</path><initiator-group>openstack-573d9b2b-ae32-457f-8227-119707531793</initiator-group></lun-map></netapp>'

The response that we get is: NetApp API failed. Reason - 13003:Insufficient privileges .

Regards

Christoph

gasparuben · ‎2015-11-06

Just checking but it looks like the cinder.conf is good:

2896 [netapp02]
2897 volume_backend_name=netapp
2898 volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
2899 netapp_server_hostname=10.X.X.X
2900 netapp_server_port=443
2901 netapp_storage_protocol=iscsi
2902 netapp_storage_family=ontap_cluster
2903 netapp_login=vsadmin
2904 netapp_password=XXXXXX
2905 netapp_vserver=vsiscsi
2906 netapp_size_multiplier=1.0
2907 reserved_percentage=5
2908 #use_multipath_for_image_xfer=True
2909 netapp_transport_type=https
2910 nfs_shares_config=/etc/cinder/shares.conf
2911 netapp_eseries_host_type=linux_dm_mp
2912 netapp_storage_pools=
2913 expiry_thres_minutes=720
2914 netapp_vfiler=
2915 thres_avl_size_perc_stop=60
2916 netapp_copyoffload_tool_path=
2917 thres_avl_size_perc_start=20
2918 netapp_controller_ips=
2919 netapp_volume_list=
2920 netapp_webservice_path=/devmgr/v2
2921 netapp_partner_backend_name=

I have tried (together with my Openstack colleague) with a vsadmin account and a cluster account with full rights account. With both it worked. It still doesnt work with a cluster account with svm scope as per documentation.

It's clearly a permission issue.

Thank you,

Ruben

DougCarman · ‎2015-11-18

Ruben,

I think we need to clarify that there are different environments involved in this thread. On the storage side, the NetApp cluster "rac51" provides the back end storage to your production OpenStack environment, where the NetApp cluster c02 provides storage to your test environment. On the nodes of cluster c02, we observed a clear indication of an issue with permission to a specific API call:

0000004c.001e2477 02290ef2 Mon Nov 02 2015 14:33:17  01:00 [kern_command-history:info:934] ontapi :: 128.142.140.42 :: ois_admin :: <netapp xmlns="http://www.netapp.com/filer/admin" version="1.31" vfiler="vsiscsi"><lun-map><path>/vol/openstack_vol01/volume-49196581-99df-4351-b689-9351af0a4f4f</path><initiator-group>openstack-bd 18dce0-0922-4269-8a74-2520a902a8c5</initiator-group></lun-map></netapp> :: Pending
0000004c.001e2479 02290ef2 Mon Nov 02 2015 14:33:17  01:00 [kern_command-history:info:934] ontapi :: 128.142.140.42 :: ois_admin :: Insufficient privileges: user 'ois_admin' does not have write access to this resource :: ONTAPI :: Error

In this case the call to "lun map" results in "Insufficient privileges".

On the "rac51" cluster, we observe an entirely different error returned for the "lun map" API call:

00000009.00c2f308 11a76240 Mon Nov 02 2015 09:34:31  01:00 [kern_command-history:info:1040] ontapi :: 188.184.64.171 :: cinder_admin :: <netapp xmlns="http://www.netapp.com/filer/admin" version="1.21" vfiler="vs2rac51"><lun-map><path>/vol/cinder01/volume-388e4dd6-0fc4-4876-a305-3b1b57e9dcb1</path><initiator-group>openstack-3b99cdc9 -888c-4074-a47f-bfa17886f810</initiator-group></lun-map></netapp> :: Pending 
00000009.00c2f30f 11a76240 Mon Nov 02 2015 09:34:31 01:00 [kern_command-history:info:1040] ontapi :: 188.184.64.171 :: cinder_admin :: LUN already mapped to this group :: ONTAPI :: Error

Here, we see the error "LUN already mapped to this group" being returned from the "lun map" API call. We don't see any indication in the logs from the production storage cluster of a permission issue to the API calls being made from OpenStack. The error "LUN already mapped" began to appear in the logs of the "rac51" cluster on September 30th.

There are two different issues here, both of which impact the "lun map" API calls. We need to treat these two issues separately in our troubleshooting.

Getting 13003:Insufficient privileges after more than a year running with no issues

Re: Getting 13003:Insufficient privileges after more than a year running with no issues

Re: Getting 13003:Insufficient privileges after more than a year running with no issues

Re: Getting 13003:Insufficient privileges after more than a year running with no issues

Re: Getting 13003:Insufficient privileges after more than a year running with no issues

Re: Getting 13003:Insufficient privileges after more than a year running with no issues

Re: Getting 13003:Insufficient privileges after more than a year running with no issues

Re: Getting 13003:Insufficient privileges after more than a year running with no issues