NFS permission denied UNIX mode >16 groups

[kt99]

Hello everyone. I've got a permission issue with NFS (ONTap 9.7). Let me try to explain a little of the set up and what we're trying to do.

We have linux systems joined to an AD domain using SSSD, and because of that, each user is a member of a bunch of groups (could vary, but definitely more than 16). The AD server is not under my control, and has no Unix Attributes associated with any accounts. We've worked around the issue in a couple of ways in the past (using a system that also had a 16 group limitation for NFS).  

1. Had the users perform a newgrp command before accessing the storage, setting their primary group to be the one with the correct permissions
2. Modified the SSSD configuration to only return groups that we care about for accessing the storage (this works ok and actually speeds up group enumeration as it only returns a few groups)

With the Netapp we'd like to move forward past these methods of workarounds and use the extended group settings, so to assist with that, I set up an LDAP server whose sole function it is is to grab the SSSD user/group info and store it so that the Netapp can do user lookups with the correct UIDs/GIDs for the domain joined systems. I believe I have this working as doing:

vserver services name-service getxxbyyy getpwbyname/getgrbyname -vserver svm-mystuff -node node1 -username(groupname) myuser(or group) 

returns the correct info. The groups list multiple users, the users resolve UIDs, so I thought we'd be good.

I set the NFS server settings to use extended groups, and set it to 256, LDAP to use RFC2307, name services is set to files,ldap for passwd and group.

But when I mount the NFS volume on a linux client, I get a permission denied trying to access a group-owned directory unless I do a newgrp first.

I tried to run a trace, and did see this, and am wondering if this indicates anything:

[000.001.053] info : Determined UNIX id 1444 is UNIX user 'myuser' { in secd_rpc_auth_user_id_to_unix_ext_creds_1_svc() at src/authorization/secd_rpc_authorization.cpp:971 }
00000017.0009bde5 05ebdb82 Fri Sep 24 2021 07:14:06 -04:00 [kern_secd:info:9547] | [000.001.061] debug: unixCredFlags=1, domainId=0, uid=1444, gid=1440, additionalGids=1 { in secd_rpc_auth_user_id_to_unix_ext_creds_1_svc() at src/authorization/secd_rpc_authorization.cpp:1024 }

I see that the user is resolved, and that it found the primary gid ok, but I noticed it said 'additionalGids=1'. That concerns me, is it still not looking up the secondary groups? 

Thanks for any insight anyone might have. 

[parisi]

In AD LDAP, are the UNIX attributes extended? If so, use AD LDAP with ONTAP. 

But if SSSD in Linux is generating UIDs/GIDs based on AD SIDs, then the issue is probably a mismatch between what the client thinks the groups are and what ONTAP thinks the groups are.

Does the group show the user is a member in the ONTAP getxxbyyy output? With extended GIDs, we don't populate the entire group list - instead, we fetch the requested group. TR-4067 covers it on page 127.

Bur honestly, if you're getting a failure until you do newgrp on the client, the issue is likely on the client. If the client is pulling the groups from AD LDAP using SSSD, then the GIDs/UIDs are likely different. They'd need to match what you have in your LDAP. Did you configure the client to use your LDAP server as well? You can add a 2nd LDAP config in SSSD as per TR-4835 starting on page 77:

https://www.netapp.com/us/media/tr-4835.pdf 

[kt99]

Our AD LDAP does not have Unix attributes in it. The ONTAP getxxbyyy output does show that the user is a member of groups that resolve to the same GIDs as what SSSD provides (that's why we set up the intermediary LDAP server). The GID/UIDs should be the same between the client and what Netapp is seeing, but it's an interesting idea trying do send the client to the same LDAP server, and I can try that just to see if it makes a difference at all. I would hope it wouldn't, but you never know.

[parisi]

So if ONTAP can find the group memberships, then it's likely not a problem with ONTAP.

If the client is joined to the domain using SSSD, it's likely generating its own UID/GID (see the section on SSSD algorithms in TR-4835 on page 80) - that wouldn't match with what ONTAP is pulling from LDAP and why using "newgrp" fixes the issue. Newgrp adds a local entry to /etc/groups and the client can then resolve the necessary group name/ID to allow access.

[kt99]

I understand, I think we're not connecting on one of the issues. I created my own LDAP server that mimics the same UID/GID information as what SSSD is providing. 

[kt99]

What attributes specifically is ONTAP looking for in an LDAP server? Maybe we're missing some? For groups I just have them defined as a posixGroup with memberUid fields.

For a user I have a posixAccount with uid/cn/givenName/sn/homeDirectory is there more that's expected and might be causing ONTAP some grief?

[parisi]

ONTAP looks for the attributes defined in the specified LDAP client schema (ldap client schema show)

You define that when you create the LDAP client configuration. 

If you need to customize it, you can use "ldap client schema copy" to copy a readonly schema template to an editable schema.

[kt99 · Accepted solution]

I think I have everything working now. There must have been some caching going on somewhere that managed to filter out over the weekend. Last week, things actually were generally working, except for me and one other user (who were also the ones tinkering around in the netapp trying to get things working, so we may have cached ourselves in the numerous test we were running). 

To summarize what we're doing to get this working in the way we want:

- Have a system joined to the AD domain using SSSD which can resolve the UID and GIDs that we want to use (SSSD derived)

- That system has an OpenLDAP server on it and a couple of scripts that run regularly to keep the LDAP server updated with the users/groups (so they'll be added/removed as managed through AD).

- Synchronize this LDAP server with a secondary one for redundancy

- Point the Netapp to these LDAP servers

[parisi]

Glad you got it working.

For future reference, TR-4835 has a number of cache viewing/clearing commands to avoid caching from biting you again.