Kerberos CIFS SVM pre-authentication failure

[BenBrazil]

Hi All,

I've enabled AES encryption on one of our CIFS SVM's. 

I ran the vserver cifs security modify -vserver vservername -is-aes-encryption-enabled true

This process updates the CIFS server machine account password in Active Directory. The password was updated successfully in Active Directory and I can now see client connections to the CIFS SVM using Kerberos successfully.

I have however seen errors in the NetApp event log showing a kerberos pre-authentication failure for the SVM:

secd.kerberos.preauth: A Kerberos pre-authentication failure occurred for SVM "vserver" due to invalid credentials for VSERVERNAME$@DOMAIN.LOCAL.

These errors only occur maybe 3 or 4 times in a 24 hour period. The errors only appear on one node out of a 4 node cluster. Additionally, there are no volumes owned by the node reporting the error - apart from one Load Sharing replica volume for the SVM root.

User CIFS connections are not impacted, so the only indication of the issue are the event log errors.

The error message states the following as the reason why there is an authentication issue@

This message occurs when invalid credentials are provided for an Active Directory user or the machine account password is out of sync with the credentials set in the Active Directory.

The resolution is to run the password reset command to update the password again.

I have run vserver cifs password-reset -vserver vservername which did update the Active Directory password again, but I still have the same error message.

NetApp support suggested to run an alternative cifs domain password reset -vserver vservername command. This hasn't fixed the issue either.

Any suggestions as to what to do next? The cluster is running at 9.12.1P2.

Thanks

Ben

[Ontapforrum]

Found some related kbs: (If you have already seen these kbs, then it may be worth just raising a ticket with NetApp to investigate this issue)
https://kb.netapp.com/onprem/ontap/da/NAS/secd.kerberos.preauth_event_is_being_logged
https://kb.netapp.com/onprem/ontap/da/NAS/Kerberos_error_secd.kerberos.preauth_due_to_out-of-sync_machine_account_password_due_to_invalid_spn

Are all users logged in via Kerberos (not using ntlm):
::> vserver cifs session show -fields auth-mechanism

[BenBrazil]

Thanks for the reply.

All new user sessions since enabling AES encryption are using Kerberos. Previously they were using NTLMv2. The errors only appear a few times a day and don't correspond to when users connect a new CIFS session. I would expect to see hundreds all of the time if the error log events were as a result of each CIFS session.

I've got an open case with support, but they've just said to run the cifs domain password reset -vserver vservername again because they think I'm using the wrong password

"So kindly run the command again and make sure you are using the right credentials and that will stop the alerts."

That resolution doesn't make sense to me as when running that command, you are only prompted to enter an account with permission in Active Directory to make password changes. There is no prompt to enter the password for the CIFS server object in AD. The password does get changed each time in AD when running the command, so I at least know it is doing what it is meant to be doing in AD.

I've not found any Kerberos or SPN related errors in the secd log. I did check the SPN with Support and they didnt think there was an issue there either.

Thanks,

[Ontapforrum]

Yes, that's a correct observation. : "That resolution doesn't make sense to me as when running that command, you are only prompted to enter an account with permission in Active Directory to make password changes." 

Actually, you don't change the CIFS Server machine-account password , it is refreshed automatically based on the AD policy. You just need to "reset" the password if it has been manually changed on the AD side without NetApp(SVM) knowledge. In this situation, you just have to reset it by running the command as suggested. The Password you provide while running this command, is only needed to authenticate to AD to make sure you have the permission to do so.

Following command resets the password for the "CIFS server" on the supplied vserver:
::> vserver cifs domain password reset -vserver <server>

Basically, once that command is run,  you don't need to do anything else on the password side. Could you tell us if there has been 'fresh' generated events in the last 24 hrs ?

[BenBrazil]

Hi,

There have been some fresh generated events over the past few days, none for the past couple of days however.

I've taken a tcp trace and NetApp support are looking through to try and see what is happening.

[lplataforma-sis]

Any news abouyt this? it is happening also to me, version 9.9.1P15.

'reseating' the password didn't help:

::> vserver cifs domain password reset -vserver <server>

::>  vserver cifs password-reset -vserver <server>

I'm still getting these errors:

secd.kerberos.preauth: Kerberos pre-authentication failure due to out-of-sync machine account password for vserver

[AIRBORNERIC1989]

Any update?  Running into the same thing 9.12.1P5

[HdJ]

We have the same issue on 9.12.1P4, tried everything but no solution yet. Manual password updates work but does not resolve the issues.

[MaGr]

Hi,

we had a similar issue with authentication that passes the domain tunnel.

It's fixed with 9.11.P12. Bug ID 1513324

[bauernfeind]

Hi,

we have a similar issue, in connection with kerberized NFS.
Sometimes, when a user tries to access a kerberized LIF, the error message is triggered.
on the KDC, it is first shown that the SVM tries to authenticate via kerberos, which fails. Afterwards it falls back to NTLM and succeeds.

We are on 9.12.1P10 (but i first encountered the problem end of september which 9.12.1P4)

I already opened a still open case in November, but not much has come out of it.

[darkavich]

I have the same problem that has got worse since we upgrade to 9.12.1P9 from 9.11.1. Support is telling us that it's a domain trust issue but our domain admins claim that's not true. Support is trying to push me down the path of turning off domain discover, but that's not where the problem lies nor makes any sense to me. Hopefully we can get to the bottom of this soon.

[bauernfeind]

Hi, I just want to reply that our error is fixed after enabling ldap session sealing for the ldap client.

ldap client modify -client-config <LDAP-PROFILE> -session-security seal

https://mysupport.netapp.com/site/bugs-online/product/ONTAP/BURT/1522219

[toaster]

Using seal (or sign) for LDAP client session security partially fixed the problem for us.  However, we were still seeing sporadic Kerberos preauth failures (due to invalid SVM credentials) in the event logs at random times, after applying that setting.  We gathered detailed secd traces and NetApp opened the following bug, related to concurrent I/O to a local Kerberos cache file by competing RPCs.  It is suspected that a race condition between those parallel RPCs is causing the preauth errors in our particular situation.  There is no fix yet.

https://mysupport.netapp.com/site/bugs-online/product/ONTAP/JiraNgage/CONTAP-358837