DFM : A CIFS domain controller connection to the filer has failed.

[netapp_ramesh]

Hi Guys,

We are getting the below alerts or warnings frequently in the netapp filers console .When we login and check the domain controllers everything is fine and there is no issue reported in client accessing the cifs shares.

Please let me know if you have encountered similar issue. Any help on this is highly appreciated.

[xxxxxxxxx@xxxxxxx:cifs.pipe.errorMsg:error]: CIFS: Error on named pipe with xxxxxxx: Error connecting to server, open pipe failed 

[xxxxxxxx@xxxxxxx:smbrpc.pipeCreate.fail:error]: CIFSRPC: Attempt to create pipe LSA for LsarLookupSids failed with error 0xc000005e. 

[xxxxxxxx@xxxxxxx:smbrpc.exceptionCaught:error]: CIFSRPC: An RPC exception with a server of type domain controller occurred. 

[xxxxxxxx@xxxxxxx:smbrpc.pipeClose.fail:error]: CIFSRPC: Attempt to close pipe LSA failed with error 0xc0000022.

Regards,

Ramesh

[netapp_ramesh]

Any help on this is highly appreciated.

[mbeattie]

Hi Ramesh,

I'd advise reading this KB article:

https://kb.netapp.com/support/s/article/ka11A0000001RTdQAM/controller-does-not-contact-other-dcs-when-there-are-issues-with-a-connected-dc

What is the output of "cifs domaininfo"? Do you have multiple Domain controllers in the AD Site\subnet that your controller\vFiler is located in?

The KB article is saying that if you only have a single DC in the AD Site that the vfiler is associated with (via AD sites and services) and there is an issue with the DC, the vfiler won't automatically attemp to find another one. EG:

C:\>dsquery server -site testlab
"CN=TESTDC01,CN=Servers,CN=testlab,CN=Sites,CN=Configuration,DC=testlab,DC=local"

Where -site <%site_name%> is the name of the AD site you are troubleshooting. Short answer, to guarentee redundancy there should be atleast two domain controllers

/Matt

[netapp_ramesh]

Hi Matt,

Thanks for your quick response. Find below cifs domaininfo output. We are not seeing any issue in client access but we get lot of incident tickets because of this.We want these errors not to occur. Please let me know if there is any solution to supress the errors.

==============================

xx00xx1c@xx00001> cifs domaininfo
NetBIOS Domain:                         XXX
Windows Domain Name:                    XXX.xxx.local
Domain Controller Functionality:        Windows 2012 R2
Domain Functionality:                   Windows 2003
Forest Functionality:                   Windows 2003
Filer AD Site:                          XXX

Current Connected DCs:                  \\XX00111
Total DC addresses found:               3
Preferred Addresses:
                                        None
Favored Addresses:
                                        10.x.x.2     XX00111          PDC
                                        10.x.x.1                      PDC
Other Addresses:
                                        10.x.x.17                    PDC

Connected AD LDAP Server:               \\XX00111.xxx.xxx.local
Preferred Addresses:
                                        None
Favored Addresses:
                                        10.x.x.2    
                                         xx00111.xxx.xxx.local
                                        10.x.x.1    
                                         xx00110.xxx.xxx.local
Other Addresses:
                                        10.x.x.17  
                                         xxc00100.xxx.xxx.local

==================================================

Regards,

Ramesh

[netapp_ramesh]

We get the below alert in the DFM for this issue.

DFM Alert:

A CIFS domain controller connection to the filer has failed.Product trap Data- CIFS: Domain controller server XX00111 (10.x.x.2) connection lost: DC has disconnected from the filer  Serial num -xx00000xxxxx.

Regards,

Ramesh

[mbeattie]

Hi Ramesh,

What is the exact version of data ONTAP running on your controller? What's the output the "version" command? As you have multiple DC's in your AD site available to the controller the previous KB article isn't the issue. It sounds like a BUG to me (possibly #390540) but i'd need to the version of ONTAP you are running to see if that is applies.

/Matt

[mbeattie]

Hi Ramesh,

Also are you running a mixed mode environment? IE windows and unix clients accessing the same CIFS shares? If so this might be relevent (the error messages in KB article match the error messages you initially posted)

https://kb.netapp.com/support/s/article/ka11A000000137OQAQ/cifs-rpc-attempt-to-close-pipe-lsa-failed-with-error-0xc0000022

/Matt

[netapp_ramesh]

Hi Matt,

We have this issue in multiple versions of Ontap and in more than 20+ filers we get these alerts .

Ex:  7.3.2  ,  8.2.4P4 

Regards,

Ramesh

[netapp_ramesh]

We are not using mixed mode environment. We are having only windows client accessing the cifs shares.

/Ramesh

[mbeattie]

Hi Ramesh,

The bug #390540 could apply to 7-Mode systems prior to 7.3.6P2 but as you are seeing the issue on other versions ONTAP which include the bug fix i don't think that's the issue (and if it were it would likely be causing your clients disruptions). I'd suggest looking at KB article (in particular point 4 given you are not using mixed mode):

https://kb.netapp.com/support/s/article/ka11A000000137OQAQ/cifs-rpc-attempt-to-close-pipe-lsa-failed-with-error-0xc0000022

Whats the output of "options wafl.default_nt_user"

Do you have any AD trusts in your environment that may no longer be valid? What's the output of:

C:\>netdom query trust

You can then use:

NETDOM TRUST trusting_domain_name /Domain:trusted_domain_name [/UserD:user]
           [/PasswordD:[password | *]] [/UserO:user] [/PasswordO:[password | *]]
           [/Verify] [/RESEt] [/PasswordT:new_realm_trust_password]
           [/Add] [/REMove] [/Twoway] [/REAlm] [/Kerberos]
           [/Transitive[:{yes | no}]]
           [/OneSide:{trusted | trusting}] [/Force] [/Quarantine[:{yes | no}]]
           [/NameSuffixes:trust_name [/ToggleSuffix:#]]
           [/EnableSIDHistory[:{yes | no}]]
           [/ForestTRANsitive[:{yes | no}]]
           [/CrossORGanization[:{yes | no}]]
           [/AddTLN:TopLevelName]
           [/AddTLNEX:TopLevelNameExclusion]
           [/RemoveTLN:TopLevelName]
           [/RemoveTLNEX:TopLevelNameExclusion]
           [/SecurePasswordPrompt]

If that still doesn't resolve the issue, have you logged a case? Guessing you might not be able to given some of the older versions of Data ONTAP you are running.

/Matt

[netapp_ramesh]

Hi Matt,

Please find below output of command options wafl.default_nt_user. There is nothing specified for this option

xx0001> options wafl.default_nt_user

options wafl.default_nt_user

Please find below output  of netdom command.

C:\WINDOWS\system32>netdom query trust

Direction Trusted\Trusting domain                         Trust type

========= =======================                         ==========

<->       x-dom.u-xxx.net

Direct

The command completed successfully.

-----------------------------------------

We dont have support for these filers as most of them are out of support.

Regards,

Ramesh

[mbeattie]

Hi Ramesh,

You might want to try:

• Explicity setting the default NT user to null (as the KB states it can appear as null but it may have been set to contain a space).
• Check the qtree status to ensure they are all set to NTFS (if there are any qtrees that are set to UNIX then you may have mixed mode issues)
• Check the entries in your usermap.cfg file
• Check the domain trust is still valid.

So for each controller...EG:

TESTNS01> options wafl.default_nt_user ""

TESTNS01> options wafl.default_nt_user
wafl.default_nt_user

TESTNS01> qtree status
Volume   Tree     Style Oplocks  Status
-------- -------- ----- -------- ---------
vol0              ntfs  enabled  normal
testnv01          ntfs  enabled  normal
testvol2          ntfs  enabled  normal
testvol2 qtree1   ntfs  enabled  normal
testvol2 qtree2   ntfs  enabled  normal
testvol2 qtree3   ntfs  enabled  normal
testnv02          ntfs  enabled  normal
testvol1          ntfs  enabled  normal
testvol1 qtree1   ntfs  enabled  normal
testvol1 qtree2   ntfs  enabled  normal
testvol1 qtree3   ntfs  enabled  normal

TESTNS01> rdfile /etc/usermap.cfg

C:>NETDOM TRUST <%trusting_domain_name%> /Domain:<%trusted_domain_name%> /verify

I'd be more concerned about having 20+ controllers that are out of support and maintence.

Hopefully you have a plan to upgrade and migrate these to supported systems?

/Matt

[ramesh_netapp]

Hi matt,

I have set the  below option to null

options wafl.default_nt_user ""

Please find below qtree status.

xxxxxxx> qtree status
Volume Tree Style Oplocks Status
-------- -------- ----- -------- ---------
vol01 ntfs enabled normal
vol01 f_user ntfs enabled normal
vol0 ntfs enabled normal
vol02 ntfs enabled normal
vol02 fteam ntfs enabled normal

There is no entry in usermap.cfg file.

============================

 We are seeing these alerts daily twice in the filers /etc/messages files.

Sun Sep 3 07:08:16 CEST [xxxxxx:cifs.pipe.errorMsg:error]: CIFS: Error on named pipe with DC: Error connecting to server, open pipe failed
Sun Sep 3 07:08:16 CEST [xxxxxx:smbrpc.pipeCreate.fail:error]: CIFSRPC: Attempt to create pipe LSA for LsarLookupSids failed with error 0xc000005e.

Sun Sep 3 17:13:39 CEST [xxxxxx:cifs.pipe.errorMsg:error]: CIFS: Error on named pipe with DC: Error connecting to server, open pipe failed
Sun Sep 3 17:13:39 CEST [xxxxxx:smbrpc.pipeCreate.fail:error]: CIFSRPC: Attempt to create pipe LSA for LsarLookupSids failed with error 0xc000005e.

Mon Sep 4 04:00:03 CEST [xxxxxx:cifs.pipe.errorMsg:error]: CIFS: Error on named pipe with DC: Error connecting to server, open pipe failed
Mon Sep 4 04:00:03 CEST [xxxxxx:smbrpc.pipeCreate.fail:error]: CIFSRPC: Attempt to create pipe LSA for LsarLookupSids failed with error 0xc000005e.

Mon Sep 4 14:09:45 CEST [xxxxxx:cifs.pipe.errorMsg:error]: CIFS: Error on named pipe with DC: Error connecting to server, open pipe failed
Mon Sep 4 14:09:45 CEST [xxxxxx:smbrpc.pipeCreate.fail:error]: CIFSRPC: Attempt to create pipe LSA for LsarLookupSids failed with error 0xc000005e.

We get the below alert from DFM 2 to 3 times a month for each filer and we get incident ticket for this. 

A CIFS domain controller connection to the filer has failed.Product trap Data- CIFS: Domain controller server DC connection lost:
DC has disconnected from the filer Serial num -6x00000xxxxxx

=====================================================================

Regards,

Ramesh

[netapp_ramesh]

Hi Matt,

We also get the below alerts on some filers .

auth.dc.trace.DCConnection.errorMsg:error]: AUTH: Domain Controller error: NetLogon error 0xc0000022: - Filer's security information differs from domain controller \\DC

Regards,

Ramesh

[mbeattie]

Hi,

See this KB. Have you tried setting "cifs prefdc"? You could also try a "cifs resetdc" to reset your DC connections after setting a preferred domain controller

https://kb.netapp.com/support/s/article/ka11A00000015x2QAA/storage-system-s-security-information-differs-from-domain-controller

/Matt

[netapp_ramesh]

Hi matt,

Thanks for the response.

The other DC which is connected to the filer as per below output also throws similar output in other filers. So setting Prefer domain controller may not resolve this issue.

As per the KB article we have checked solutions in step2 and 3 which is set as per the KB. Coming to Solution1 . We have never re run cifs setup in the filers. What all the things we need to change while re running the cifs setup?. Can you please advice on this.

  1. The Auth message could be due to the DC machine account differing from that of the storage system, in which case  cifs setup should be run again on the storage system.
       Note: Running cifs setup will be disruptive

==========================================

xx00xx1c@xx00001> cifs domaininfo
NetBIOS Domain:                         XXX
Windows Domain Name:                    XXX.xxx.local
Domain Controller Functionality:        Windows 2012 R2
Domain Functionality:                   Windows 2003
Forest Functionality:                   Windows 2003
Filer AD Site:                          XXX

Current Connected DCs:                  \\XX00111
Total DC addresses found:               3
Preferred Addresses:
                                        None
Favored Addresses:
                                        10.x.x.2     XX00111          PDC
                                        10.x.x.1                      PDC
Other Addresses:
                                        10.x.x.17                    PDC

Connected AD LDAP Server:               \\XX00111.xxx.xxx.local
Preferred Addresses:
                                        None
Favored Addresses:
                                        10.x.x.2    
                                         xx00111.xxx.xxx.local
                                        10.x.x.1    
                                         xx00110.xxx.xxx.local
Other Addresses:
                                        10.x.x.17  
                                         xxc00100.xxx.xxx.local

======================

Regards,

Ramesh