ONTAP Hardware

FAS 2020 - Include file not found Issue

GERALDOMAGELLA
5,724 Views

Hi there everyone!

I'm experiencing a wierd problem with Netapp FAS2020 box and I'm looking for anyone that can help me figure it out.

Let me draw you a picture of our environment

We have a vFiler that holds all of our websites. 1TB of Data, disposed in arround 11 million files.

This box is provinding CIFS share for around 50 webservers (most windows 2003, some windows 2008) disposed in webfarms of 3, 4, 10, 13 webservers.

Not too long ago we're experiencing some errors, like this:

Active Server Pages, ASP 0126 (0x80004005)

The include file '/XXX.asp' was not found.
or even in our FTP Server:
[28/03/2011 18:17:45] 550 can't access file.
[28/03/2011 18:17:45] Requested action not taken (e.g., file or directory not found, no access).

But if you go look for the file, you'll find it. If you try access again, you may be able to get it working.

We tried to set cifs.oplocks.opendelta to 35, did'nt change the behavior, now we tried to disable OptLocks... but no luck.

Does anyone have any ideas on this?

Here is our CIFS Status:

webaula@ptvsto01*>cifs status


                        reject        3  0%

                         mkdir        0  0%

                         rmdir     3251  0%

                          open        0  0%

                        create        0  0%

                         close 638533246 22%

                       X&close        0  0%

                         flush       43  0%

                       X&flush        0  0%

                        delete        0  0%

                        rename    10646  0%

                      NTRename        0  0%

                        getatr      124  0%

                        setatr        0  0%

                          read        0  0%

                        X&read        0  0%

                         write       13  0%

                       X&write        0  0%

                          lock        0  0%

                        unlock        0  0%

                         mknew        0  0%

                        chkpth      124  0%

                          exit        0  0%

                         lseek        0  0%

                      lockread        0  0%

                    X&lockread        0  0%

                   writeunlock        0  0%

                      readbraw        0  0%

                     writebraw        0  0%

                        writec        0  0%

                     gettattre        0  0%

                     settattre        0  0%

                      lockingX  4337146  0%

                           IPC     2344  0%

                         open2        0  0%

                   find_first2  6492642  0%

                    find_next2   306268  0%

                 query_fs_info 40916829  1%

               query_path_info 380715563 13%

                 set_path_info        0  0%

               query_file_info 753553137 26%

                 set_file_info  7733739  0%

                   create_dir2        0  0%

                  Dfs_referral     4766  0%

                    Dfs_report        0  0%

                          echo    39926  0%

                    writeclose        0  0%

                         openX        0  0%

                         readX 307148750 10%

                        writeX 20687737  1%

                     findclose        0  0%

                          tcon        0  0%

                          tdis   347914  0%

                       negprot     4197  0%

                         login   326890  0%

                        logout   321730  0%

                         tconX   326927  0%

                       dskattr        0  0%

                        search        0  0%

                        fclose     9313  0%

                     NTCreateX 648195387 22%

                 NTTransCreate     2014  0%

                  NTTransIoctl     3804  0%

                 NTTransNotify  1082945  0%

                 NTTransSetSec        0  0%

               NTTransQuerySec  7978879  0%

              NTNamedPipeMulti        0  0%

                   NTCancel CN    10416  0%

                NTCancel Other        0  0%

                      SMB2Echo        0  0%

                   SMB2Negprot    16884  0%

              SMB2TreeConnnect    26157  0%

            SMB2TreeDisconnect    26090  0%

                     SMB2Login    35222  0%

                    SMB2Create 23382301  1%

                      SMB2Read  8941274  0%

                     SMB2Write   713610  0%

                      SMB2Lock        0  0%

                    SMB2Unlock        0  0%

                SMB2OplkBrkAck   614905  0%

                    SMB2ChgNfy   155743  0%

                     SMB2CLose 21568588  1%

                     SMB2Flush        0  0%

                    SMB2Logout    18628  0%

                    SMB2Cancel     4776  0%

                 SMB2IPCCreate       53  0%

                   SMB2IPCRead       58  0%

                  SMB2IPCWrite       58  0%

                  SMB2QueryDir  3299433  0%

        SMB2QueryFileBasicInfo        8  0%

         SMB2QueryFileStndInfo       22  0%

          SMB2QueryFileIntInfo 13399599  0%

           SMB2QueryFileEAInfo 15425281  1%

          SMB2QueryFileFEAInfo   172610  0%

         SMB2QueryFileModeInfo        0  0%

          SMB2QueryAltNameInfo        0  0%

       SMB2QueryFileStreamInfo  2025682  0%

          SMB2QueryNetOpenInfo   163489  0%

          SMB2QueryAttrTagInfo        0  0%

           SMB2QueryAccessInfo        0  0%

      SMB2QueryFileUnsupported        0  0%

          SMB2QueryFileInvalid     1966  0%

            SMB2QueryFSVolInfo 10896731  0%

           SMB2QueryFSSizeInfo  1405979  0%

            SMB2QueryFSDevInfo        0  0%

           SMB2QueryFSAttrInfo 10896731  0%

         SMB2QueryFSFullSzInfo       33  0%

          SMB2QueryFSObjIdInfo        0  0%

            SMB2QueryFSInvalid        0  0%

         SMB2QuerySecurityInfo   333395  0%

              SMB2SetBasicInfo  2748413  0%

             SMB2SetRenameInfo      325  0%

           SMB2SetFileLinkInfo        0  0%

           SMB2SetFileDispInfo    97923  0%

             SMB2SetFullEAInfo        0  0%

               SMB2SetModeInfo        0  0%

              SMB2SetAllocInfo   148762  0%

                SMB2SetEOFInfo   453088  0%

            SMB2SetUnsupported        0  0%

            SMB2SetInfoInvalid        0  0%

           SMB2SetSecurityInfo        0  0%

       SMB2FsctlPipeTransceive       48  0%

             SMB2FsctlPipePeek        0  0%

        SMB2FsctlEnumSnapshots        3  0%

         SMB2FsctlDfsReferrals     7639  0%

            SMB2FsctlSetSparse        0  0%

          SMB2FsctlSecureShare        0  0%

      SMB2FsctlFileUnsupported      229  0%

       SMB2FsctlIpcUnsupported        0  0%

                   cancel lock        0

                     wait lock        0

                 copy to align   403347

                  alignedSmall  5846804

                  alignedLarge 23628054

               alignedSmallRel        0

               alignedLargeRel        0

                 FidHashAllocs    47674

                 TidHashAllocs       14

                 UidHashAllocs       14

                      mbufWait        0

                       nbtWait        0

                      pBlkWait        0

              BackToBackCPWait        0

                       cwaWait        0

             short msg prevent     1084

                   multipleVCs   324678

                   SMB signing        0

              mapped null user        0

                    PDCupcalls        0

                     nosupport        0

                read pipe busy        0

               write pipe busy        0

               trans pipe busy        0

              read pipe broken        0

             write pipe broken        0

             trans pipe broken        0

               queued writeraw        0

                nbt disconnect    20732

                smb disconnect     1052

                dup disconnect        4

            OpLkBkXorBatchToL2  4952674

          OpLkBkXorBatchToNone        0

                OpLkBkL2ToNone     1826

              OpLkBkNoBreakAck     1040

            OpLkBkNoBreakAck95        0

            OpLkBkNoBreakAckNT     1040

              OpLkBkIgnoredAck      309

          OpLkBkWaiterTimedOut        0

            OpLkBkDelayedBreak 963519407

           SharingErrorRetries      172

                  FoldAttempts        0

                   FoldRenames        0

            FoldRenameFailures        0

                 FoldOverflows        0

                FoldDuplicates        0

               FoldWAFLTooBusy        0

               NoAllocCredStat        0

             RetryRPCcollision        0

                  TconCloseTID        0

               GetNTAPExtAttrs     2250

               SetNTAPExtAttrs      998

                    SearchBusy        0

                ChgNfyNoMemory        0

                ChgNfyNewWatch     4775

               ChgNfyLastWatch     4646

             UsedMIDTblCreated       35

           UnusedMIDTblCreated       20

             InvalidMIDRejects        0

          SMB2InvalidSignature        0

     SMB2DurableCreateReceived 23382250

    SMB2DurableCreateSucceeded 14732394

    SMB2DurableReclaimReceived        3

   SMB2DurableReclaimSucceeded        2

    SMB2DurableHandlePreserved       86

       SMB2DurableHandlePurged       81

      SMB2DurableHandleExpired        3

               SMB2FileDirInfo        0

           SMB2FileFullDirInfo     5456

         SMB2FileIdFullDirInfo        0

           SMB2FileBothDirInfo  1117277

         SMB2FileIdBothDirInfo   830378

             SMB2FileNamesInfo        2

        SMB2FileDirUnsupported        0

                 SMB2QueryInfo 54721528

                   SMB2SetInfo  3448511

                     SMB2Ioctl     7919

        SMB2RelatedCompRequest 14498565

      SMB2UnRelatedCompRequest        0

               SMB2FileRequest 116864901

               SMB2PipeRequest    15425

                 SMB2nosupport       71

Max Multiplex = 118, Max pBlk Exhaust = 0, Max pBlk Reserve Exhaust = 0

Max FIDs = 2483, Max FIDs on one tree = 1029

Max Searches on one tree = 7, Max Core Searches on one tree = 0

Max sessions = 97

Max trees = 437

Max shares = 7

Max session UIDs = 91, Max session TIDs = 91

Max locks = 2637

Max credentials = 437

Max group SIDs per credential = 17

Max pBlks = 896 Current pBlks = 858 Num Logons = 0

Max reserved pBlks = 32 Current reserved pBlks = 32

Max gAuthQueue depth         = 17

Max gSMBBlockingQueue depth  = 11

Max gSMBTimerQueue depth     = 7

Max gSMBAlfQueue depth       = 4

Max gSMBRPCWorkerQueue depth = 3

Max gOffloadQueue depth      = 4

Local groups: builtins = 6, user-defined = 1, SIDs = 6

RPC group count = 10, RPC group active count = 0

Max Watched Directories = 429, Current Watched Directories = 339

Max Pending ChangeNotify Requests = 426, Current Pending ChangeNotify Requests = 339

Max Pending DeleteOnClose Requests = 2688, Current Pending DeleteOnClose Requests = 0

7 REPLIES 7

GERALDOMAGELLA
5,724 Views

Nobody??

AIDENHENDERSON
5,724 Views

max multiplex is greater than 32. It's showing as 118. Max Multiplex = 118

Have a read of this https://kb.netapp.com/support/index?page=content&id=3011249

I would recommend increasing from the default of 50. be aware that you will need to restart cifs for the change to come into affect.

This change needs to be made at the vfiler level.

Regards

GERALDOMAGELLA
5,724 Views

Hello, Thanks for your response.. I don't seem to have access to reas this KB. What's wrong? When I hit your link I get "Access Denied" page. Same for Wcotton's link.

AIDENHENDERSON
5,724 Views

from the link

CIFS max not going above 50.

In order to operate more efficiently, some applications such as Citrix and IIS combine many CIFS sessions into the same TCP connection.  These connections are known as multiplexed connections.  The 'cifs.max_mpx option' determines how many CIFS sessions can be multiplexed into a single TCP connection.  When that value is reached, creation of new CIFS sessions within that TCP connection will fail and those users may not be able to access files via CIFS.

The value associated with the 'cifs.max_mpx option' should never be set below the Windows NT default of 50. If CIFS statistical output shows the Max Multiplex to be greater than 32, then this number should be raised. The only legal values are 50, 126, 253, 255* and 1124. Do not set this to any other values without an explicit conversation with NetApp customer service. It should be noted that automatically setting this option to a high value is not recommended. High values consume a lot of resources on the clients. Setting this to an unapproved value will cause errors on some clients.

AIDENHENDERSON
5,724 Views

Sorry forgot to add. Your CIFS maxmultiplex out was 118. So you need to increase the cifs max_mpx value as per the instructions.

wcotton
5,724 Views

Try running the NetApp system analyzer util for CIFS http://now.netapp.com/NOW/download/tools/ntapsa_cifs/

What version of Ontap are you running? as you may have hit a bug, also is there any error messages in syslog relating to oplocks?

OLIVER_CLAPTON
5,724 Views

Hi, I am troubleshooting cifs problems too.

I have an output from the command: cifs stat with an Max Multiplex value of 62.

ccifs.max_mpx ist 255 (default)

So must I set the cifs.max_mpx value higher now because my Max Multiplex value is greater 32. What does the 32 mean? Why 32.

Hope someone can give advice.

Thank you very much!
Regards

Public