FAS 2020 - Include file not found Issue

GERALDOMAGELLA · ‎2011-04-11

Hi there everyone!

I'm experiencing a wierd problem with Netapp FAS2020 box and I'm looking for anyone that can help me figure it out.

Let me draw you a picture of our environment

We have a vFiler that holds all of our websites. 1TB of Data, disposed in arround 11 million files.

This box is provinding CIFS share for around 50 webservers (most windows 2003, some windows 2008) disposed in webfarms of 3, 4, 10, 13 webservers.

Not too long ago we're experiencing some errors, like this:

Active Server Pages, ASP 0126 (0x80004005)

The include file '/XXX.asp' was not found.

or even in our FTP Server:

[28/03/2011 18:17:45] 550 can't access file.
[28/03/2011 18:17:45] Requested action not taken (e.g., file or directory not found, no access).

But if you go look for the file, you'll find it. If you try access again, you may be able to get it working.

We tried to set cifs.oplocks.opendelta to 35, did'nt change the behavior, now we tried to disable OptLocks... but no luck.

Does anyone have any ideas on this?

Here is our CIFS Status:

webaula@ptvsto01*>cifs status

reject 3 0%

mkdir 0 0%

rmdir 3251 0%

open 0 0%

create 0 0%

close 638533246 22%

X&close 0 0%

flush 43 0%

X&flush 0 0%

delete 0 0%

rename 10646 0%

NTRename 0 0%

getatr 124 0%

setatr 0 0%

read 0 0%

X&read 0 0%

write 13 0%

X&write 0 0%

lock 0 0%

unlock 0 0%

mknew 0 0%

chkpth 124 0%

exit 0 0%

lseek 0 0%

lockread 0 0%

X&lockread 0 0%

writeunlock 0 0%

readbraw 0 0%

writebraw 0 0%

writec 0 0%

gettattre 0 0%

settattre 0 0%

lockingX 4337146 0%

IPC 2344 0%

open2 0 0%

find_first2 6492642 0%

find_next2 306268 0%

query_fs_info 40916829 1%

query_path_info 380715563 13%

set_path_info 0 0%

query_file_info 753553137 26%

set_file_info 7733739 0%

create_dir2 0 0%

Dfs_referral 4766 0%

Dfs_report 0 0%

echo 39926 0%

writeclose 0 0%

openX 0 0%

readX 307148750 10%

writeX 20687737 1%

findclose 0 0%

tcon 0 0%

tdis 347914 0%

negprot 4197 0%

login 326890 0%

logout 321730 0%

tconX 326927 0%

dskattr 0 0%

search 0 0%

fclose 9313 0%

NTCreateX 648195387 22%

NTTransCreate 2014 0%

NTTransIoctl 3804 0%

NTTransNotify 1082945 0%

NTTransSetSec 0 0%

NTTransQuerySec 7978879 0%

NTNamedPipeMulti 0 0%

NTCancel CN 10416 0%

NTCancel Other 0 0%

SMB2Echo 0 0%

SMB2Negprot 16884 0%

SMB2TreeConnnect 26157 0%

SMB2TreeDisconnect 26090 0%

SMB2Login 35222 0%

SMB2Create 23382301 1%

SMB2Read 8941274 0%

SMB2Write 713610 0%

SMB2Lock 0 0%

SMB2Unlock 0 0%

SMB2OplkBrkAck 614905 0%

SMB2ChgNfy 155743 0%

SMB2CLose 21568588 1%

SMB2Flush 0 0%

SMB2Logout 18628 0%

SMB2Cancel 4776 0%

SMB2IPCCreate 53 0%

SMB2IPCRead 58 0%

SMB2IPCWrite 58 0%

SMB2QueryDir 3299433 0%

SMB2QueryFileBasicInfo 8 0%

SMB2QueryFileStndInfo 22 0%

SMB2QueryFileIntInfo 13399599 0%

SMB2QueryFileEAInfo 15425281 1%

SMB2QueryFileFEAInfo 172610 0%

SMB2QueryFileModeInfo 0 0%

SMB2QueryAltNameInfo 0 0%

SMB2QueryFileStreamInfo 2025682 0%

SMB2QueryNetOpenInfo 163489 0%

SMB2QueryAttrTagInfo 0 0%

SMB2QueryAccessInfo 0 0%

SMB2QueryFileUnsupported 0 0%

SMB2QueryFileInvalid 1966 0%

SMB2QueryFSVolInfo 10896731 0%

SMB2QueryFSSizeInfo 1405979 0%

SMB2QueryFSDevInfo 0 0%

SMB2QueryFSAttrInfo 10896731 0%

SMB2QueryFSFullSzInfo 33 0%

SMB2QueryFSObjIdInfo 0 0%

SMB2QueryFSInvalid 0 0%

SMB2QuerySecurityInfo 333395 0%

SMB2SetBasicInfo 2748413 0%

SMB2SetRenameInfo 325 0%

SMB2SetFileLinkInfo 0 0%

SMB2SetFileDispInfo 97923 0%

SMB2SetFullEAInfo 0 0%

SMB2SetModeInfo 0 0%

SMB2SetAllocInfo 148762 0%

SMB2SetEOFInfo 453088 0%

SMB2SetUnsupported 0 0%

SMB2SetInfoInvalid 0 0%

SMB2SetSecurityInfo 0 0%

SMB2FsctlPipeTransceive 48 0%

SMB2FsctlPipePeek 0 0%

SMB2FsctlEnumSnapshots 3 0%

SMB2FsctlDfsReferrals 7639 0%

SMB2FsctlSetSparse 0 0%

SMB2FsctlSecureShare 0 0%

SMB2FsctlFileUnsupported 229 0%

SMB2FsctlIpcUnsupported 0 0%

cancel lock 0

wait lock 0

copy to align 403347

alignedSmall 5846804

alignedLarge 23628054

alignedSmallRel 0

alignedLargeRel 0

FidHashAllocs 47674

TidHashAllocs 14

UidHashAllocs 14

mbufWait 0

nbtWait 0

pBlkWait 0

BackToBackCPWait 0

cwaWait 0

short msg prevent 1084

multipleVCs 324678

SMB signing 0

mapped null user 0

PDCupcalls 0

nosupport 0

read pipe busy 0

write pipe busy 0

trans pipe busy 0

read pipe broken 0

write pipe broken 0

trans pipe broken 0

queued writeraw 0

nbt disconnect 20732

smb disconnect 1052

dup disconnect 4

OpLkBkXorBatchToL2 4952674

OpLkBkXorBatchToNone 0

OpLkBkL2ToNone 1826

OpLkBkNoBreakAck 1040

OpLkBkNoBreakAck95 0

OpLkBkNoBreakAckNT 1040

OpLkBkIgnoredAck 309

OpLkBkWaiterTimedOut 0

OpLkBkDelayedBreak 963519407

SharingErrorRetries 172

FoldAttempts 0

FoldRenames 0

FoldRenameFailures 0

FoldOverflows 0

FoldDuplicates 0

FoldWAFLTooBusy 0

NoAllocCredStat 0

RetryRPCcollision 0

TconCloseTID 0

GetNTAPExtAttrs 2250

SetNTAPExtAttrs 998

SearchBusy 0

ChgNfyNoMemory 0

ChgNfyNewWatch 4775

ChgNfyLastWatch 4646

UsedMIDTblCreated 35

UnusedMIDTblCreated 20

InvalidMIDRejects 0

SMB2InvalidSignature 0

SMB2DurableCreateReceived 23382250

SMB2DurableCreateSucceeded 14732394

SMB2DurableReclaimReceived 3

SMB2DurableReclaimSucceeded 2

SMB2DurableHandlePreserved 86

SMB2DurableHandlePurged 81

SMB2DurableHandleExpired 3

SMB2FileDirInfo 0

SMB2FileFullDirInfo 5456

SMB2FileIdFullDirInfo 0

SMB2FileBothDirInfo 1117277

SMB2FileIdBothDirInfo 830378

SMB2FileNamesInfo 2

SMB2FileDirUnsupported 0

SMB2QueryInfo 54721528

SMB2SetInfo 3448511

SMB2Ioctl 7919

SMB2RelatedCompRequest 14498565

SMB2UnRelatedCompRequest 0

SMB2FileRequest 116864901

SMB2PipeRequest 15425

SMB2nosupport 71

Max Multiplex = 118, Max pBlk Exhaust = 0, Max pBlk Reserve Exhaust = 0

Max FIDs = 2483, Max FIDs on one tree = 1029

Max Searches on one tree = 7, Max Core Searches on one tree = 0

Max sessions = 97

Max trees = 437

Max shares = 7

Max session UIDs = 91, Max session TIDs = 91

Max locks = 2637

Max credentials = 437

Max group SIDs per credential = 17

Max pBlks = 896 Current pBlks = 858 Num Logons = 0

Max reserved pBlks = 32 Current reserved pBlks = 32

Max gAuthQueue depth = 17

Max gSMBBlockingQueue depth = 11

Max gSMBTimerQueue depth = 7

Max gSMBAlfQueue depth = 4

Max gSMBRPCWorkerQueue depth = 3

Max gOffloadQueue depth = 4

Local groups: builtins = 6, user-defined = 1, SIDs = 6

RPC group count = 10, RPC group active count = 0

Max Watched Directories = 429, Current Watched Directories = 339

Max Pending ChangeNotify Requests = 426, Current Pending ChangeNotify Requests = 339

Max Pending DeleteOnClose Requests = 2688, Current Pending DeleteOnClose Requests = 0

GERALDOMAGELLA · ‎2011-05-25

Nobody??

AIDENHENDERSON · ‎2011-07-22

max multiplex is greater than 32. It's showing as 118. Max Multiplex = 118

Have a read of this https://kb.netapp.com/support/index?page=content&id=3011249

I would recommend increasing from the default of 50. be aware that you will need to restart cifs for the change to come into affect.

This change needs to be made at the vfiler level.

Regards

GERALDOMAGELLA · ‎2011-07-25

Hello, Thanks for your response.. I don't seem to have access to reas this KB. What's wrong? When I hit your link I get "Access Denied" page. Same for Wcotton's link.

AIDENHENDERSON · ‎2011-07-27

from the link

CIFS max not going above 50.

In order to operate more efficiently, some applications such as Citrix and IIS combine many CIFS sessions into the same TCP connection. These connections are known as multiplexed connections. The 'cifs.max_mpx option' determines how many CIFS sessions can be multiplexed into a single TCP connection. When that value is reached, creation of new CIFS sessions within that TCP connection will fail and those users may not be able to access files via CIFS.

The value associated with the 'cifs.max_mpx option' should never be set below the Windows NT default of 50. If CIFS statistical output shows the Max Multiplex to be greater than 32, then this number should be raised. The only legal values are 50, 126, 253, 255* and 1124. Do not set this to any other values without an explicit conversation with NetApp customer service. It should be noted that automatically setting this option to a high value is not recommended. High values consume a lot of resources on the clients. Setting this to an unapproved value will cause errors on some clients.

AIDENHENDERSON · ‎2011-07-27

Sorry forgot to add. Your CIFS maxmultiplex out was 118. So you need to increase the cifs max_mpx value as per the instructions.

wcotton · ‎2011-07-23

Try running the NetApp system analyzer util for CIFS http://now.netapp.com/NOW/download/tools/ntapsa_cifs/.

What version of Ontap are you running? as you may have hit a bug, also is there any error messages in syslog relating to oplocks?

OLIVER_CLAPTON · ‎2013-02-01

Hi, I am troubleshooting cifs problems too.

I have an output from the command: cifs stat with an Max Multiplex value of 62.

ccifs.max_mpx ist 255 (default)

So must I set the cifs.max_mpx value higher now because my Max Multiplex value is greater 32. What does the 32 mean? Why 32.

Hope someone can give advice.

Thank you very much!
Regards