max multiplex

[philipp_pluess]

Hello Everybody

my customers ar moaning about slow cifs performance. 

now i checked every value of the cifs options, the connections to the dc and connections inbetween the filer and the clients. everything seems to be fine.

what i saw is, max multiplex, which could be a problem.

cifs stat

reject 63 0%
mkdir 5390 0%
rmdir 18829 0%
open 0 0%
create 0 0%
close 12298271 0%
X&close 0 0%
flush 450000 0%
X&flush 0 0%
delete 90008 0%
rename 51730 0%
NTRename 1258 0%
getatr 31353 0%
setatr 18435 0%
read 0 0%
X&read 0 0%
write 451114 0%
X&write 0 0%
lock 0 0%
unlock 0 0%
mknew 0 0%
chkpth 22756 0%
exit 0 0%
lseek 0 0%
lockread 0 0%
X&lockread 0 0%
writeunlock 0 0%
readbraw 0 0%
writebraw 0 0%
writec 0 0%
gettattre 0 0%
settattre 18136 0%
lockingX 494697 0%
IPC 18288 0%
open2 0 0%
find_first2 9802437 0%
find_next2 48747 0%
query_fs_info 4815490 0%
query_path_info 56025025 0%
set_path_info 11868 0%
query_file_info 57949889 0%
set_file_info 1044818 0%
create_dir2 0 0%
Dfs_referral 11347 0%
Dfs_report 0 0%
echo 366436 0%
writeclose 0 0%
openX 28915 0%
readX 131143714 1%
writeX 484363027 4%
findclose 0 0%
tcon 0 0%
tdis 6915968 0%
negprot 1328748 0%
login 43088 0%
logout 17273 0%
tconX 41518 0%
dskattr 0 0%
search 0 0%
fclose 14744 0%
NTCreateX 14481311 0%
NTTransCreate 4 0%
NTTransIoctl 30143 0%
NTTransNotify 92099 0%
NTTransSetSec 189 0%
NTTransQuerySec 196995 0%
NTNamedPipeMulti 0 0%
NTCancel CN 1914 0%
NTCancel Other 35 0%
SMB2Echo 341 0%
SMB2Negprot 4060210 0%
SMB2TreeConnnect 49813799 0%
SMB2TreeDisconnect 6884001 0%
SMB2Login 5312291 0%
SMB2Create 3308532482 28%
SMB2Read 1117406262 9%
SMB2Write 303741281 3%
SMB2Lock 14682683 0%
SMB2Unlock 13055148 0%
SMB2OplkBrkAck 3625 0%
SMB2ChgNfy 9654246 0%
SMB2CLose 3053125213 25%
SMB2Flush 8097014 0%
SMB2Logout 3143411 0%
SMB2Cancel 3958296 0%
SMB2IPCCreate 4863809 0%
SMB2IPCRead 6131903 0%
SMB2IPCWrite 6118746 0%
SMB2QueryDir 489200375 4%
SMB2QueryFileBasicInfo 649118 0%
SMB2QueryFileStndInfo 429367 0%
SMB2QueryFileIntInfo 52022521 0%
SMB2QueryFileEAInfo 58870195 0%
SMB2QueryFileFEAInfo 18354 0%
SMB2QueryFileModeInfo 0 0%
SMB2QueryAltNameInfo 0 0%
SMB2QueryFileStreamInfo 29477570 0%
SMB2QueryNetOpenInfo 2228774898 19%
SMB2QueryAttrTagInfo 4040 0%
SMB2QueryAccessInfo 0 0%
SMB2QueryFileUnsupported 7101 0%
SMB2QueryFileInvalid 287481 0%
SMB2QueryFSVolInfo 47603358 0%
SMB2QueryFSSizeInfo 9571648 0%
SMB2QueryFSDevInfo 0 0%
SMB2QueryFSAttrInfo 47708805 0%
SMB2QueryFSFullSzInfo 762389 0%
SMB2QueryFSObjIdInfo 9 0%
SMB2QueryFSInvalid 612 0%
SMB2QuerySecurityInfo 24706489 0%
SMB2SetBasicInfo 42413683 0%
SMB2SetRenameInfo 2365987 0%
SMB2SetFileLinkInfo 114 0%
SMB2SetFileDispInfo 7358178 0%
SMB2SetFullEAInfo 487 0%
SMB2SetModeInfo 0 0%
SMB2SetAllocInfo 6140092 0%
SMB2SetEOFInfo 36221712 0%
SMB2SetUnsupported 9 0%
SMB2SetInfoInvalid 0 0%
SMB2SetSecurityInfo 2774540 0%
SMB2FsctlPipeTransceive 3784239 0%
SMB2FsctlPipePeek 0 0%
SMB2FsctlEnumSnapshots 65363 0%
SMB2FsctlDfsReferrals 40095211 0%
SMB2FsctlSetSparse 3 0%
SMB2FsctlSecureShare 557 0%
SMB2FsctlFileUnsupported 142738241 1%
SMB2FsctlIpcUnsupported 762592 0%
cancel lock 0
wait lock 0
copy to align 5979
alignedSmall 481216690
alignedLarge 3309867146
alignedSmallRel 0
alignedLargeRel 0
FidHashAllocs 307405
TidHashAllocs 537
UidHashAllocs 0
mbufWait 0
nbtWait 0
pBlkWait 0
BackToBackCPWait 0
cwaWait 0
short msg prevent 440015
multipleVCs 19354
SMB signing 2834344
mapped null user 0
PDCupcalls 0
nosupport 0
read pipe busy 0
write pipe busy 0
trans pipe busy 0
read pipe broken 0
write pipe broken 0
trans pipe broken 0
queued writeraw 0
nbt disconnect 4050847
smb disconnect 25226
dup disconnect 7979
OpLkBkXorBatchToL2 233576
OpLkBkXorBatchToNone 9
OpLkBkL2ToNone 33151
OpLkBkNoBreakAck 66
OpLkBkNoBreakAck95 0
OpLkBkNoBreakAckNT 66
OpLkBkIgnoredAck 31558
OpLkBkWaiterTimedOut 0
OpLkBkDelayedBreak 0
SharingErrorRetries 4890
FoldAttempts 0
FoldRenames 0
FoldRenameFailures 0
FoldOverflows 0
FoldDuplicates 0
FoldWAFLTooBusy 0
NoAllocCredStat 0
RetryRPCcollision 46
TconCloseTID 0
GetNTAPExtAttrs 0
SetNTAPExtAttrs 0
SearchBusy 0
ChgNfyNoMemory 0
ChgNfyNewWatch 3084954
ChgNfyLastWatch 3081988
UsedMIDTblCreated 130553
UnusedMIDTblCreated 37801
InvalidMIDRejects 160
SMB2InvalidSignature 0
SMB2DurableCreateReceived 3261297835
SMB2DurableCreateSucceeded 7471983
SMB2DurableReclaimReceived 1072
SMB2DurableReclaimSucceeded 730
SMB2DurableHandlePreserved 5783
SMB2DurableHandlePurged 170
SMB2DurableHandleExpired 4883
SMB2FileDirInfo 256963
SMB2FileFullDirInfo 664980
SMB2FileIdFullDirInfo 0
SMB2FileBothDirInfo 23762527
SMB2FileIdBothDirInfo 234424723
SMB2FileNamesInfo 2277175
SMB2FileDirUnsupported 0
SMB2QueryInfo 2502017464
SMB2SetInfo 97279941
SMB2Ioctl 187446208
SMB2RelatedCompRequest 332997174
SMB2UnRelatedCompRequest 0
SMB2FileRequest 2469575685
SMB2PipeRequest 87482026
SMB2_1_LeaseBreaks 3578437
SMB2_1_LeaseUpgrades 838538
SMB2_1_LeaseBreakExcuses 126533428
SMB2_1_LeaseBreakAckTimeouts 401917
SMB2_1_HandleLeaseBreaks 1094440
SMB2_1_LeaseBreaksToNone 904088
SMB2_1_LeaseBreakAcksIgnored 491
SMB2nosupport 16345407
Max Multiplex = 459, Max pBlk Exhaust = 0, Max pBlk Reserve Exhaust = 0
Max FIDs = 29449, Max FIDs on one tree = 9811
Max Searches on one tree = 23, Max Core Searches on one tree = 0
Max sessions = 4730
Max trees = 3131
Max shares = 35
Max session UIDs = 4, Max session TIDs = 41
Max locks = 30168
Max credentials = 4372
Max group SIDs per credential = 318
Max pBlks = 1344 Current pBlks = 1339 Num Logons = 0
Max reserved pBlks = 32 Current reserved pBlks = 32
Max gAuthQueue depth = 49
Max gSMBBlockingQueue depth = 82
Max gSMBTimerQueue depth = 13
Max gSMBAlfQueue depth = 3
Max gSMBRPCWorkerQueue depth = 15
Max gOffloadQueue depth = 6
Max gSMB2SigningQueue depth = 4
Local groups: builtins = 6, user-defined = 4, SIDs = 17
RPC group count = 40, RPC group active count = 0
Max Watched Directories = 4655, Current Watched Directories = 3191
Max Pending ChangeNotify Requests = 4280, Current Pending ChangeNotify Requests = 2881
Max Pending DeleteOnClose Requests = 4032, Current Pending DeleteOnClose Requests = 3

options cifs.max_mpx 253

should i increase this value to 1124?

would that help?

can i do that without terminating cifs?

thanks four your information and knowledge.

regards philipp

[avbohemen · Accepted solution]

Yes, setting max_mpx to 1124 will definately help. The storage system receives requests for 459 concurrent operations in one session, but it will only handle 253, so clients are delayed unless you raise max_mpx.

Which ONTAP version are you on? I see you are using SMB2.1. NetApp has had several bugs with SMB2.1, which are only solved in the very latest version, 8.2.3 (for 7-mode). If you are considering an upgrade, use 8.2.3P1, as it solves a bug in SNMP for use with OnCommand Unified Manager 5.x.

For more info on SMB2.1 in 8.2.3 see https://library.netapp.com/ecmdocs/ECMP12365050/html/GUID-B71EF086-A4EC-4A2E-86D0-D5AC921C8C82.html and https://kb.netapp.com/support/index?page=content&id=1015064

[philipp_pluess]

thanks for your reply.

we are currently running 8.1.4P1 and are updating to the latest 8.1.4 shortly. we cannot upgrade to 8.2.x as we have some old exchange in our domain.

so disabling smb2.1 would be the way to go, isn't it?

many thanks again 

regards

philipp

[avbohemen]

I see. Yes, then disabling SMB2.1 is the way to go. If you are a partner, you can lookup Support Bulletin 7010114 for the procedure. Otherwise, contact NetApp Support.

[philipp_pluess]

other short question, netapp isn't that consistent in giving the answer:

is it save to change cifs.max_mpx with cifs not beeing terminated?

this one

https://kb.netapp.com/index?page=content&id=3011249&actp=LIST_POPULAR

says i could do it and only new sessions would take the new value

on the other side, options tells

"CAUTION - This number should only be changed while cifs is terminated."

has anyone tried this during cifs running?

thanks

philipp

[avbohemen]

To be honest, I always change it with CIFS running. Most cifs options, including max_mpx, are effective immediately for new sessions. To be sure you have a new session, you must reboot the client though (which could be a windows terminal server).