Hi
I have inherited some NetApp filers with a mixture of CIFS and NFS exports and this morning I received some complaints about performance issues when copying files from multiple windows hosts to the CIFS shares. It seems that maximum speeds being seen are around 30MBps which is causing some teams issues as they move large files to and from the filers regularly.
The network team have been involved and they have cleared the network of any issues.
The filer is set up as follows:
NetApp Release 8.1.2 7-Mode
Windows 2003 Domain
2 x 1GB NICS LACP
I have been researching the issue from a filer end and this what I have done so far:
ran systat when copying files - CPU usage was around 60% mark
checked cifs options:
cifs.LMCompatibilityLevel 1
cifs.W2K_password_change off
cifs.W2K_password_change_interval 4w
cifs.W2K_password_change_within 3600h
cifs.audit.account_mgmt_events.enable off
cifs.audit.autosave.file.extension
cifs.audit.autosave.file.limit 0
cifs.audit.autosave.onsize.enable off
cifs.audit.autosave.onsize.threshold 75%
cifs.audit.autosave.ontime.enable off
cifs.audit.autosave.ontime.interval 1d
cifs.audit.enable off
cifs.audit.file_access_events.enable on
cifs.audit.liveview.allowed_users
cifs.audit.liveview.enable off
cifs.audit.logon_events.enable on
cifs.audit.logsize 1048576
cifs.audit.nfs.enable off
cifs.audit.nfs.filter.filename
cifs.audit.saveas /etc/log/adtlog.evt
cifs.bypass_traverse_checking on
cifs.client.dup-detection ip-address
cifs.comment
cifs.enable_share_browsing on
cifs.gpo.enable off
cifs.gpo.trace.enable off
cifs.grant_implicit_exe_perms off
cifs.guest_account
cifs.home_dir_namestyle hidden
cifs.home_dirs_public_for_admin on
cifs.idle_timeout 900
cifs.ipv6.enable off
cifs.max_mpx 255
cifs.ms_snapshot_mode xp
cifs.netbios_aliases
cifs.netbios_over_tcp.enable on
cifs.nfs_root_ignore_acl off
cifs.oplocks.enable on
cifs.oplocks.opendelta 0
cifs.per_client_stats.enable on
cifs.perfmon.allowed_users
cifs.perm_check_ro_del_ok off
cifs.perm_check_use_gid on
cifs.preserve_unix_security off
cifs.restrict_anonymous 0
cifs.restrict_anonymous.enable off
cifs.save_case on
cifs.scopeid
cifs.search_domains
cifs.show_dotfiles on
cifs.show_snapshot off
cifs.shutdown_msg_level 2
cifs.sidcache.enable on
cifs.sidcache.lifetime 1440
cifs.signing.enable off
cifs.smb2.enable off
cifs.smb2.signing.required off
cifs.smb2_1.branch_cache.enable off
cifs.smb2_1.branch_cache.hash_time_out 3600 (value might be overwritten in takeover)
cifs.snapshot_file_folding.enable off
cifs.symlinks.cycleguard on
cifs.symlinks.enable on
cifs.trace_dc_connection off
cifs.trace_login off
cifs.universal_nested_groups.enable on
cifs.widelink.ttl 10m
In /etc/messages I can see the following lines which I think are causing us problems:
[xxxxx:vscan.virus.detected:error]: CIFS: Possible Virus Detected
[xxxxx:cifs.oplock.break.timeout:warning]
and in cifs stat I can see the following line:
OpLkBkNoBreakAckNT 106
below is the full output
reject 10 0%
mkdir 54 0%
rmdir 15834 0%
open 0 0%
create 0 0%
close 52156559 7%
X&close 0 0%
flush 296023 0%
X&flush 0 0%
delete 135550 0%
rename 151025 0%
NTRename 0 0%
getatr 6225 0%
setatr 0 0%
read 0 0%
X&read 0 0%
write 4183 0%
X&write 0 0%
lock 0 0%
unlock 0 0%
mknew 0 0%
chkpth 538 0%
exit 0 0%
lseek 0 0%
lockread 0 0%
X&lockread 0 0%
writeunlock 0 0%
readbraw 0 0%
writebraw 0 0%
writec 0 0%
gettattre 0 0%
settattre 0 0%
lockingX 2108257 0%
IPC 2253559 0%
open2 0 0%
find_first2 54941892 8%
find_next2 882988 0%
query_fs_info 2504167 0%
query_path_info 212338402 30%
set_path_info 0 0%
query_file_info 83846879 12%
set_file_info 5709567 1%
create_dir2 0 0%
Dfs_referral 390994 0%
Dfs_report 0 0%
echo 2281704 0%
writeclose 0 0%
openX 0 0%
readX 178701014 25%
writeX 39636900 6%
findclose 0 0%
tcon 0 0%
tdis 272871 0%
negprot 98020 0%
login 128524 0%
logout 35938 0%
tconX 433031 0%
dskattr 0 0%
search 0 0%
fclose 25639 0%
NTCreateX 61632267 9%
NTTransCreate 15438 0%
NTTransIoctl 1484179 0%
NTTransNotify 624750 0%
NTTransSetSec 1449022 0%
NTTransQuerySec 9239161 1%
NTNamedPipeMulti 0 0%
NTCancel CN 377285 0%
NTCancel Other 203 0%
SMB2Echo 0 0%
SMB2Negprot 0 0%
SMB2TreeConnnect 0 0%
SMB2TreeDisconnect 0 0%
SMB2Login 0 0%
SMB2Create 0 0%
SMB2Read 0 0%
SMB2Write 0 0%
SMB2Lock 0 0%
SMB2Unlock 0 0%
SMB2OplkBrkAck 0 0%
SMB2ChgNfy 0 0%
SMB2CLose 0 0%
SMB2Flush 0 0%
SMB2Logout 0 0%
SMB2Cancel 0 0%
SMB2IPCCreate 0 0%
SMB2IPCRead 0 0%
SMB2IPCWrite 0 0%
SMB2QueryDir 0 0%
SMB2QueryFileBasicInfo 0 0%
SMB2QueryFileStndInfo 0 0%
SMB2QueryFileIntInfo 0 0%
SMB2QueryFileEAInfo 0 0%
SMB2QueryFileFEAInfo 0 0%
SMB2QueryFileModeInfo 0 0%
SMB2QueryAltNameInfo 0 0%
SMB2QueryFileStreamInfo 0 0%
SMB2QueryNetOpenInfo 0 0%
SMB2QueryAttrTagInfo 0 0%
SMB2QueryAccessInfo 0 0%
SMB2QueryFileUnsupported 0 0%
SMB2QueryFileInvalid 0 0%
SMB2QueryFSVolInfo 0 0%
SMB2QueryFSSizeInfo 0 0%
SMB2QueryFSDevInfo 0 0%
SMB2QueryFSAttrInfo 0 0%
SMB2QueryFSFullSzInfo 0 0%
SMB2QueryFSObjIdInfo 0 0%
SMB2QueryFSInvalid 0 0%
SMB2QuerySecurityInfo 0 0%
SMB2SetBasicInfo 0 0%
SMB2SetRenameInfo 0 0%
SMB2SetFileLinkInfo 0 0%
SMB2SetFileDispInfo 0 0%
SMB2SetFullEAInfo 0 0%
SMB2SetModeInfo 0 0%
SMB2SetAllocInfo 0 0%
SMB2SetEOFInfo 0 0%
SMB2SetUnsupported 0 0%
SMB2SetInfoInvalid 0 0%
SMB2SetSecurityInfo 0 0%
SMB2FsctlPipeTransceive 0 0%
SMB2FsctlPipePeek 0 0%
SMB2FsctlEnumSnapshots 0 0%
SMB2FsctlDfsReferrals 0 0%
SMB2FsctlSetSparse 0 0%
SMB2FsctlSecureShare 0 0%
SMB2FsctlFileUnsupported 0 0%
SMB2FsctlIpcUnsupported 0 0%
cancel lock 0
wait lock 0
copy to align 275978
alignedSmall 417926
alignedLarge 89143
alignedSmallRel 0
alignedLargeRel 0
FidHashAllocs 25513
TidHashAllocs 99
UidHashAllocs 0
mbufWait 0
nbtWait 0
pBlkWait 0
BackToBackCPWait 0
cwaWait 0
short msg prevent 1732
multipleVCs 60277
SMB signing 0
mapped null user 0
PDCupcalls 0
nosupport 0
read pipe busy 0
write pipe busy 0
trans pipe busy 0
read pipe broken 0
write pipe broken 0
trans pipe broken 0
queued writeraw 0
nbt disconnect 27406
smb disconnect 67285
dup disconnect 3145
OpLkBkXorBatchToL2 244417
OpLkBkXorBatchToNone 25
OpLkBkL2ToNone 40963
OpLkBkNoBreakAck 106
OpLkBkNoBreakAck95 0
OpLkBkNoBreakAckNT 106
OpLkBkIgnoredAck 283
OpLkBkWaiterTimedOut 0
OpLkBkDelayedBreak 0
SharingErrorRetries 27775
FoldAttempts 0
FoldRenames 0
FoldRenameFailures 0
FoldOverflows 0
FoldDuplicates 0
FoldWAFLTooBusy 0
NoAllocCredStat 0
RetryRPCcollision 1
TconCloseTID 0
GetNTAPExtAttrs 6
SetNTAPExtAttrs 0
SearchBusy 0
ChgNfyNoMemory 0
ChgNfyNewWatch 295743
ChgNfyLastWatch 295498
UsedMIDTblCreated 0
UnusedMIDTblCreated 0
InvalidMIDRejects 0
SMB2InvalidSignature 0
SMB2DurableCreateReceived 0
SMB2DurableCreateSucceeded 0
SMB2DurableReclaimReceived 0
SMB2DurableReclaimSucceeded 0
SMB2DurableHandlePreserved 0
SMB2DurableHandlePurged 0
SMB2DurableHandleExpired 0
SMB2FileDirInfo 0
SMB2FileFullDirInfo 0
SMB2FileIdFullDirInfo 0
SMB2FileBothDirInfo 0
SMB2FileIdBothDirInfo 0
SMB2FileNamesInfo 0
SMB2FileDirUnsupported 0
SMB2QueryInfo 0
SMB2SetInfo 0
SMB2Ioctl 0
SMB2RelatedCompRequest 0
SMB2UnRelatedCompRequest 0
SMB2FileRequest 0
SMB2PipeRequest 0
SMB2_1_LeaseBreaks 0
SMB2_1_LeaseUpgrades 0
SMB2_1_LeaseBreakExcuses 0
SMB2_1_LeaseBreakAckTimeouts 0
SMB2_1_HandleLeaseBreaks 0
SMB2_1_LeaseBreaksToNone 0
SMB2_1_LeaseBreakAcksIgnored 0
SMB2nosupport 0
Max Multiplex = 49, Max pBlk Exhaust = 0, Max pBlk Reserve Exhaust = 0
Max FIDs = 9519, Max FIDs on one tree = 5069
Max Searches on one tree = 11, Max Core Searches on one tree = 0
Max sessions = 343
Max trees = 931
Max shares = 102
Max session UIDs = 13, Max session TIDs = 82
Max locks = 9701
Max credentials = 496
Max group SIDs per credential = 898
Max pBlks = 1024 Current pBlks = 1024 Num Logons = 0
Max reserved pBlks = 32 Current reserved pBlks = 32
Max gAuthQueue depth = 2
Max gSMBBlockingQueue depth = 5
Max gSMBTimerQueue depth = 5
Max gSMBAlfQueue depth = 1
Max gSMBRPCWorkerQueue depth = 12
Max gOffloadQueue depth = 18
Local groups: builtins = 6, user-defined = 5, SIDs = 42
RPC group count = 10, RPC group active count = 3
Max Watched Directories = 383, Current Watched Directories = 280
Max Pending ChangeNotify Requests = 353, Current Pending ChangeNotify Requests = 262
Max Pending DeleteOnClose Requests = 3072, Current Pending DeleteOnClose Requests = 0
The volume on which the shares are presented from has plenty of space and we have plenty of inodes left
also stats show cifs shows latency of less than 1ms when I was running the file copy.
I know a bit of NetApp but not enough to consider myself an exprt so I am just looking for pointers on what other bits I should be looking into and if the following could be causing the performance issue and how I can resolve them:
[xxxxx:vscan.virus.detected:error]: CIFS: Possible Virus Detected
[xxxxx:cifs.oplock.break.timeout:warning]
OpLkBkNoBreakAckNT 106
Any help will be much appreciated
