Hi,
we have a FAS2220 with 2 controllers, both have the same setup, same network settings, same cifs settings, plugged in the same switch, switch ports the same settings, connected to the same preferred DC, in general all settings the same but CIFS has a big difference.
Read/Write on a CIFS share from first controller goes 100MBps+, while on the other barely reaches 20-30 and goes up and down.
Strange thing is that there is no performance hit on the second controller for read/write on a VM shared disk over iSCSI.
Here are cifs stat for First controller (good one)
| | reject | 0 0% |
| | mkdir | 0 0% |
| | rmdir | 0 0% |
| | open | 0 0% |
| | create | 0 0% |
| | close | 0 0% |
| | X&close | 0 0% |
| | flush | 0 0% |
| | X&flush | 0 0% |
| | delete | 0 0% |
| | rename | 0 0% |
| | NTRename | 0 0% |
| | getatr | 0 0% |
| | setatr | 0 0% |
| | read | 0 0% |
| | X&read | 0 0% |
| | write | 0 0% |
| | X&write | 0 0% |
| | lock | 0 0% |
| | unlock | 0 0% |
| | mknew | 0 0% |
| | chkpth | 0 0% |
| | exit | 0 0% |
| | lseek | 0 0% |
| | lockread | 0 0% |
| | X&lockread | 0 0% |
| | writeunlock | 0 0% |
| | readbraw | 0 0% |
| | writebraw | 0 0% |
| | writec | 0 0% |
| | gettattre | 0 0% |
| | settattre | 0 0% |
| | lockingX | 0 0% |
| | IPC | 192 1% |
| | open2 | 0 0% |
| | find_first2 | 0 0% |
| | find_next2 | 0 0% |
| | query_fs_info | 0 0% |
| | query_path_info | 0 0% |
| | set_path_info | 0 0% |
| | query_file_info | 0 0% |
| | set_file_info | 0 0% |
| | create_dir2 | 0 0% |
| | Dfs_referral | 0 0% |
| | Dfs_report | 0 0% |
| | echo | 0 0% |
| | writeclose | 0 0% |
| | openX | 0 0% |
| | readX | 0 0% |
| | writeX | 0 0% |
| | findclose | 0 0% |
| | tcon | 0 0% |
| | tdis | 33 0% |
| | negprot | 39 0% |
| | login | 4 0% |
| | logout | 0 0% |
| | tconX | 2 0% |
| | dskattr | 0 0% |
| | search | 0 0% |
| | fclose | 0 0% |
| | NTCreateX | 16 0% |
| | NTTransCreate | 0 0% |
| | NTTransIoctl | 0 0% |
| | NTTransNotify | 0 0% |
| | NTTransSetSec | 0 0% |
| | NTTransQuerySec | 0 0% |
| | NTNamedPipeMulti | 0 0% |
| | NTCancel CN | 0 0% |
| | NTCancel Other | 0 0% |
| | SMB2Echo | 0 0% |
| | SMB2Negprot | 41 0% |
| | SMB2TreeConnnect | 62 0% |
| | SMB2TreeDisconnect | 33 0% |
| | SMB2Login | 42 0% |
| | SMB2Create | 2163 6% |
| | SMB2Read | 17042 44% |
| | SMB2Write | 12915 34% |
| | SMB2Lock | 0 0% |
| | SMB2Unlock | 0 0% |
| | SMB2OplkBrkAck | 0 0% |
| | SMB2ChgNfy | 86 0% |
| | SMB2CLose | 1914 5% |
| | SMB2Flush | 3 0% |
| | SMB2Logout | 11 0% |
| | SMB2Cancel | 59 0% |
| | SMB2IPCCreate | 7 0% |
| | SMB2IPCRead | 7 0% |
| | SMB2IPCWrite | 7 0% |
| | SMB2QueryDir | 1500 4% |
| | SMB2QueryFileBasicInfo | 0 0% |
| | SMB2QueryFileStndInfo | 0 0% |
| | SMB2QueryFileIntInfo | 45 0% |
| | SMB2QueryFileEAInfo | 55 0% |
| | SMB2QueryFileFEAInfo | 0 0% |
| | SMB2QueryFileModeInfo | 0 0% |
| | SMB2QueryAltNameInfo | 0 0% |
| | SMB2QueryFileStreamInfo | 21 0% |
| | SMB2QueryNetOpenInfo | 209 1% |
| | SMB2QueryAttrTagInfo | 0 0% |
| | SMB2QueryAccessInfo | 0 0% |
| | SMB2QueryFileUnsupported | 0 0% |
| | SMB2QueryFileInvalid | 0 0% |
| | SMB2QueryFSVolInfo | 70 0% |
| | SMB2QueryFSSizeInfo | 10 0% |
| | SMB2QueryFSDevInfo | 0 0% |
| | SMB2QueryFSAttrInfo | 70 0% |
| | SMB2QueryFSFullSzInfo | 38 0% |
| | SMB2QueryFSObjIdInfo | 0 0% |
| | SMB2QueryFSInvalid | 0 0% |
| | SMB2QuerySecurityInfo | 106 0% |
| | SMB2SetBasicInfo | 30 0% |
| | SMB2SetRenameInfo | 25 0% |
| | SMB2SetFileLinkInfo | 0 0% |
| | SMB2SetFileDispInfo | 14 0% |
| | SMB2SetFullEAInfo | 0 0% |
| | SMB2SetModeInfo | 0 0% |
| | SMB2SetAllocInfo | 13 0% |
| | SMB2SetEOFInfo | 18 0% |
| | SMB2SetUnsupported | 0 0% |
| | SMB2SetInfoInvalid | 0 0% |
| | SMB2SetSecurityInfo | 22 0% |
| | SMB2FsctlPipeTransceive | 1331 3% |
| | SMB2FsctlPipePeek | 0 0% |
| | SMB2FsctlEnumSnapshots | 0 0% |
| | SMB2FsctlDfsReferrals | 4 0% |
| | SMB2FsctlSetSparse | 0 0% |
| | SMB2FsctlSecureShare | 0 0% |
| | SMB2FsctlFileUnsupported | 127 0% |
| | SMB2FsctlIpcUnsupported | 0 0% |
| | cancel lock | 0 |
| | wait lock | 0 |
| | copy to align | 0 |
| | alignedSmall | 2250 |
| | alignedLarge | 2272 |
| | alignedSmallRel | 0 |
| | alignedLargeRel | 0 |
| | FidHashAllocs | 0 |
| | TidHashAllocs | 0 |
| | UidHashAllocs | 0 |
| | mbufWait | 0 |
| | nbtWait | 0 |
| | pBlkWait | 0 |
| | BackToBackCPWait | 0 |
| | cwaWait | 0 |
| | short msg prevent | 2 |
| | multipleVCs | 0 |
| | SMB signing | 41 |
| | mapped null user | 0 |
| | PDCupcalls | 0 |
| | nosupport | 0 |
| | read pipe busy | 0 |
| | write pipe busy | 0 |
| | trans pipe busy | 0 |
| | read pipe broken | 0 |
| | write pipe broken | 0 |
| | trans pipe broken | 0 |
| | queued writeraw | 0 |
| | nbt disconnect | 16 |
| | smb disconnect | 17 |
| | dup disconnect | 0 |
| | OpLkBkXorBatchToL2 | 0 |
| | OpLkBkXorBatchToNone | 0 |
| | OpLkBkL2ToNone | 0 |
| | OpLkBkNoBreakAck | 0 |
| | OpLkBkNoBreakAck95 | 0 |
| | OpLkBkNoBreakAckNT | 0 |
| | OpLkBkIgnoredAck | 0 |
| | OpLkBkWaiterTimedOut | 0 |
| | OpLkBkDelayedBreak | 0 |
| | SharingErrorRetries | 0 |
| | FoldAttempts | 0 |
| | FoldRenames | 0 |
| | FoldRenameFailures | 0 |
| | FoldOverflows | 0 |
| | FoldDuplicates | 0 |
| | FoldWAFLTooBusy | 0 |
| | NoAllocCredStat | 0 |
| | RetryRPCcollision | 0 |
| | TconCloseTID | 0 |
| | GetNTAPExtAttrs | 0 |
| | SetNTAPExtAttrs | 0 |
| | SearchBusy | 0 |
| | ChgNfyNoMemory | 0 |
| | ChgNfyNewWatch | 67 |
| | ChgNfyLastWatch | 57 |
| | UsedMIDTblCreated | 2 |
| | UnusedMIDTblCreated | 0 |
| | InvalidMIDRejects | 0 |
| | SMB2InvalidSignature | 0 |
| | SMB2DurableCreateReceived | 2126 |
| | SMB2DurableCreateSucceeded | 227 |
| | SMB2DurableReclaimReceived | 0 |
| | SMB2DurableReclaimSucceeded | 0 |
| | SMB2DurableHandlePreserved | 0 |
| | SMB2DurableHandlePurged | 0 |
| | SMB2DurableHandleExpired | 0 |
| | SMB2FileDirInfo | 0 |
| | SMB2FileFullDirInfo | 0 |
| | SMB2FileIdFullDirInfo | 0 |
| | SMB2FileBothDirInfo | 0 |
| | SMB2FileIdBothDirInfo | 750 |
| | SMB2FileNamesInfo | 0 |
| | SMB2FileDirUnsupported | 0 |
| | SMB2QueryInfo | 624 |
| | SMB2SetInfo | 122 |
| | SMB2Ioctl | 1462 |
| | SMB2RelatedCompRequest | 1018 |
| | SMB2UnRelatedCompRequest | 0 |
| | SMB2FileRequest | 36520 |
| | SMB2PipeRequest | 1390 |
| | SMB2_1_LeaseBreaks | 0 |
| | SMB2_1_LeaseUpgrades | 0 |
| | SMB2_1_LeaseBreakExcuses | 135 |
| | SMB2_1_LeaseBreakAckTimeouts | 0 |
| | SMB2_1_HandleLeaseBreaks | 0 |
| | SMB2_1_LeaseBreaksToNone | 0 |
| | SMB2_1_LeaseBreakAcksIgnored | 0 |
| SMB2_1_StatLeaseWorkObjAllocFail | 0 |
| | SMB2nosupport | 25 |
Max Multiplex = 50, Max pBlk Exhaust = 0, Max pBlk Reserve Exhaust = 0
Max FIDs = 22, Max FIDs on one tree = 13
Max Searches on one tree = 0, Max Core Searches on one tree = 0
Max sessions = 17
Max trees = 16
Max shares = 6
Max session UIDs = 1, Max session TIDs = 2
Max locks = 28
Max credentials = 16
Max group SIDs per credential = 71
Max pBlks = 1024 Current pBlks = 1024 Num Logons = 0
Max reserved pBlks = 32 Current reserved pBlks = 32
Max gSMBBlockingQueue depth = 1
| Max gSMBTimerQueue depth | = 5 |
| Max gSMBAlfQueue depth | = 1 |
Max gSMBRPCWorkerQueue depth = 1
| Max gOffloadQueue depth | = 2 |
Max gSMB2SigningQueue depth = 0
Local groups: builtins = 6, user-defined = 1, SIDs = 4
RPC group count = 10, RPC group active count = 1
Max Watched Directories = 11, Current Watched Directories = 10
Max Pending ChangeNotify Requests = 10, Current Pending ChangeNotify Requests = 9
Max Pending DeleteOnClose Requests = 3072, Current Pending DeleteOnClose Requests = 0
and cifs stat for second controller (bad)
| | reject | 0 0% |
| | mkdir | 0 0% |
| | rmdir | 0 0% |
| | open | 0 0% |
| | create | 0 0% |
| | close | 0 0% |
| | X&close | 0 0% |
| | flush | 0 0% |
| | X&flush | 0 0% |
| | delete | 0 0% |
| | rename | 0 0% |
| | NTRename | 0 0% |
| | getatr | 0 0% |
| | setatr | 0 0% |
| | read | 0 0% |
| | X&read | 0 0% |
| | write | 0 0% |
| | X&write | 0 0% |
| | lock | 0 0% |
| | unlock | 0 0% |
| | mknew | 0 0% |
| | chkpth | 0 0% |
| | exit | 0 0% |
| | lseek | 0 0% |
| | lockread | 0 0% |
| | X&lockread | 0 0% |
| | writeunlock | 0 0% |
| | readbraw | 0 0% |
| | writebraw | 0 0% |
| | writec | 0 0% |
| | gettattre | 0 0% |
| | settattre | 0 0% |
| | lockingX | 0 0% |
| | IPC | 192 1% |
| | open2 | 0 0% |
| | find_first2 | 0 0% |
| | find_next2 | 0 0% |
| | query_fs_info | 0 0% |
| | query_path_info | 0 0% |
| | set_path_info | 0 0% |
| | query_file_info | 0 0% |
| | set_file_info | 0 0% |
| | create_dir2 | 0 0% |
| | Dfs_referral | 0 0% |
| | Dfs_report | 0 0% |
| | echo | 0 0% |
| | writeclose | 0 0% |
| | openX | 0 0% |
| | readX | 0 0% |
| | writeX | 0 0% |
| | findclose | 0 0% |
| | tcon | 0 0% |
| | tdis | 33 0% |
| | negprot | 30 0% |
| | login | 4 0% |
| | logout | 0 0% |
| | tconX | 2 0% |
| | dskattr | 0 0% |
| | search | 0 0% |
| | fclose | 0 0% |
| | NTCreateX | 16 0% |
| | NTTransCreate | 0 0% |
| | NTTransIoctl | 0 0% |
| | NTTransNotify | 0 0% |
| | NTTransSetSec | 0 0% |
| | NTTransQuerySec | 0 0% |
| | NTNamedPipeMulti | 0 0% |
| | NTCancel CN | 0 0% |
| | NTCancel Other | 0 0% |
| | SMB2Echo | 0 0% |
| | SMB2Negprot | 36 0% |
| | SMB2TreeConnnect | 61 0% |
| | SMB2TreeDisconnect | 33 0% |
| | SMB2Login | 45 0% |
| | SMB2Create | 2310 8% |
| | SMB2Read | 18405 64% |
| | SMB2Write | 339 1% |
| | SMB2Lock | 1 0% |
| | SMB2Unlock | 0 0% |
| | SMB2OplkBrkAck | 0 0% |
| | SMB2ChgNfy | 61 0% |
| | SMB2CLose | 2142 7% |
| | SMB2Flush | 2 0% |
| | SMB2Logout | 4 0% |
| | SMB2Cancel | 385 1% |
| | SMB2IPCCreate | 23 0% |
| | SMB2IPCRead | 23 0% |
| | SMB2IPCWrite | 23 0% |
| | SMB2QueryDir | 2778 10% |
| | SMB2QueryFileBasicInfo | 0 0% |
| | SMB2QueryFileStndInfo | 0 0% |
| | SMB2QueryFileIntInfo | 8 0% |
| | SMB2QueryFileEAInfo | 18 0% |
| | SMB2QueryFileFEAInfo | 0 0% |
| | SMB2QueryFileModeInfo | 0 0% |
| | SMB2QueryAltNameInfo | 0 0% |
| | SMB2QueryFileStreamInfo | 18 0% |
| | SMB2QueryNetOpenInfo | 111 0% |
| | SMB2QueryAttrTagInfo | 0 0% |
| | SMB2QueryAccessInfo | 0 0% |
| | SMB2QueryFileUnsupported | 0 0% |
| | SMB2QueryFileInvalid | 0 0% |
| | SMB2QueryFSVolInfo | 133 0% |
| | SMB2QueryFSSizeInfo | 8 0% |
| | SMB2QueryFSDevInfo | 0 0% |
| | SMB2QueryFSAttrInfo | 133 0% |
| | SMB2QueryFSFullSzInfo | 11 0% |
| | SMB2QueryFSObjIdInfo | 0 0% |
| | SMB2QueryFSInvalid | 0 0% |
| | SMB2QuerySecurityInfo | 77 0% |
| | SMB2SetBasicInfo | 24 0% |
| | SMB2SetRenameInfo | 17 0% |
| | SMB2SetFileLinkInfo | 0 0% |
| | SMB2SetFileDispInfo | 12 0% |
| | SMB2SetFullEAInfo | 0 0% |
| | SMB2SetModeInfo | 0 0% |
| | SMB2SetAllocInfo | 3 0% |
| | SMB2SetEOFInfo | 12 0% |
| | SMB2SetUnsupported | 0 0% |
| | SMB2SetInfoInvalid | 0 0% |
| | SMB2SetSecurityInfo | 16 0% |
| | SMB2FsctlPipeTransceive | 1287 4% |
| | SMB2FsctlPipePeek | 0 0% |
| | SMB2FsctlEnumSnapshots | 0 0% |
| | SMB2FsctlDfsReferrals | 10 0% |
| | SMB2FsctlSetSparse | 0 0% |
| | SMB2FsctlSecureShare | 0 0% |
| | SMB2FsctlFileUnsupported | 36 0% |
| | SMB2FsctlIpcUnsupported | 21 0% |
| | cancel lock | 0 |
| | wait lock | 0 |
| | copy to align | 0 |
| | alignedSmall | 2847 |
| | alignedLarge | 2389 |
| | alignedSmallRel | 0 |
| | alignedLargeRel | 0 |
| | FidHashAllocs | 0 |
| | TidHashAllocs | 0 |
| | UidHashAllocs | 0 |
| | mbufWait | 0 |
| | nbtWait | 0 |
| | pBlkWait | 0 |
| | BackToBackCPWait | 0 |
| | cwaWait | 0 |
| | short msg prevent | 5 |
| | multipleVCs | 0 |
| | SMB signing | 32 |
| | mapped null user | 0 |
| | PDCupcalls | 0 |
| | nosupport | 0 |
| | read pipe busy | 0 |
| | write pipe busy | 0 |
| | trans pipe busy | 0 |
| | read pipe broken | 0 |
| | write pipe broken | 0 |
| | trans pipe broken | 0 |
| | queued writeraw | 0 |
| | nbt disconnect | 13 |
| | smb disconnect | 21 |
| | dup disconnect | 0 |
| | OpLkBkXorBatchToL2 | 0 |
| | OpLkBkXorBatchToNone | 0 |
| | OpLkBkL2ToNone | 0 |
| | OpLkBkNoBreakAck | 0 |
| | OpLkBkNoBreakAck95 | 0 |
| | OpLkBkNoBreakAckNT | 0 |
| | OpLkBkIgnoredAck | 0 |
| | OpLkBkWaiterTimedOut | 0 |
| | OpLkBkDelayedBreak | 0 |
| | SharingErrorRetries | 0 |
| | FoldAttempts | 0 |
| | FoldRenames | 0 |
| | FoldRenameFailures | 0 |
| | FoldOverflows | 0 |
| | FoldDuplicates | 0 |
| | FoldWAFLTooBusy | 0 |
| | NoAllocCredStat | 0 |
| | RetryRPCcollision | 0 |
| | TconCloseTID | 0 |
| | GetNTAPExtAttrs | 0 |
| | SetNTAPExtAttrs | 0 |
| | SearchBusy | 0 |
| | ChgNfyNoMemory | 0 |
| | ChgNfyNewWatch | 50 |
| | ChgNfyLastWatch | 47 |
| | UsedMIDTblCreated | 6 |
| | UnusedMIDTblCreated | 0 |
| | InvalidMIDRejects | 0 |
| | SMB2InvalidSignature | 0 |
| | SMB2DurableCreateReceived | 2286 |
| | SMB2DurableCreateSucceeded | 135 |
| | SMB2DurableReclaimReceived | 0 |
| | SMB2DurableReclaimSucceeded | 0 |
| | SMB2DurableHandlePreserved | 0 |
| | SMB2DurableHandlePurged | 0 |
| | SMB2DurableHandleExpired | 0 |
| | SMB2FileDirInfo | 0 |
| | SMB2FileFullDirInfo | 12 |
| | SMB2FileIdFullDirInfo | 0 |
| | SMB2FileBothDirInfo | 5 |
| | SMB2FileIdBothDirInfo | 1414 |
| | SMB2FileNamesInfo | 0 |
| | SMB2FileDirUnsupported | 0 |
| | SMB2QueryInfo | 517 |
| | SMB2SetInfo | 84 |
| | SMB2Ioctl | 1354 |
| | SMB2RelatedCompRequest | 1579 |
| | SMB2UnRelatedCompRequest | 0 |
| | SMB2FileRequest | 27016 |
| | SMB2PipeRequest | 1548 |
| | SMB2_1_LeaseBreaks | 1 |
| | SMB2_1_LeaseUpgrades | 0 |
| | SMB2_1_LeaseBreakExcuses | 92 |
| | SMB2_1_LeaseBreakAckTimeouts | 0 |
| | SMB2_1_HandleLeaseBreaks | 0 |
| | SMB2_1_LeaseBreaksToNone | 1 |
| | SMB2_1_LeaseBreakAcksIgnored | 0 |
| SMB2_1_StatLeaseWorkObjAllocFail | 0 |
| | SMB2nosupport | 128 |
Max Multiplex = 50, Max pBlk Exhaust = 0, Max pBlk Reserve Exhaust = 0
Max FIDs = 18, Max FIDs on one tree = 14
Max Searches on one tree = 0, Max Core Searches on one tree = 0
Max sessions = 16
Max trees = 18
Max shares = 7
Max session UIDs = 1, Max session TIDs = 4
Max locks = 25
Max credentials = 15
Max group SIDs per credential = 71
Max pBlks = 1024 Current pBlks = 1024 Num Logons = 0
Max reserved pBlks = 32 Current reserved pBlks = 32
Max gSMBBlockingQueue depth = 1
| Max gSMBTimerQueue depth | = 6 |
| Max gSMBAlfQueue depth | = 1 |
Max gSMBRPCWorkerQueue depth = 1
| Max gOffloadQueue depth | = 2 |
Max gSMB2SigningQueue depth = 0
Local groups: builtins = 6, user-defined = 1, SIDs = 4
RPC group count = 10, RPC group active count = 1
Max Watched Directories = 6, Current Watched Directories = 3
Max Pending ChangeNotify Requests = 4, Current Pending ChangeNotify Requests = 3
Max Pending DeleteOnClose Requests = 3072, Current Pending DeleteOnClose Requests = 0
On the second controller I'm specifically doing the test on a newly created volume without dedup. Tried already to disable vscan but changes nothing, besides the first controller performs great with dedup and vscan enabled, as I said all settings the same.
Doing a quick packet sniff on the client I see that after a couple of seconds in a file transfer sessions there are only read requests from the client and no responses from the filer and then it responds again and then the same all over again.
Any help appreciated.
