FAS2220 CIFS 2 controllers same settings big performance difference

g_billios · ‎2014-09-02

Hi,

we have a FAS2220 with 2 controllers, both have the same setup, same network settings, same cifs settings, plugged in the same switch, switch ports the same settings, connected to the same preferred DC, in general all settings the same but CIFS has a big difference.

Read/Write on a CIFS share from first controller goes 100MBps+, while on the other barely reaches 20-30 and goes up and down.

Strange thing is that there is no performance hit on the second controller for read/write on a VM shared disk over iSCSI.

Here are cifs stat for First controller (good one)

	reject	0 0%
	mkdir	0 0%
	rmdir	0 0%
	open	0 0%
	create	0 0%
	close	0 0%
	X&close	0 0%
	flush	0 0%
	X&flush	0 0%
	delete	0 0%
	rename	0 0%
	NTRename	0 0%
	getatr	0 0%
	setatr	0 0%
	read	0 0%
	X&read	0 0%
	write	0 0%
	X&write	0 0%
	lock	0 0%
	unlock	0 0%
	mknew	0 0%
	chkpth	0 0%
	exit	0 0%
	lseek	0 0%
	lockread	0 0%
	X&lockread	0 0%
	writeunlock	0 0%
	readbraw	0 0%
	writebraw	0 0%
	writec	0 0%
	gettattre	0 0%
	settattre	0 0%
	lockingX	0 0%
	IPC	192 1%
	open2	0 0%
	find_first2	0 0%
	find_next2	0 0%
	query_fs_info	0 0%
	query_path_info	0 0%
	set_path_info	0 0%
	query_file_info	0 0%
	set_file_info	0 0%
	create_dir2	0 0%
	Dfs_referral	0 0%
	Dfs_report	0 0%
	echo	0 0%
	writeclose	0 0%
	openX	0 0%
	readX	0 0%
	writeX	0 0%
	findclose	0 0%
	tcon	0 0%
	tdis	33 0%
	negprot	39 0%
	login	4 0%
	logout	0 0%
	tconX	2 0%
	dskattr	0 0%
	search	0 0%
	fclose	0 0%
	NTCreateX	16 0%
	NTTransCreate	0 0%
	NTTransIoctl	0 0%
	NTTransNotify	0 0%
	NTTransSetSec	0 0%
	NTTransQuerySec	0 0%
	NTNamedPipeMulti	0 0%
	NTCancel CN	0 0%
	NTCancel Other	0 0%
	SMB2Echo	0 0%
	SMB2Negprot	41 0%
	SMB2TreeConnnect	62 0%
	SMB2TreeDisconnect	33 0%
	SMB2Login	42 0%
	SMB2Create	2163 6%
	SMB2Read	17042 44%
	SMB2Write	12915 34%
	SMB2Lock	0 0%
	SMB2Unlock	0 0%
	SMB2OplkBrkAck	0 0%
	SMB2ChgNfy	86 0%
	SMB2CLose	1914 5%
	SMB2Flush	3 0%
	SMB2Logout	11 0%
	SMB2Cancel	59 0%
	SMB2IPCCreate	7 0%
	SMB2IPCRead	7 0%
	SMB2IPCWrite	7 0%
	SMB2QueryDir	1500 4%
	SMB2QueryFileBasicInfo	0 0%
	SMB2QueryFileStndInfo	0 0%
	SMB2QueryFileIntInfo	45 0%
	SMB2QueryFileEAInfo	55 0%
	SMB2QueryFileFEAInfo	0 0%
	SMB2QueryFileModeInfo	0 0%
	SMB2QueryAltNameInfo	0 0%
	SMB2QueryFileStreamInfo	21 0%
	SMB2QueryNetOpenInfo	209 1%
	SMB2QueryAttrTagInfo	0 0%
	SMB2QueryAccessInfo	0 0%
	SMB2QueryFileUnsupported	0 0%
	SMB2QueryFileInvalid	0 0%
	SMB2QueryFSVolInfo	70 0%
	SMB2QueryFSSizeInfo	10 0%
	SMB2QueryFSDevInfo	0 0%
	SMB2QueryFSAttrInfo	70 0%
	SMB2QueryFSFullSzInfo	38 0%
	SMB2QueryFSObjIdInfo	0 0%
	SMB2QueryFSInvalid	0 0%
	SMB2QuerySecurityInfo	106 0%
	SMB2SetBasicInfo	30 0%
	SMB2SetRenameInfo	25 0%
	SMB2SetFileLinkInfo	0 0%
	SMB2SetFileDispInfo	14 0%
	SMB2SetFullEAInfo	0 0%
	SMB2SetModeInfo	0 0%
	SMB2SetAllocInfo	13 0%
	SMB2SetEOFInfo	18 0%
	SMB2SetUnsupported	0 0%
	SMB2SetInfoInvalid	0 0%
	SMB2SetSecurityInfo	22 0%
	SMB2FsctlPipeTransceive	1331 3%
	SMB2FsctlPipePeek	0 0%
	SMB2FsctlEnumSnapshots	0 0%
	SMB2FsctlDfsReferrals	4 0%
	SMB2FsctlSetSparse	0 0%
	SMB2FsctlSecureShare	0 0%
	SMB2FsctlFileUnsupported	127 0%
	SMB2FsctlIpcUnsupported	0 0%
	cancel lock	0
	wait lock	0
	copy to align	0
	alignedSmall	2250
	alignedLarge	2272
	alignedSmallRel	0
	alignedLargeRel	0
	FidHashAllocs	0
	TidHashAllocs	0
	UidHashAllocs	0
	mbufWait	0
	nbtWait	0
	pBlkWait	0
	BackToBackCPWait	0
	cwaWait	0
	short msg prevent	2
	multipleVCs	0
	SMB signing	41
	mapped null user	0
	PDCupcalls	0
	nosupport	0
	read pipe busy	0
	write pipe busy	0
	trans pipe busy	0
	read pipe broken	0
	write pipe broken	0
	trans pipe broken	0
	queued writeraw	0
	nbt disconnect	16
	smb disconnect	17
	dup disconnect	0
	OpLkBkXorBatchToL2	0
	OpLkBkXorBatchToNone	0
	OpLkBkL2ToNone	0
	OpLkBkNoBreakAck	0
	OpLkBkNoBreakAck95	0
	OpLkBkNoBreakAckNT	0
	OpLkBkIgnoredAck	0
	OpLkBkWaiterTimedOut	0
	OpLkBkDelayedBreak	0
	SharingErrorRetries	0
	FoldAttempts	0
	FoldRenames	0
	FoldRenameFailures	0
	FoldOverflows	0
	FoldDuplicates	0
	FoldWAFLTooBusy	0
	NoAllocCredStat	0
	RetryRPCcollision	0
	TconCloseTID	0
	GetNTAPExtAttrs	0
	SetNTAPExtAttrs	0
	SearchBusy	0
	ChgNfyNoMemory	0
	ChgNfyNewWatch	67
	ChgNfyLastWatch	57
	UsedMIDTblCreated	2
	UnusedMIDTblCreated	0
	InvalidMIDRejects	0
	SMB2InvalidSignature	0
	SMB2DurableCreateReceived	2126
	SMB2DurableCreateSucceeded	227
	SMB2DurableReclaimReceived	0
	SMB2DurableReclaimSucceeded	0
	SMB2DurableHandlePreserved	0
	SMB2DurableHandlePurged	0
	SMB2DurableHandleExpired	0
	SMB2FileDirInfo	0
	SMB2FileFullDirInfo	0
	SMB2FileIdFullDirInfo	0
	SMB2FileBothDirInfo	0
	SMB2FileIdBothDirInfo	750
	SMB2FileNamesInfo	0
	SMB2FileDirUnsupported	0
	SMB2QueryInfo	624
	SMB2SetInfo	122
	SMB2Ioctl	1462
	SMB2RelatedCompRequest	1018
	SMB2UnRelatedCompRequest	0
	SMB2FileRequest	36520
	SMB2PipeRequest	1390
	SMB2_1_LeaseBreaks	0
	SMB2_1_LeaseUpgrades	0
	SMB2_1_LeaseBreakExcuses	135
	SMB2_1_LeaseBreakAckTimeouts	0
	SMB2_1_HandleLeaseBreaks	0
	SMB2_1_LeaseBreaksToNone	0
	SMB2_1_LeaseBreakAcksIgnored	0
	SMB2_1_StatLeaseWorkObjAllocFail	0
	SMB2nosupport	25

Max Multiplex = 50, Max pBlk Exhaust = 0, Max pBlk Reserve Exhaust = 0

Max FIDs = 22, Max FIDs on one tree = 13

Max Searches on one tree = 0, Max Core Searches on one tree = 0

Max sessions = 17

Max trees = 16

Max shares = 6

Max session UIDs = 1, Max session TIDs = 2

Max locks = 28

Max credentials = 16

Max group SIDs per credential = 71

Max pBlks = 1024 Current pBlks = 1024 Num Logons = 0

Max reserved pBlks = 32 Current reserved pBlks = 32

Max gAuthQueue depth

= 4

Max gSMBBlockingQueue depth = 1

Max gSMBTimerQueue depth	= 5
Max gSMBAlfQueue depth	= 1

Max gSMBRPCWorkerQueue depth = 1

Max gOffloadQueue depth

= 2

Max gSMB2SigningQueue depth = 0

Local groups: builtins = 6, user-defined = 1, SIDs = 4

RPC group count = 10, RPC group active count = 1

Max Watched Directories = 11, Current Watched Directories = 10

Max Pending ChangeNotify Requests = 10, Current Pending ChangeNotify Requests = 9

Max Pending DeleteOnClose Requests = 3072, Current Pending DeleteOnClose Requests = 0

and cifs stat for second controller (bad)

	reject	0 0%
	mkdir	0 0%
	rmdir	0 0%
	open	0 0%
	create	0 0%
	close	0 0%
	X&close	0 0%
	flush	0 0%
	X&flush	0 0%
	delete	0 0%
	rename	0 0%
	NTRename	0 0%
	getatr	0 0%
	setatr	0 0%
	read	0 0%
	X&read	0 0%
	write	0 0%
	X&write	0 0%
	lock	0 0%
	unlock	0 0%
	mknew	0 0%
	chkpth	0 0%
	exit	0 0%
	lseek	0 0%
	lockread	0 0%
	X&lockread	0 0%
	writeunlock	0 0%
	readbraw	0 0%
	writebraw	0 0%
	writec	0 0%
	gettattre	0 0%
	settattre	0 0%
	lockingX	0 0%
	IPC	192 1%
	open2	0 0%
	find_first2	0 0%
	find_next2	0 0%
	query_fs_info	0 0%
	query_path_info	0 0%
	set_path_info	0 0%
	query_file_info	0 0%
	set_file_info	0 0%
	create_dir2	0 0%
	Dfs_referral	0 0%
	Dfs_report	0 0%
	echo	0 0%
	writeclose	0 0%
	openX	0 0%
	readX	0 0%
	writeX	0 0%
	findclose	0 0%
	tcon	0 0%
	tdis	33 0%
	negprot	30 0%
	login	4 0%
	logout	0 0%
	tconX	2 0%
	dskattr	0 0%
	search	0 0%
	fclose	0 0%
	NTCreateX	16 0%
	NTTransCreate	0 0%
	NTTransIoctl	0 0%
	NTTransNotify	0 0%
	NTTransSetSec	0 0%
	NTTransQuerySec	0 0%
	NTNamedPipeMulti	0 0%
	NTCancel CN	0 0%
	NTCancel Other	0 0%
	SMB2Echo	0 0%
	SMB2Negprot	36 0%
	SMB2TreeConnnect	61 0%
	SMB2TreeDisconnect	33 0%
	SMB2Login	45 0%
	SMB2Create	2310 8%
	SMB2Read	18405 64%
	SMB2Write	339 1%
	SMB2Lock	1 0%
	SMB2Unlock	0 0%
	SMB2OplkBrkAck	0 0%
	SMB2ChgNfy	61 0%
	SMB2CLose	2142 7%
	SMB2Flush	2 0%
	SMB2Logout	4 0%
	SMB2Cancel	385 1%
	SMB2IPCCreate	23 0%
	SMB2IPCRead	23 0%
	SMB2IPCWrite	23 0%
	SMB2QueryDir	2778 10%
	SMB2QueryFileBasicInfo	0 0%
	SMB2QueryFileStndInfo	0 0%
	SMB2QueryFileIntInfo	8 0%
	SMB2QueryFileEAInfo	18 0%
	SMB2QueryFileFEAInfo	0 0%
	SMB2QueryFileModeInfo	0 0%
	SMB2QueryAltNameInfo	0 0%
	SMB2QueryFileStreamInfo	18 0%
	SMB2QueryNetOpenInfo	111 0%
	SMB2QueryAttrTagInfo	0 0%
	SMB2QueryAccessInfo	0 0%
	SMB2QueryFileUnsupported	0 0%
	SMB2QueryFileInvalid	0 0%
	SMB2QueryFSVolInfo	133 0%
	SMB2QueryFSSizeInfo	8 0%
	SMB2QueryFSDevInfo	0 0%
	SMB2QueryFSAttrInfo	133 0%
	SMB2QueryFSFullSzInfo	11 0%
	SMB2QueryFSObjIdInfo	0 0%
	SMB2QueryFSInvalid	0 0%
	SMB2QuerySecurityInfo	77 0%
	SMB2SetBasicInfo	24 0%
	SMB2SetRenameInfo	17 0%
	SMB2SetFileLinkInfo	0 0%
	SMB2SetFileDispInfo	12 0%
	SMB2SetFullEAInfo	0 0%
	SMB2SetModeInfo	0 0%
	SMB2SetAllocInfo	3 0%
	SMB2SetEOFInfo	12 0%
	SMB2SetUnsupported	0 0%
	SMB2SetInfoInvalid	0 0%
	SMB2SetSecurityInfo	16 0%
	SMB2FsctlPipeTransceive	1287 4%
	SMB2FsctlPipePeek	0 0%
	SMB2FsctlEnumSnapshots	0 0%
	SMB2FsctlDfsReferrals	10 0%
	SMB2FsctlSetSparse	0 0%
	SMB2FsctlSecureShare	0 0%
	SMB2FsctlFileUnsupported	36 0%
	SMB2FsctlIpcUnsupported	21 0%
	cancel lock	0
	wait lock	0
	copy to align	0
	alignedSmall	2847
	alignedLarge	2389
	alignedSmallRel	0
	alignedLargeRel	0
	FidHashAllocs	0
	TidHashAllocs	0
	UidHashAllocs	0
	mbufWait	0
	nbtWait	0
	pBlkWait	0
	BackToBackCPWait	0
	cwaWait	0
	short msg prevent	5
	multipleVCs	0
	SMB signing	32
	mapped null user	0
	PDCupcalls	0
	nosupport	0
	read pipe busy	0
	write pipe busy	0
	trans pipe busy	0
	read pipe broken	0
	write pipe broken	0
	trans pipe broken	0
	queued writeraw	0
	nbt disconnect	13
	smb disconnect	21
	dup disconnect	0
	OpLkBkXorBatchToL2	0
	OpLkBkXorBatchToNone	0
	OpLkBkL2ToNone	0
	OpLkBkNoBreakAck	0
	OpLkBkNoBreakAck95	0
	OpLkBkNoBreakAckNT	0
	OpLkBkIgnoredAck	0
	OpLkBkWaiterTimedOut	0
	OpLkBkDelayedBreak	0
	SharingErrorRetries	0
	FoldAttempts	0
	FoldRenames	0
	FoldRenameFailures	0
	FoldOverflows	0
	FoldDuplicates	0
	FoldWAFLTooBusy	0
	NoAllocCredStat	0
	RetryRPCcollision	0
	TconCloseTID	0
	GetNTAPExtAttrs	0
	SetNTAPExtAttrs	0
	SearchBusy	0
	ChgNfyNoMemory	0
	ChgNfyNewWatch	50
	ChgNfyLastWatch	47
	UsedMIDTblCreated	6
	UnusedMIDTblCreated	0
	InvalidMIDRejects	0
	SMB2InvalidSignature	0
	SMB2DurableCreateReceived	2286
	SMB2DurableCreateSucceeded	135
	SMB2DurableReclaimReceived	0
	SMB2DurableReclaimSucceeded	0
	SMB2DurableHandlePreserved	0
	SMB2DurableHandlePurged	0
	SMB2DurableHandleExpired	0
	SMB2FileDirInfo	0
	SMB2FileFullDirInfo	12
	SMB2FileIdFullDirInfo	0
	SMB2FileBothDirInfo	5
	SMB2FileIdBothDirInfo	1414
	SMB2FileNamesInfo	0
	SMB2FileDirUnsupported	0
	SMB2QueryInfo	517
	SMB2SetInfo	84
	SMB2Ioctl	1354
	SMB2RelatedCompRequest	1579
	SMB2UnRelatedCompRequest	0
	SMB2FileRequest	27016
	SMB2PipeRequest	1548
	SMB2_1_LeaseBreaks	1
	SMB2_1_LeaseUpgrades	0
	SMB2_1_LeaseBreakExcuses	92
	SMB2_1_LeaseBreakAckTimeouts	0
	SMB2_1_HandleLeaseBreaks	0
	SMB2_1_LeaseBreaksToNone	1
	SMB2_1_LeaseBreakAcksIgnored	0
	SMB2_1_StatLeaseWorkObjAllocFail	0
	SMB2nosupport	128

Max Multiplex = 50, Max pBlk Exhaust = 0, Max pBlk Reserve Exhaust = 0

Max FIDs = 18, Max FIDs on one tree = 14

Max Searches on one tree = 0, Max Core Searches on one tree = 0

Max sessions = 16

Max trees = 18

Max shares = 7

Max session UIDs = 1, Max session TIDs = 4

Max locks = 25

Max credentials = 15

Max group SIDs per credential = 71

Max pBlks = 1024 Current pBlks = 1024 Num Logons = 0

Max reserved pBlks = 32 Current reserved pBlks = 32

Max gAuthQueue depth

= 2

Max gSMBBlockingQueue depth = 1

Max gSMBTimerQueue depth	= 6
Max gSMBAlfQueue depth	= 1

Max gSMBRPCWorkerQueue depth = 1

Max gOffloadQueue depth

= 2

Max gSMB2SigningQueue depth = 0

Local groups: builtins = 6, user-defined = 1, SIDs = 4

RPC group count = 10, RPC group active count = 1

Max Watched Directories = 6, Current Watched Directories = 3

Max Pending ChangeNotify Requests = 4, Current Pending ChangeNotify Requests = 3

Max Pending DeleteOnClose Requests = 3072, Current Pending DeleteOnClose Requests = 0

On the second controller I'm specifically doing the test on a newly created volume without dedup. Tried already to disable vscan but changes nothing, besides the first controller performs great with dedup and vscan enabled, as I said all settings the same.

Doing a quick packet sniff on the client I see that after a couple of seconds in a file transfer sessions there are only read requests from the client and no responses from the filer and then it responds again and then the same all over again.

Any help appreciated.

DOMINIC_WYSS · ‎2014-09-03

how much disks do the aggregates have? is one bigger than the other?

sysconfig -r

g_billios · ‎2014-09-03

Actually they are exactly the same:

First

Aggregate aggr0 (online, raid4) (block checksums)

Plex /aggr0/plex0 (online, normal, active)

RAID group /aggr0/plex0/rg0 (normal, block checksums)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

parity 0a.00.2 0a 0 2 SA:A 0 FSAS 7200 3807816/7798408704 3815447/7814037168

data 0a.00.4 0a 0 4 SA:A 0 FSAS 7200 3807816/7798408704 3815447/7814037168

data 0a.00.6 0a 0 6 SA:A 0 FSAS 7200 3807816/7798408704 3815447/7814037168

data 0a.00.8 0a 0 8 SA:A 0 FSAS 7200 3807816/7798408704 3815447/7814037168

data 0a.00.10 0a 0 10 SA:A 0 FSAS 7200 3807816/7798408704 3815447/7814037168

Pool1 spare disks (empty)

Pool0 spare disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

Spare disks for block checksum

spare 0a.00.0 0a 0 0 SA:A 0 FSAS 7200 3807816/7798408704 3815447/7814037168 (not zeroed)

Partner disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

partner 0b.00.11 0b 0 11 SA:B 0 FSAS 7200 0/0 3815447/7814037168

partner 0b.00.5 0b 0 5 SA:B 0 FSAS 7200 0/0 3815447/7814037168

partner 0b.00.9 0b 0 9 SA:B 0 FSAS 7200 0/0 3815447/7814037168

partner 0b.00.7 0b 0 7 SA:B 0 FSAS 7200 0/0 3815447/7814037168

partner 0b.00.3 0b 0 3 SA:B 0 FSAS 7200 0/0 3815447/7814037168

partner 0b.00.1 0b 0 1 SA:B 0 FSAS 7200 0/0 3815447/7814037168

Second

Aggregate aggr0 (online, raid4) (block checksums)

Plex /aggr0/plex0 (online, normal, active)

RAID group /aggr0/plex0/rg0 (normal, block checksums)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

parity 0a.00.3 0a 0 3 SA:B 0 FSAS 7200 3807816/7798408704 3815447/7814037168

data 0a.00.5 0a 0 5 SA:B 0 FSAS 7200 3807816/7798408704 3815447/7814037168

data 0a.00.7 0a 0 7 SA:B 0 FSAS 7200 3807816/7798408704 3815447/7814037168

data 0a.00.9 0a 0 9 SA:B 0 FSAS 7200 3807816/7798408704 3815447/7814037168

data 0a.00.11 0a 0 11 SA:B 0 FSAS 7200 3807816/7798408704 3815447/7814037168

Pool1 spare disks (empty)

Pool0 spare disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

Spare disks for block checksum

spare 0a.00.1 0a 0 1 SA:B 0 FSAS 7200 3807816/7798408704 3815447/7814037168 (not zeroed)

Partner disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

--------- ------ ------------- ---- ---- ---- ----- -------------- --------------

partner 0b.00.10 0b 0 10 SA:A 0 FSAS 7200 0/0 3815447/7814037168

partner 0b.00.6 0b 0 6 SA:A 0 FSAS 7200 0/0 3815447/7814037168

partner 0b.00.0 0b 0 0 SA:A 0 FSAS 7200 0/0 3815447/7814037168

partner 0b.00.8 0b 0 8 SA:A 0 FSAS 7200 0/0 3815447/7814037168

partner 0b.00.2 0b 0 2 SA:A 0 FSAS 7200 0/0 3815447/7814037168

partner 0b.00.4 0b 0 4 SA:A 0 FSAS 7200 0/0 3815447/7814037168

DOMINIC_WYSS · ‎2014-09-03

then I don't have an idea why this happens

I recommend to fetch perfstats, open a case at Netapp and let them analyse the stats.

NetApp_SEAL · ‎2014-09-03

Mundane question - are the options for TCP Window Size set to the same values on both controllers?

filer>options cifs.tcp_window_size

Believe the recommendation is to ensure it's set to at least 64,240 (and I think as of ONTAP 8.2.1 the default is set to that).

It can go as high as 8,388,608, however.

Just something to check.

g_billios · ‎2014-09-03

I have already tried to changed the tcp window size but nothing changed. The issue occurs regardless of this setting.

Probably will raise a support ticket.