Slow writing FAS 3250 versión 8.1.4P1-Mode

[JuanCarlosrl]

Hi,

I have a dual controller netapp, Fas 3250 with version 8.1.4P1-Mode. I have a trunk of 2 RJ45 ports of 1Gb (total 2GB) and also created on the switch. I have a CIFS service up and I have enabled it with default values.

The cabin has an aggregate of 3 disks for the OS, and two aggregates of 66 disks each. The disks are of the 1.2Tb SAS type. The problem that arises is that in a CIFS type folder, the readings are giving me a speed of 85-95 MB per second which is a speed that is fine, because from the server where I am doing the test it has a 1Gb cable. The problem arises with writing, it is around 20-25Mb per second. It's very slow, and I don't know why it is ...

Can someone help me find out why with reading the transfer is good and with reading it is very slow?

Tests I have done:

The cabin has a 2TB flash cache PCI card, in the graphical interface it shows that it is operating, I have disabled it but the rendering is the same.

I have tried to make 2 writes at the same time, and the speed of 20-25Mb per second, is divided into 2 the 2 copies, giving 12Mb per second in each writing.

I have tried to enable a second CIFS service on the "passive" controller that only has 3 disks associated with the OS. I have enabled a 1GB share, I have written tests and I get the same speeds.

Thanks, 

Regards,

[paul_stejskal]

Without any perf data from the controller, we are guessing.

If you run sysstat -x 1, do you see any CPU/disk 100% utilization issues, or back-to-back consistency points?

Another thing to check is if vscan and fpolicy are enabled.

[JuanCarlosrl]

Hi,

I execute the instruction that you indicate to me and I get this on the screen:

sysstat -x 1
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
70% 0 8200 0 8200 29573 1835 35460 108980 0 0 10s 100% 83% Tf 13% 0 0 0 0 0 0 0
47% 0 4442 0 4442 10850 967 348 128016 0 0 10s 100% 100% :f 9% 0 0 0 0 0 0 0
54% 0 6101 0 6106 18120 1357 7184 45200 0 0 10s 100% 60% : 5% 5 0 0 0 0 0 0
49% 0 5243 0 5243 13663 1093 32 24 0 0 10s 100% 0% - 2% 0 0 0 0 0 0 0
52% 0 6353 0 6353 19844 1374 0 0 0 0 10s 100% 0% - 0% 0 0 0 0 0 0 0
50% 0 6250 0 6250 20122 1395 0 0 0 0 10s 100% 0% - 0% 0 0 0 0 0 0 0
49% 0 5609 0 5609 15771 1199 24 32 0 0 10s 100% 0% - 2% 0 0 0 0 0 0 0
58% 0 9090 0 9095 33811 2030 0 0 0 0 10s 100% 0% - 0% 5 0 0 0 0 0 0
58% 0 9062 0 9062 32123 2084 8 0 0 0 10s 100% 0% - 0% 0 0 0 0 0 0 0
60% 0 9622 0 9622 34206 3365 24 24 0 0 10s 100% 0% - 2% 0 0 0 0 0 0 0
45% 0 6423 0 6423 20542 1451 0 0 0 0 10s 100% 0% - 0% 0 0 0 0 0 0 0
62% 0 9535 0 9535 34523 2668 0 8 0 0 10s 100% 0% - 1% 0 0 0 0 0 0 0
68% 0 8565 0 8569 28955 5169 32436 58724 0 0 10s 99% 40% Tf 10% 4 0 0 0 0 0 0
58% 0 6574 0 6574 18957 7498 3724 126900 0 0 10s 99% 100% :f 10% 0 0 0 0 0 0 0
60% 0 7405 0 7405 24012 3950 584 105532 0 0 10s 100% 100% :f 7% 0 0 0 0 0 0 0
49% 0 5356 0 5356 17239 1679 1612 15236 0 0 10s 100% 28% : 5% 0 0 0 0 0 0 0
31% 0 3821 0 3821 16216 937 0 8 0 0 10s 100% 0% - 1% 0 0 0 0 0 0 0
43% 0 6243 0 6247 26991 1707 0 0 0 0 10s 100% 0% - 0% 4 0 0 0 0 0 0
45% 0 6178 0 6180 31936 2485 36 24 0 0 10s 100% 0% - 2% 2 0 0 0 0 0 0
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
48% 0 3154 0 3154 25290 905 4 0 0 0 10s 97% 0% - 1% 0 0 0 0 0 0 0
38% 0 4459 0 4459 18936 1540 0 0 0 0 10s 100% 0% - 0% 0 0 0 0 0 0 0
43% 0 5347 0 5347 47511 2853 588 32 0 0 10s 100% 0% - 2% 0 0 0 0 0 0 0
51% 0 5618 0 5624 74137 6767 2180 0 0 0 10s 99% 0% - 2% 6 0 0 0 0 0 0
51% 0 5198 0 5198 68085 4466 792 0 0 0 56s 99% 0% - 3% 0 0 0 0 0 0 0
51% 0 3567 0 3567 64536 2652 49336 117180 0 0 14 100% 87% Mf 11% 0 0 0 0 0 0 0
45% 0 2912 0 2912 74927 1778 3584 122216 0 0 14 100% 100% :f 9% 0 0 0 0 0 0 0
42% 0 3436 0 3436 55769 1499 164 125548 0 0 14 100% 100% :f 8% 0 0 0 0 0 0 0
33% 0 3258 0 3262 13298 680 96 38176 0 0 14 100% 64% : 6% 4 0 0 0 0 0 0
58% 0 5870 0 5870 35629 2789 8 0 0 0 14 98% 0% - 0% 0 0 0 0 0 0 0
43% 0 6026 0 6026 25360 1645 0 0 0 0 14 100% 0% - 0% 0 0 0 0 0 0 0
46% 0 6229 0 6229 50585 2093 24 24 0 0 14 100% 0% - 2% 0 0 0 0 0 0 0
45% 0 5464 0 5464 64787 1973 10192 24 0 0 14 100% 7% Mn 3% 0 0 0 0 0 0 0
44% 0 2844 0 2851 61772 1529 42712 212568 0 0 14 100% 100% :f 13% 7 0 0 0 0 0 0
46% 0 2620 0 2620 78178 1800 800 135116 0 0 14 100% 100% :f 12% 0 0 0 0 0 0 0
45% 0 2452 0 2452 77506 1766 72 41820 0 0 14 100% 57% : 6% 0 0 0 0 0 0 0
44% 0 4269 0 4269 45463 1482 4 0 0 0 14 100% 0% - 1% 0 0 0 0 0 0 0
49% 0 6102 0 6102 28249 2735 28 32 0 0 14 100% 0% - 2% 0 0 0 0 0 0 0
66% 0 5496 0 5500 32200 1449 0 0 0 0 14 98% 0% - 0% 4 0 0 0 0 0 0
41% 0 4925 0 4925 25467 1727 16 0 0 0 14 100% 0% - 1% 0 0 0 0 0 0 0
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
48% 0 2657 0 2657 74084 1825 53060 185372 0 0 14 99% 82% Mf 10% 0 0 0 0 0 0 0
50% 0 4867 0 4867 74390 4904 1696 131756 0 0 14 99% 100% :f 11% 0 0 0 0 0 0 0
48% 0 4320 0 4324 70467 6882 3048 69132 0 0 14 99% 97% : 5% 4 0 0 0 0 0 0
42% 0 4790 0 4794 47114 5575 412 24 0 0 14 100% 0% - 2% 4 0 0 0 0 0 0
38% 0 3753 0 3753 46461 1744 48 0 0 0 14 100% 0% - 1% 0 0 0 0 0 0 0
37% 0 3726 0 3726 44478 1617 0 0 0 0 14 100% 0% - 0% 0 0 0 0 0 0 0
29% 0 2722 0 2808 14920 3080 556 24 0 0 14 100% 0% - 2% 86 0 0 0 0 0 0
28% 0 2599 0 2599 10471 7985 10184 8 0 0 13 98% 0% - 2% 0 0 0 0 0 0 0
35% 0 2303 0 2307 7034 74270 59480 0 0 0 13 96% 0% - 4% 4 0 0 0 0 0 0
43% 0 4666 0 4666 19979 24353 21508 17796 0 0 13 99% 13% Mn 3% 0 0 0 0 0 0 0
42% 0 4332 0 4332 18731 993 38296 156820 0 0 13 99% 100% :f 11% 0 0 0 0 0 0 0
11% 0 694 0 694 2794 142 256 126576 0 0 13 100% 100% :f 9% 0 0 0 0 0 0 0
43% 0 5785 0 5785 24950 1824 464 105396 0 0 13 100% 100% :f 7% 0 0 0 0 0 0 0
40% 0 5546 0 5551 22451 4445 72 4144 0 0 13 100% 19% : 4% 5 0 0 0 0 0 0
35% 0 3914 0 4157 14959 3186 952 0 0 0 13 99% 0% - 1% 243 0 0 0 0 0 0
25% 0 2647 0 2647 11163 788 24 24 0 0 13 100% 0% - 2% 0 0 0 0 0 0 0
37% 0 4573 0 4573 18121 2030 0 0 0 0 13 100% 0% - 0% 0 0 0 0 0 0 0
17% 0 1964 0 1964 8079 701 4 8 0 0 13 100% 0% - 1% 0 0 0 0 0 0 0
26% 0 2351 0 2355 9938 725 24 24 0 0 13 100% 0% - 2% 4 0 0 0 0 0 0
29% 0 3354 0 3354 14490 750 12 0 0 0 13 100% 0% - 1% 0 0 0 0 0 0 0
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
36% 0 4425 0 4425 18267 1034 8 0 0 0 13 100% 0% - 1% 0 0 0 0 0 0 0
36% 0 4951 0 4951 20998 1231 24 24 0 0 13 100% 0% - 2% 0 0 0 0 0 0 0
46% 0 6805 0 6805 28142 1570 4 8 0 0 13 100% 0% - 1% 0 0 0 0 0 0 0
42% 0 4084 0 4089 15930 2167 37392 85088 0 0 13 100% 81% Tf 8% 5 0 0 0 0 0 0
41% 0 4380 0 4380 18726 1070 408 110904 0 0 13 100% 100% :f 9% 0 0 0 0 0 0 0
39% 0 2497 0 2497 10634 1080 5408 33544 0 0 13 100% 57% : 7% 0 0 0 0 0 0 0
18% 0 1051 0 1051 4442 240 0 0 0 0 13 100% 0% - 0% 0 0 0 0 0 0 0
2% 0 175 0 175 774 39 24 32 0 0 13 100% 0% - 2% 0 0 0 0 0 0 0
24% 0 2286 0 2291 8707 4196 0 0 0 0 13 99% 0% - 0% 5 0 0 0 0 0 0
29% 0 3143 0 3143 11621 3249 952 0 0 0 13 98% 0% - 2% 0 0 0 0 0 0 0
6% 0 693 0 693 1509 1167 24 24 0 0 13 100% 0% - 2% 0 0 0 0 0 0 0
24% 0 2401 0 2401 9827 473 0 0 0 0 13 100% 0% - 0% 0 0 0 0 0 0 0
26% 0 2503 0 2503 10440 1005 0 8 0 0 13 100% 0% - 1% 0 0 0 0 0 0 0

it seems that the cpu is without any problem ...

CLCALC03-01> vscan

Virus scanning is disabled.

No vscan servers are connected.

List of extensions to scan:
Extensions-to-scan list is empty.

List of extensions not to scan:
Extensions-not-to-scan list is empty.

Number of files scanned: 0
Number of scan failures: 0
Number of throttled requests: 0

CLCALC03-01> fpolicy

CIFS file policy is enabled.

Should I disable fpolicy? I've tried but I can't disable it, since I don't know what the policyName is called...

enclosed screenshots of cabin performance

Thank you very much, 

Regards, 

[paul_stejskal]

Was the sysstat when you were testing a write? Documentation on fpolicy: https://library.netapp.com/ecmdocs/ECMP1401220/html/GUID-07D368C8-F4DD-4F14-9E68-99C63881EBE7.html. Look there for the commands.

The only other thought I had is maybe TCP window size is still the default:

filer> options cifs

Also is SMB2 enabled?

[JuanCarlosrl]

Hi Paul, 

I have launched the sysstat with the cabin in production. It is currently in production, both in writings and readings and at that time and had also launched a writing of a single 250Mb file.

Options cifs output:

CLCALC03-01> options cifs
cifs.AD.retry_delay 0
cifs.LMCompatibilityLevel 1
cifs.W2K_password_change off
cifs.W2K_password_change_interval 4w
cifs.W2K_password_change_within 3600h
cifs.audit.account_mgmt_events.enable off
cifs.audit.autosave.file.extension
cifs.audit.autosave.file.limit 0
cifs.audit.autosave.onsize.enable off
cifs.audit.autosave.onsize.threshold 75%
cifs.audit.autosave.ontime.enable off
cifs.audit.autosave.ontime.interval 1d
cifs.audit.enable off
cifs.audit.file_access_events.enable on
cifs.audit.liveview.allowed_users
cifs.audit.liveview.enable off
cifs.audit.logon_events.enable on
cifs.audit.logsize 1048576
cifs.audit.nfs.enable off
cifs.audit.nfs.filter.filename
cifs.audit.saveas /etc/log/adtlog.evt
cifs.bypass_traverse_checking on
cifs.client.dup-detection ip-address
cifs.comment
cifs.enable_share_browsing on
cifs.gpo.enable off
cifs.gpo.trace.enable off
cifs.grant_implicit_exe_perms off
cifs.guest_account
cifs.home_dir_namestyle
cifs.home_dirs_public_for_admin on
cifs.idle_timeout 900
cifs.ipv6.enable off
cifs.max_mpx 253
cifs.ms_snapshot_mode xp
cifs.netbios_aliases
cifs.netbios_over_tcp.enable on
cifs.nfs_root_ignore_acl off
cifs.oplocks.enable on
cifs.oplocks.opendelta 0
cifs.per_client_stats.enable off
cifs.perfmon.allowed_users
cifs.perm_check_ro_del_ok off
cifs.perm_check_use_gid on
cifs.preserve_unix_security off
cifs.restrict_anonymous 0
cifs.restrict_anonymous.enable off
cifs.save_case on
cifs.scopeid
cifs.search_domains
cifs.show_dotfiles on
cifs.show_snapshot off
cifs.shutdown_msg_level 2
cifs.sidcache.enable on
cifs.sidcache.lifetime 1440
cifs.signing.enable off
cifs.smb2.enable on
cifs.smb2.signing.required off
cifs.smb2_1.branch_cache.enable off
cifs.smb2_1.branch_cache.hash_time_out 3600 (value might be overwritten in takeover)
cifs.snapshot_file_folding.enable off
cifs.symlinks.cycleguard on
cifs.symlinks.enable on
cifs.trace_dc_connection off
cifs.trace_login off
cifs.universal_nested_groups.enable on
cifs.widelink.ttl 10m

Also is SMB2 enabled?

according to the output of options cifs it seems that SMB2 is enabled if I'm not mistaken

Should fpolicy disable it and then do a writing test? Can disabling it affect production or can I do it without problem?

Thank you, 

[paul_stejskal]

Disabling it will only affect fpolicy logging. Just do it real quick as a test.

Also, if that doesn't work one command I want to try is:

filer> options cifs.tcp_window_size 65535

After setting this, you will need to disconnect and reconnect your test machine. To make sure it works, ideally you would reboot but offlining and onlining the ethernet connection on your Windows PC should do (since it's a test machine).

The default is 17520 I believe in this release, and being larger should help. To double check, issue the options cifs.tcp_window_size without a value before changing.

[JuanCarlosrl]

I have changed the tcp value

options cifs.tcp_window_size 65535

I have restarted the passive controller and now I get 55Mb per second of transfer. This has improved, but it still doesn't reach 90MB per second that the CIFS resource is giving me in reading.

I have seen in netapp that the maximum value of tcp is this cifs.tcp_window_size 8388608
I have introduced it and rebooted the controller and now with this value if it is giving me writes of 80-85 Mb per second, they already resemble the read speed

I have various doubts,
Is it harmful to set the maximum value?

Could you just restart the tcp cab service instead of restarting the controller completely? How do you do it if you can?

Finally, I could not disable fpolicy, I am not able to list which one is active to put it in the instruction

CLCALC03-02> fpolicy

CIFS file policy is enabled.

CLCALC03-02> fpolicy show

Usage:
fpolicy show <PolicyName>

CLCALC03-02> fpolicy _
The following commands are available; for more information
type "fpolicy help <command>"
create enable mon[itor] show
destroy ext[ension] options vol[ume]
disable help servers

fpolicy help - List of file policy commands.
fpolicy show - File policy status command.
fpolicy create - Create a new file policy.
fpolicy destroy - Destroy an existing file policy.
fpolicy enable - Enable specified file policy command.
fpolicy disable - Disable specified file policy command.
fpolicy ext[ension] - File policy file extension command.
fpolicy options - File policy configuration command.
fpolicy servers - File policy server command.
fpolicy vol[ume] - File policy volume filtering command.
fpolicy mon[itor] - File policy monitor command.

CLCALC03-02> fpolicy disable

Usage:
fpolicy disable <PolicyName>

CLCALC03-02>

I can't get the name of the policy to make it disabled. Sometimes it is no longer necessary and my problem is only the value of TCP

Thank you,

[paul_stejskal]

Ok don't worry about fpolicy. I wouldn't say 8M of a TCP window size, but maybe something in between 65k and 8M. You'll have to test it. Also the restart I mean was on the Windows client, not the filer.

[JuanCarlosrl]

After testing and setting the maximum allowed value and checking that the transfers have improved, I have been able to verify that the transfer is also not constant and I have lowered the tcp value.
I remain in tests to see how it evolves.

the value that I have put

options cifs.tcp_window_size 786420

I have been multiplying the value of 65536 by 2 and testing until it has already given me a transfer close to 90Mb per second.

thank you very much for your help.

[paul_stejskal]

Glad you got it working and glad to help.