hello netapp community.. I am reaching out because I wanted to get some input on a CIFS performance issue we are experiencing from the perspective of the end user.. Like taking updwards of 10 minutes to open a folder in some cases. The end user environment is predominently Mac with just a handful of windows 7/10.
The volume in question is 13T total of which 11T are used with 114,646,233 inodes used (about 30% use).
I mounted this up to my test machine and there is a spiderweb of directories and subdirectories upon subdirectories which have even more subdirectories and filer... some of these take minutes to ls -la and hundreds of directories/files are listed. It's a bit of mess. I was thinking about performing a rellocate on the volume but it would probably take days just to finish the first step (measure) and it seems to impact the filer pretty heavily while scanning. I was also thinking abouit increasing the tcp_window_size possibly but a restart of CIFS is needed which is pretty disruptive so I would need to have a window for that. Current setting is:
ourfiler*> options cifs.tcp_window_size
cifs.tcp_window_size 64240
- What are the limitations of
cifs.tcp_window_size?
Default value = 17520 bytes
Minimum value = 1600 bytes
Maximum value = 8388608 bytes(8M)
We have a dual 10G LACP connection for data. The filer is an FAS8020. NetApp Release 8.2.5 7-Mode: Wed Jul 19 03:55:53 PDT 2017
Some information about our CIFS environment
nss2-ord> stats show cifs
cifs:cifs:instance_name:cifs
cifs:cifs:node_name:
cifs:cifs:node_uuid:
cifs:cifs:cifs_ops:2427/s
cifs:cifs:cifs_latency:0.09ms
cifs:cifs:cifs_read_ops:20/s
cifs:cifs:cifs_write_ops:0/s
cifs:cifs:cifs_dlc_ops:0
cifs:cifs:cifs_dlc_ops_queued:0
We have audit logging off /etc is not filling up
nss2-ord> stats show cifs
cifs:cifs:instance_name:cifs
cifs:cifs:node_name:
cifs:cifs:node_uuid:
cifs:cifs:cifs_ops:2427/s
cifs:cifs:cifs_latency:0.09ms
cifs:cifs:cifs_read_ops:20/s
cifs:cifs:cifs_write_ops:0/s
cifs:cifs:cifs_dlc_ops:0
cifs:cifs:cifs_dlc_ops_queued:0
nss2-ord> options cifs
cifs.AD.retry_delay 0
cifs.LMCompatibilityLevel 1
cifs.W2K_password_change off
cifs.W2K_password_change_interval 4w
cifs.W2K_password_change_within 1h
cifs.audit.account_mgmt_events.enable off
cifs.audit.autosave.file.extension
cifs.audit.autosave.file.extension.nanosecond_precision off
cifs.audit.autosave.file.limit 0
cifs.audit.autosave.onsize.enable off
cifs.audit.autosave.onsize.threshold 75%
cifs.audit.autosave.ontime.enable off
cifs.audit.autosave.ontime.interval 1d
cifs.audit.enable off
cifs.audit.file_access_events.enable on
cifs.audit.liveview.allowed_users
cifs.audit.liveview.enable off
cifs.audit.logon_events.enable on
cifs.audit.logsize 1048576
cifs.audit.nfs.enable off
cifs.audit.nfs.filter.filename
cifs.audit.saveas /etc/log/adtlog.evt
cifs.bypass_traverse_checking on
cifs.client.dup-detection ip-address
cifs.comment
cifs.enable_share_browsing on
cifs.gpo.enable off
cifs.gpo.trace.enable off
cifs.grant_implicit_exe_perms off
cifs.guest_account
cifs.home_dir.generic_share_access_level 3
cifs.home_dir.generic_share_access_warn on
cifs.home_dir_namestyle
cifs.home_dirs_public_for_admin on
cifs.idle_timeout 900
cifs.ipv6.enable off
cifs.max_mpx 253
cifs.ms_snapshot_mode xp
cifs.netbios_aliases
cifs.netbios_over_tcp.enable on
cifs.nfs_root_ignore_acl off
cifs.oplocks.enable on
cifs.oplocks.opendelta 0
cifs.per_client_stats.enable off
cifs.perfmon.allowed_users
cifs.perm_check_ro_del_ok off
cifs.perm_check_use_gid on
cifs.preserve_unix_security off
cifs.restrict_anonymous 0
cifs.restrict_anonymous.enable off
cifs.save_case on
cifs.scopeid
cifs.search_domains
cifs.show_dotfiles on
cifs.show_snapshot off
cifs.shutdown_msg_level 2
cifs.sidcache.enable on
cifs.sidcache.lifetime 1440
cifs.signing.enable off
cifs.smb2.enable on
cifs.smb2.signing.max_threads 5
cifs.smb2.signing.multiprocessing default
cifs.smb2.signing.required off
cifs.smb2_1.branch_cache.enable off
cifs.smb2_1.branch_cache.hash_time_out 3600 (value might be overwritten in takeover)
cifs.smbx_signing_required off
cifs.snapshot_file_folding.enable off
cifs.symlinks.cycleguard on
cifs.symlinks.enable on
cifs.trace_dc_connection off
cifs.trace_login off
cifs.universal_nested_groups.enable on
cifs.widelink.ttl 10m
nss2-ord> sysstat -x 1
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
71% 343 8928 0 9271 2106 59835 1652 0 0 0 19 98% 0% - 10% 0 0 0 0 0 0 0
72% 255 7589 0 7844 1866 61397 1420 24 0 0 19 98% 0% - 9% 0 0 0 0 0 0 0
65% 234 6501 0 6735 1577 46512 13396 13704 0 0 19 98% 58% 3 19% 0 0 0 0 0 0 0
67% 385 7070 0 7455 1726 48205 1652 0 0 0 19 97% 0% - 15% 0 0 0 0 0 0 0
64% 259 7270 0 7535 1764 53944 1648 32 0 0 19 97% 0% - 9% 6 0 0 0 0 0 0
57% 216 8213 0 8429 1886 48937 1816 0 0 0 19 97% 0% - 8% 0 0 0 0 0 0 0
The highest spike in Disk Out I have seen is 51%... cache hits are nice and high which is great. the filer is very busy in general though.
I am thinking the spiderweb of directories which goes 15 to 20 levels deep is the cause of this but I wanted to get some ideas of what more I can test on the filer end and possible tuning implementations.
Thanks for looking..
