topic SMB Tuning in Network and Storage Protocols

SMB Tuning

connoisseur — Wed, 13 Mar 2013 14:24:12 GMT

Hi!

Running a FAS3270 Active-Active configuration providing NAS service via vFilers to multiple customers.

At the moment only CIFS is used, no NFS customers in this environment.

Some of our customers experience slow performance, some don´t.

I know this has been discussed several times, but this seems a little different.

If we test regular file copy we get very good result.

10-11 MB/s (90-95 Mbps) which is what the customers pays for.

But here´s the tricky part.

When they are trying to list files and folders in Explorer, the files won´t show all at the same time but it can take several seconds for all files to list.

Also saving a regular PPTX file (3 pages 1285kb) it takes several seconds for the folders to list when pressing 'Save as', then several more seconds to save it (the save bar comes up and shows a slow saving progress).

Also, if they create a folder inside a current share it takes a long time for the folder to show up.

Home folder is synced for offline use.

After restart of PC the problems seem non-existent, after a while the problem appears again.

So, could this be SMB issue?

Should we disable offline files on the folder ("cifs shares -change <sharename> -no_caching")?

Could disabling 8.3 filename support for the clients be helpful?

Could NOT using cifs homedir make for slow connection..I mean,does the SMB list metadata for the complete searchpath?
(Exampel, user trying to save file to \\filer\users$\office1\group2\user\Documents\folder, does it list from users$ or from 'users' folder?)

Do you guys have any other experience where the filecopy speed is fine, but the user experience is slow??

Re: SMB Tuning

connoisseur — Tue, 26 Mar 2013 12:15:08 GMT

Updated:

today we did some copy tests on customers Head Office.

On a regular LAN speed copy test we get Read/Write :203/111 Mbps.

That´s really good and they shoudn´t experience any problems.

We did this on a "vanilla" Win7 laptop with local admin account to the NetApp NAS server. Both on UNC and IP.

But still, when they try to save a document, for exampel Word, excel or even a .txt file it takes 20 seconds to list all folders and another 20 seconds to save it..

Settings we have tried to change on client is:

From the Start Menu , type in 'Indexing Options' in the Search Bar
Click 'Modify'
Uncheck 'Offline Files'
Restart the workstation (not sure if this is necessary).
Turning on the Web Client service (which is mentioned here ( http://goo.gl/4HPQ1)

But still no luck....

both browse in explorer and "save as" seems to be slow. compared to regular filecopy

Re: SMB Tuning

connoisseur — Wed, 10 Apr 2013 06:17:28 GMT

We have now found the root cause of our problems.

I´ll specify them here first, and later I´ll link to all the KB´s and options I´ v found investigating this problem.

So it seems we had 2 issues here,

One on the client side, and one in the controller.

On the client side there was an issue with desktop.ini. As soon as a user would mapp their home folder the computer started to scan the entire folder structure for desktop.ini. Of course it got access deny on everyone (except its own). It also did 4 retries to be certian of its access. This caused 20-30 MS delay and with thousands of users with desktop.ini in the folders structure we got a huge delay.
There is a KB on how to fix this from MS http://support.microsoft.com/kb/326549 , but our customer to another approach at this.
As the desktop.ini file only function as a config file for the MS FilesSystem it´s not necessary, so the customer simply deleted all of them.
Now, instead of 40-45 seconds to map a folder, it takes 0,5 seconds.
The problem in the controller was due to Burt 609208 (http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=609208)
The value for the "cifs.tcp_window_size" option is not persistent for non-default vfilers after giveback, so even if we hade the tcp_window
_size correct it set it to default 17520. This is fixed in 8.1.2P3

So other things I found to speed up the access is the following.
From the client:

1.From the Start Menu , type in 'Indexing Options' in the Search Bar

2.Click 'Modify'

3.Uncheck 'Offline Files'

4.Restart the workstation (not sure if this is necessary).

Also there´s a service that was running on XP, but set as Manual in Win7 called 'Web Client'.
In some cases it seems like enabling that will improve the access speed.

Re: SMB Tuning

thomas_glodde — Wed, 10 Apr 2013 07:18:21 GMT

In citrix/terminal services we experienced issues when temp files didnt get cleared up properly, it generated plenty of SMB file/dir not found errors in a tcp dump.

Re: SMB Tuning

connoisseur — Wed, 10 Apr 2013 07:34:08 GMT

Thanks Thomas!

Well we do see a lot of "files locked", and I think that is due to temp files not being removed also.

But From previous investigation, that also seems to be a client/OS issue, or do you know about any Netapp specific tweeks on that?

Re: SMB Tuning

thomas_glodde — Mon, 15 Apr 2013 19:37:02 GMT

For our customer temp files as well as desktop.ini has been the biggest pain. Besides that he had some issues with appsense which i currently do not remember. Your best choise would be to do a network trace and analyze that one.Try colasoft capsa 7.

Re: SMB Tuning

fugropelagos — Mon, 15 Apr 2013 20:17:12 GMT

203/111 Mbps

I'm praying this is MBps not Mbps...

Do you not find this utterly insane? You're getting double the performance on reads vs. writes when NetApp is a *write-optimized* filesystem! It should be the exact opposite, and yet it's not.

Re: SMB Tuning

thomas_glodde — Mon, 15 Apr 2013 20:36:06 GMT

when it comes to 1gbit lan, any recent netapp can max that with sequential i/o if window size is big enough.

when it comes to 10fbit tho it depends on the machine, ranging from 200/300 megabytes per second up to wirespeed, which is around 850 megabytes per second if i remember corectly.

besides that, read is usualy always faster than write on any server or storage system.

Re: SMB Tuning

fugropelagos — Wed, 17 Apr 2013 20:41:14 GMT

Yes, but not on NetApp systems. WAFL provides for writes to be instantly ACK'd the moment they hit NVRAM. Further, writes are coalesced prior to be sent to disk. NetApp is 180 degrees out from every other storage vendor on the subject of write performance and methodology.

Re: SMB Tuning

thomas_glodde — Thu, 18 Apr 2013 07:41:51 GMT

Still, write needs to go into write cache and write log transactions hits the NVRAM, slower than just pump out data from read cache. Check netapp internal performance comparison reports, reads are always faster than write, even on netapp.

Re: SMB Tuning

john_higgins — Wed, 23 Sep 2015 11:38:17 GMT

Hi,

I have had issues for both for SnapMirror (opnly in one direction and on one filer being incredibly slow) and a few users not being able to even see CIFS shares where other users on same subnet next to them could.

The fix for this was ip.fastpath.enable being on, noiw the default. Chnaged to off and issues went away, ip.fastpath is not compatible with 10Gb networks when HSRP is in use, which it nearly always is.

Result was instant as soon as option disabled.