ONTAP 7.2.5.1 Simulator & Slow writes

[jvcherney]

Anyone else seeing slow write speeds when using the netapp simulator??

I get the expected 100Mbit read speads... but it seems like its crawing along at 10Mbit when writing...

This is for NFS writes... Machines are on the same network.

Any thought Ideas would be great.

[danielpr]

Hi Joseph,

How about the traffic on your LAN ? Did you find any time the packet drops ? You run “netdiag” to confirm no networking issues.

Thanks;

Daniel

[jvcherney]

danielpr wrote:

Hi Joseph,

How about the traffic on your LAN ? Did you find any time the packet drops ? You run “netdiag” to confirm no networking issues.

Thanks;

Daniel

Sorry, still a no go,

I am working on a separate segregated LAN so there is no other traffic,

There are no packet drops that I am seeing,

The slowdown that I am seeing is only when seen in the simulator.

If I start an NFS server on the machine I can send and receive at full speed, and seeing as this seems to only be affecting write speeds (I can read w/ no problems) I am thinking this may be a purpose buld limit of the simulator.

Results of the netdiag

NetappSIM-1> netdiag
Performing physical layer diagnostics.....OK
Performing network layer diagnostics.....OK
Performing transport layer diagnostics.....OK

NetappSIM-1>    

Thanks,

Joe

[danielpr]

Hi Joe,

You can have a look at this kb https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb36582. And also worth looking into the CLI  O/P for nfs options.

Thanks

Daniel

[jvcherney]

danielpr wrote:

Hi Joe,

You can have a look at this kb https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb36582. And also worth looking into the CLI  O/P for nfs options.

Thanks

Daniel

Thanks Daniel, but Still no joy.

I've increased the "nfs.tcp.recvwindowsize" & "nfs.tcp.xfersize" to the max (65536), tried both TCP & UDP Protocols, tried adjusting the "rsize" & "wsize" on the client.

I've even gone as far as setting an NFS server on the Host machine to make sure its not the machine. (results below)

NFS from Client to Host

     dd if=/dev/zero of=/u06/test.file bs=8192 count=2048
     2048+0 records in
     2048+0 records out
     16777216 bytes (17 MB) copied, 1.795 seconds, 9.3 MB/s

NFS from Client to Simulator

     dd if=/dev/zero of=/u04/test.file bs=8192 count=2048
     2048+0 records in
     2048+0 records out
     16777216 bytes (17 MB) copied, 19.2495 seconds, 872 kB/s

Thanks for the Direction, but still stumped.

Joe C.

[danielpr]

Hi Joe,

I tested it in RHEL server i dont find such delay in the NFS file copy.

Thanks

Daniel

[jvcherney]

Ok, I give up.

I've tried everything that I can think on.

3 different network cards, static / dhcp, even went as far as to connect the 2 machines directly via crossover.

and I still get the  same results... Blazing fast NFS transfers from the client to Simulator's Host IP... but dog slow when going to the Simulators IP.

It seems that I am cursed.

[jvcherney]

Ok Ok, So I didn't give up... But still having the Problem...

I've set up CIFS, and I still have the same slow 850K's writes...

I've Tried...

PC 2 VM(Sim)

VM 2 VM(Sim) same v-lan

VM 2 PC(Sim)

PC 2 PC(Sim) Direct via Crossover cable

Crux 2.5, CentOS 4 & 5, OULinux 4 & 5, RedHat 9

Broadcom TG3, 2x Intel E1000, 2x Intel E100

Even tried the ONTAP 7.3.1 Simulator w/ the same outcome.

Every install, Every combination results in the same 850KB's write speed.

Is there a setting that I am missing, some secret chant that I need to do???

I cannot believe that nobody out there doesn't see the same thing.

Anyone? Anyone?

Is this thing on?

Give me something Please!

Thank you,

Joe

[danielpr]

Joe,

What about the memory size in Linux machine and how much you have shared it for Data ONTAP simulator?

It doesnt seems to be a network issue since you have tired with back to back also using cross over cable.

Thanks

Daniel

[jvcherney]

For the stand alone systems, there is 1 gig install w/ 512 allocated for the Netapp Simulator (There is no swapping as its the only thing running)

For the VM's, they are set to 2 gig w/ 512 allocated for the Netapp Simulator (Again no swapping as they are the only thing running)

I've also tried disabling HyperThreading in the standalone systems thinking its getting "stuck" in the hyperthreaded core... but that of course didn't help...

If you tell me that there is no network / write throttle then something has to I am doing... although I realy don't know what... the install is as simple as it can get....

Thank any way,

Cheers,

Joe

[jedrzej_lisowski]

Hi,

I've got exactly the same problem, about 115MB/s when reading from netapp sim (iscsi, nfs) and 360-500kB/s when writing (iscsi,nfs), netapp simulator version 7.3.2, host os ubuntu 10.04 server, env VirtualBox with Host-only networking.

[jvcherney]

I was never able to get it to work right regardless of Real vs VM, or OS combnation, or simulator settings.

If you figure something out, let me know. I have abandoned the use of the simulator for this reason...

It would be nice to have something to test methods against where it dosent take 30 min to copy a simple 1 gig file.

Joe

[dan_keating · Accepted solution]

If you run sysstat -x 1 on the sim as you're writing data to it - what's happening with the CP types? back to back and deferred back to backs perhaps? (man sysstat for cp type info)

The simulator has a pitiful simulated nvram (circa 2MB I seem to remember) that get's flooded almost immediately - so it's probably constantly backing of write requests.

I could do with a bit more oomph tbh.

[jvcherney]

YOU sir seem to be correct.

4%    25     0     0      25   861    24   1624   2012     0     0   >60  100% 100%  Bv  90%      0     0     0     0     0     0
  5%    50     0     0      50  1723    49   2176   2152     0     0   >60  100% 100%  Bf  97%      0     0     0     0     0     0
  2%    25     0     0      25   861    24   2576   2588     0     0   >60  100% 100%  Bf  94%      0     0     0     0     0     0
16%    24     0     0      24   818    23   1466   1827     0     0   >60  100% 100%  Bf  93%      0     0     0     0     0     0
  4%    25     0     0      25   855    24   1772   2014     0     0   >60   99% 100%  Bf  96%      0     0     0     0     0     0
  2%    25     0     0      25   861    24   1880   2116     0     0   >60  100% 100%  Bf  92%      0     0     0     0     0     0
  2%    25     0     0      25   861    24   1760   1992     0     0   >60  100% 100%  Bf  93%      0     0     0     0     0     0
  2%    25     0     0      25   861    24   1768   1904     0     0   >60  100% 100%  Bf  93%      0     0     0     0     0     0
14%    24     0     0      24   839    24   1786   1918     0     0   >60  100% 100%  Bf  96%      0     0     0     0     0     0
  4%    50     0     0      50  1716    48   2092   1984     0     0   >60   99% 100%  Bs  91%      0     0     0     0     0     0
  5%    25     0     0      25   861    24   2308   2412     0     0   >60  100% 100%  Bf  99%      0     0     0     0     0     0
  2%    25     0     0      25   861    24   1824   2204     0     0   >60  100% 100%  Bf  92%      0     0     0     0     0     0
  2%    25     0     0      25   861    24   1912   2320     0     0   >60  100% 100%  Bf  96%      0     0     0     0     0     0
13%    25     0     0      25   861    24   1784   1848     0     0   >60  100% 100%  Bf  98%      0     0     0     0     0     0

Now.... Any idea how to get around this little hick-up?

Thank you,

Joe

[dan_keating]

There's no way around this I'm afraid. It uses a file which is 2 MB in size - and even if you were to tamper with it (naughty naughty) and, say, symlink it to a larger ram disk in memory - you would probably find that the application detects this and comes back screaming about invalid nvram sizes.

It's a shame because it really does throttle the sim to death.

[jvcherney]

What about the NetApp guys? Is there anything that can be done to work around this??

The limitation makes the simulator almost useless, waiting 20 - 30min to copy a 1gig file... makes testing nearly anything overly time consuming.

Thank you,

Joe

[jedrzej_lisowski]

I'm also waiting for some resolution (7.3.2 also has slow writes), maybe 7.3.3 will fix this? Or maybe 8.0 will be released working on AMD, and maybe different env than VMWare.

[DEADSOUL0]

I'm also using netapp 8.1.2 simulator and experiencing very slow write speeds.

Sad because I was attempting to lab some stuff as we use NetApp at work.

[UniversityOfHull]

Same issue here on simulator 8.0.5 7-Mode.

Pretty annoying when your testing transferring large file across LUNs