Thanks for the tips - I tried your method #1 with the wizard and for the last 5 minutes its been loading the available counters... I'll try the other way too with baselines Is there a TR doc for this? thanks again, Fletcher. ... View more

Hi, a followup: we recently modified the Oracle RAC on Netapp NFS workload (disabled a high IO operation) and immediately noticed a reduction in the pw.over_limit level (13/sec -> 5/sec average)  and time of day pattern. Are these .nfs files using some kind of block level IO that is unaligned? [db-03 ~]$ sudo /sbin/fuser -c /oracrs/vote03/.nfs0000000000b1298500000005 /oracrs/vote03/.nfs0000000000b1298500000005:  5487 13216 13249 14304 14408 14442 14659 [db-03 ~]$ ls -la /oracrs/vote03 total 41124 drwxrwxrwx 3 oracle oinstall     4096 Jun 20  2008 . drwxr-xr-x 5 root   root         4096 Mar 14  2008 .. -rw-r----- 1 oracle oinstall 21004288 Mar 17  2008 .nfs0000000000b1298500000005 drwxrwxrwx 3 root   root         4096 Sep 14 14:08 .snapshot -rw-r----- 1 oracle dba      21004288 Sep 14 14:18 irtcrs_dss03.ora [db-03 ~]$ ps -ef | egrep "5487| 13216 |13249| 14304|  14408 |14442 |14659" oracle    5487     1  0 Aug11 ?        00:00:00 /export/app/crs/11.1.0/bin/oclskd.bin oracle   13216 13207  0 Jun02 ?        00:21:38 /export/app/crs/11.1.0/bin/evmd.bin root     13249 11730  0 Jun02 ?        08:57:02 /export/app/crs/11.1.0/bin/crsd.bin reboot oracle   14304 13309  0 Jun02 ?        00:00:00 /export/app/crs/11.1.0/bin/diskmon.bin -d -f oracle   14408 13358  0 Jun02 ?        04:04:41 /export/app/crs/11.1.0/bin/ocssd.bin oracle   14442 14414  0 Jun02 ?        00:00:00 /export/app/crs/11.1.0/bin/oclsomon.bin root     14659     1  0 Jun02 ?        00:00:00 /export/app/crs/11.1.0/bin/oclskd.bin oracle   14715 13216  0 Jun02 ?        00:01:37 /export/app/crs/11.1.0/bin/evmlogger.bin -o /export/app/crs/11.1.0/evm/log/evmlogger.info -l /export/app/crs/11.1.0/evm/log/evmlogger.log Its hard to find any info on the alignment of Oracle RAC files on NFS. Anyone with insight to share? thanks ... View more

I opened a case and was told Netapp does not support mbralign on Solaris (and to open a case with Sun) This despite the fact the NetApp® Virtual Storage Console 2.0 for VMware® vSphereTM Installation and Administration Guide index lists grub as a step in the realignment process for Solaris Solaris reinstalling GRUB after running mbralign 58 the case engineer copy and pasted the steps to me from the guide and when I asked if he had tried those steps in the lab (because I have many times and all Solaris grub fixes fail - or say they succeed and result in a non-bootable VM) He then countered with the "Netapp does not support the Solaris grub fix - call Sun" I have yet to hear of _anyone_ successfully aligning and grub fixing a Solaris VM into a bootable, aligned VM. Since we have very few solaris VMs left this is not a huge deal for us - we're migrating all VMs to linux. thanks http://vmadmin.info ... View more

Turns out the impact of misaligned VMs can be more dramatic if the level of unaligned IO tips ONTAP into synchronous mode. One indicator is the pw.over_limit stat - ref: TR-3593.pdf 1.4 Counters that indicate Improper Alignment There are various ways of determining if you do not have proper alignment. Using perfstat counters, under the wafl_susp section, “wp.partial_writes“, “pw.over_limit“, and “pw.async_read“ are indicators of improper alignment. The “wp.partial write“ is the block counter of unaligned I/O. If more than a small number of partial writes happen, then WAFL® will launch a background read. These are counted in “pw.async_read“; “pw.over_limit“ is the block counter of the writes waiting on disk reads. This counter is not exposed via SNMP as standard, but can be trended as outlined here: http://www.vmadmin.info/2010/07/quantifying-vmdk-misalignment.html thanks ... View more

Hi, A quick followup outlining how we currently quantify the misalignment issue: http://www.vmadmin.info/2010/07/quantifying-vmdk-misalignment.html Cheers ... View more

After I took apart the 6800+ IOPS on the problem aggregate the issue turned out to be we were hitting physical limitations of the 10K RPM disks. Further analysis (surprisingly) revealed about 50% of these IOPS were snapmirror related. We rescheduled the snapmirrors to reduce this and have said goodbye to the latency spikes. If interested in the details, please see: http://www.vmadmin.info/2010/07/vmware-and-netapp-deconstructing.html I want to thank Netapp support - especially Errol Fouquet for his expertise pulling apart this problem and help isolating the cause. Fletcher. ... View more

Hi, we are experiencing HUGE 1,000,000+ microseconds (1 second+) latency spikes on ONTAP 7.3.3 NFS volumes as reported by NetApp Management Console which is disabling VMware virtual machines (Windows SQL server needs to be rebooted, Linux VMs go into read only mode and need reboots etc) We have a case open with Netapp (2001447643) and the latest analysis of perstat and archive stats from the spikes is being presented to us: "Version:1.0 StartHTML:0000000149 EndHTML:0000003705 StartFragment:0000000199 EndFragment:0000003671 StartSelection:0000000199 EndSelection:0000003671    This data is definitely good.  We are seeing the latency. Here is what I am seeing on the filer side: Server rpc: TCP: calls      badcalls    nullrecv   badlen      xdrcall 232298     4294959566  0          0           4294959566 The NetApp filer is getting a huge number of bad XDR calls, indicating that the filer is unable to read the NFS headers.   We cannot determine at this time what the source of these bad calls is. Some of the worst offending volumes during this period, regarding latency appear to be:   Vm64net Vm65net Vw65net2 Vm65net3 Ora64net02      Time   Time Delta   Volume   Parent Aggr   Total Op/s   Avg Lat (µs)   Read Op/s   Read Data (B/s)   Read Lat (µs)   Write Op/s   Write Data (B/s)   Write Lat (µs)   Other Op/s   Other Lat (µs)      Tue Jun 15 17:45:46 UTC 2010   0.00   vm64net   aggr1   311.00   6,981,691.95   35.00   336,402.00   1,336,540.48   267.00   1,561,500.00   7,953,614.68   8.00   4.29      Tue Jun 15 17:45:46 UTC 2010   0.00   vm65net   aggr1   115.00   6,283,673.41   0.00   2,453.00   38,863.33   107.00   1,441,475.00   6,714,803.11   6.00   12.41      Tue Jun 15 17:45:46 UTC 2010   0.00   vm65net3   aggr1   292.00   3,481,462.35   14.00   110,824.00   1,390,729.32   263.00   1,582,710.00   3,780,725.12   14.00   6.82      Tue Jun 15 17:45:46 UTC 2010   0.00   ora64net02   aggr1   17.00   3,280,731.47   5.00   92,421.00   4,776.50   7.00   24,536.00   7,710,536.08   4.00   2.77      Tue Jun 15 17:45:46 UTC 2010   0.00   vm65net2   aggr1   315.00   2,838,381.82   11.00   56,383.00   22,902.19   287.00   1,805,548.00   3,105,157.06   15.00   21.81  Out best bet to track down the source of the bad calls would be to capture a packet trace from the filer when this issue is occurring." My points I’d like to clarify: What is a bad XDR call and why are they relevant to the latency spike? “indicating that the filer is unable to read the NFS headers” - need you to clarify and expand We saw another smaller spike around 2:30am today: These are all volumes on aggregate aggr1 (10K RPM disks)  is this an overloaded (IOPS-wise) AGGR issue? We can't currently predict when these spikes in latency occur - they are random - so getting a packet capture of a random event does not seem feasible... Any insight is welcome - we are in major pain with this for months now thanks ... View more

No, we have 90% allocation in most (with aggregate snapshots disabled) thanks ... View more

Henrik, yes we disabled AGGR snapshots over a year ago Still searching for an explanation for the spikes ... View more

Hi, this thread shows up on the top of "Netapp Latency Spikes" searches. We have a 3040 cluster hosting 11 vSphere hosts with 200 VMs on NFS datastores. We see latency spikes 3-4 times a month as reported by Operations Manager. We hoped our upgrade from 7.3.1.1 last week to 7.3.3 would help, but we had another spike up to 1 second take out a NFS mount and all several of the VMs on Saturday. We previously  determined the High & medium IO VMs and either aligned them or migrated them to local disk - has NOT helped - still getting the spikes. I have another case opened with Netapp. Following the notes in this thread, I ran the wafl_susp -w to check the pw.over_limit Turns out ours is ZERO (is it relevant to NFS?) I suspect an internal Netapp process is responsible for these (dedup?) - we had it disabled on 7.3.1.1 - 7.3.3 was supposed to fix this (we re-enabled de-dup after the upgrade) And the latency spike outages are back Will share any info from the case thanks for any tips, Fletcher. ... View more

I opened a case - I'm seeing this in the vsc.log: Discovery cannot complete, setting overall status to Unknown thanks ... View more

I heard by March 2010 - possibly sooner ... View more

Eric, thanks for the reply - yes the volume is sis (dedup) enabled. df -sh Filesystem                used      saved            %saved /vol/vm2/               1333GB     1852GB          58% Is it the case we can not enjoy both dedup savings and rapid file level cloning simultaneously? We are looking at implementing VDI in the future so it would be very nice to have both. A few follow up questions: 1) If RCU is supported, why have the two cases I opened been closed and stalled with "RCU is not supported"? On the current case: 2000881634 - the engineer can not find documentation on how to perform the  file level clone from the command line I found it in Scott Lowe's blog post from Dec 2008: http://blog.scottlowe.org/2008/12/03/2031-enhancements-to-netapp-cloning-technology/ Is there a technical doc # that fully describes the RCU and file level cloning? 2) If RCU is based of the shared block model of dedup, why are blocks being duplicated at all ? Shouldn't the new clone file just consist of pointers to the blocks of the old file? 3) In my test I was able to run the file level clone command from vfiler0 on a volume that is actually NFS exported from a different vfiler. Could RCU be made to support vfilers in the future? thanks for clarifying ... View more

Netapp support is researching how to use file level flexcloning on the command line - I found this great explanation on Scott Lowe's blog from Dec '08: http://blog.scottlowe.org/2008/12/03/2031-enhancements-to-netapp-cloning-technology/ sounds like its not supported in a vfiler context. At least I know how to run it on the command line now - in my testing it does not seem rapid at all - copying individual blocks 1 % at a time Seems to defeat the purpose of using this over traditional VM cloning via VMWare VIC thanks ... View more

They still say its not supported - see new engineer's reply and my response below: My response; Ok, I don’t want to use RCU if I can not get support Regardless -  my goal remains the same: I want to use file level flexclone in ONTAP 7.3.1.1 to clone a file /vol/vms/vm1/vm1.vmdk to /vol/vms/vm2/vm1.vmdk (I can rename the file to vm2.vmdk later) How do I accomplish this via the ONTAP  commandline? thanks On 7/1/09 6:40 PM, "[email protected]> wrote: Hi, Fletcher, I checked with our engineers on your questions who insisted that the no-support status of this utility remains in effect. The only publicly accessible support documentation is available on the RCU2./01 Description Page where you may review Release Notes, Best Practices, and an Installation and Administration Guide. http://now.netapp.com/NOW/download/software/rapid_cloning/2.0.1/ You may consult with your NetApp Sales Engineer to learn whether there are alternatives available. I apologize for your inconvenience. ... View more

Hey thanks Adam - I re-opened the case (#2000881634) citing the internal doc you provided - will see how it goes now! ... View more

Hi Peter, thanks for the feedback - if the SVMotion plugin author was to look at incorporating Netapp calls to attempt this - what Netapp API/SDK would be most appropriate? If the offset is a value read in at boot time by the OS, VMWare may have the necessary layer of abstraction to make this change transparent to the SVMotion destination before it takes over for the source... thanks ... View more