On a related note we had a Linux VM on netapp NFS datastore with IO performance issues. Following VMware support's recommendation we were able to tune the Guest OS to improve performance: http://www.vmadmin.info/2012/10/linux-vm-io-scheduler-tuning.html
... View more
Will online alignment eventually be supported for NFS datastores? Yes we have the setting on nfs.vstorage.enable on Cloning a VM was no faster with VAAI - which operations are accelerated with VAAI on NFS? thanks
... View more
Ok, I believe I've satisfied all the requirements to get VSC 4 to show me my unaligned VMs (ESXi 5.1, OnTap 8.1.1, VAAI NFS .vib, rebooted etc) But I still see no VMs in the folders: I will open a case with Netapp
... View more
There are 2 pieces to install - VSC 4 and the more recently released NFS VAAI plugin (which is not installed via update manager, but a manual .vib download to your vCenter, then rename the file...just follow the PDF doc off the link below, and you will need to reboot the ESXi host per the doc) Download the NetApp NFS Plug-in for VMware VAAI: a) Go to the Software Downloads page at support.netapp.com/NOW/cgi-bin/software. b) Scroll down and click NetApp NFS Plug-in for VMware VAAI. c) Select the ESXi platform. d) Download either the offline bundle (.zip) or online bundle (.vib) of the most recent plug-in. I tried cloning a VM and VAAI was obviously not involved - the IO still went through the ESXi host. Linked clones are supposed to be faster? Still trying to find the Netapp doc describing the NFS operations for vmware that VAAI benefits.
... View more
I upgraded to 8.1.1 and I've got my NFS datastore showing Hardware Acceleration Supported So which operations can I now expect to be offloaded to the Netapp controller? thanks PS: and when can we get the NFS non-disruptive alignment supported?
... View more
We are seeing a 3x performance improvement when the VM is presented Netapp NFS storage as a guest OS disk vs the VM NFS mounting the same space. This is problematic in the case where our apache VM pool needs to mount a shared web tree - performance suffers - the shared mounts can not be make "local". the question is why the 3x performance difference? Specs: Dell 610s 196Gb Ram vSphere 5.0 U1 Netapp plugin for vCenter provides best NFS settings CentOS 5 VMs VM NFS mount options: rw,noatime,bg,nfsvers=3,proto=tcp,hard,nointr,timeo=600,rsize=32768,wsize=32768 thanks
... View more
I had 2003210902 open the whole time - when I requested "consult request in our internal AutoSupport knowledge exchange" another engineer took over the case. Then Netapp support recommended the static route - it worked, but I wanted them to look for root cause, why it was necessary in the first place on just this node. thanks
... View more
I did - the only routing table entry that _should_ apply is the default route (which the partner head _does_ use successfully) I added a dumb static route to force it to use the proper vif and autosupports now succeed connecting to support.netapp.com:443 This should not be necessary - feels like a bug.
... View more
Reviewing the firewall logs, I see the good head is logging the outgoing ASUP HTTPS connection to netapp.support.com on its vif associated with the default route as expected. The bad head is logging a vif NOT associated with the default route and the firewall is logging AGE OUT for this (since support.netapp.com is not responding before the timeout) The routing tables for both show the same default route Why is the bad head "preferring" the wrong vif for sending autosupports ? thanks
... View more
Hi Rudy - I have a case open on this. One node of our 3270 cluster is logging "transmission-failed MANAGEMENT_LOG support.netapp.com/put/AsupPut "couldn't connect to host" " (see history below) It is able to email us the autosupport, but the HTTPS connection to support.netapp.com is failing for this node With support we've verified routing is the same, autosupport options are the same, traceroute looks the same for both nodes, etc. The engineer wants me to add a static route - for support.netapp.com but the defauly route is working for the partner and we have any egress firewall rules... Any ideas what else to check? thanks! na04*> autosupport history show -fields seq-num,status,subject,uri,error,last-update seq-num destination last-update status subject uri error ------- ----------- -------------------- ------ -------------- --- ----- 347 smtp "6/14/2012 00:21:07" ignore MANAGEMENT_LOG - - 347 http "6/14/2012 02:48:26" transmission-failed MANAGEMENT_LOG support.netapp.com/put/AsupPut "couldn't connect to host" 347 noteto "6/14/2012 00:21:07" ignore MANAGEMENT_LOG - - 346 smtp "6/14/2012 00:14:34" ignore "PERFORMANCE DATA" - - 346 http "6/14/2012 01:29:31" transmission-failed "PERFORMANCE DATA" support.netapp.com/put/AsupPut "couldn't connect to host" 346 noteto "6/14/2012 00:14:34" ignore "PERFORMANCE DATA" - - 345 smtp "6/13/2012 01:18:38" ignore MANAGEMENT_LOG - - 345 http "6/13/2012 03:14:23" transmission-failed MANAGEMENT_LOG support.netapp.com/put/AsupPut "couldn't connect to host" 345 noteto "6/13/2012 01:18:38" ignore MANAGEMENT_LOG - - 344 smtp "6/13/2012 00:40:33" ignore "PERFORMANCE DATA" - - 344 http "6/13/2012 01:55:28" transmission-failed "PERFORMANCE DATA" support.netapp.com/put/AsupPut "couldn't connect to host" 344 noteto "6/13/2012 00:40:33" ignore "PERFORMANCE DATA" - - 343 smtp "6/12/2012 00:46:44" ignore MANAGEMENT_LOG - - 343 http "6/12/2012 03:20:36" transmission-failed MANAGEMENT_LOG support.netapp.com/put/AsupPut "couldn't connect to host" 343 noteto "6/12/2012 00:46:44" ignore MANAGEMENT_LOG - - 342 smtp "6/12/2012 00:57:16" ignore "PERFORMANCE DATA" - -
... View more
I had a case open with netapp for over a week on my vFiler stuck in bogus "migrate failed" state in NMC preventing vFiler migration and our cluster upgrade. The suggested "dfpm migrate fix" failed, so I ended up using the command line vfiler migrate to migrate the vFiler and clear the condition. Our cluster upgrade is complete to 8.1GA thanks
... View more
I opened another case with support - seeing these SnapMirror resync of stridedbdev3 to 3270head1:stridedbdev3 : Volume is a clone and scanners have not completed on the parent volume: cannot execute operation. and the swizzling looks like it its looping - will suspending snaps help? Volume stride: Scan id Type of scan progress 61342 active bitmap rearrangement fbn 32693 of 36319 w/ max_chain_len 3 Volume stridedbdev3: Scan id Type of scan progress 108832 active bitmap rearrangement fbn 36044 of 36059 w/ max_chain_len 3 Volume stridedbdev4: Scan id Type of scan progress 108837 active bitmap rearrangement fbn 368 of 36059 w/ max_chain_len 3 Volume stridedbdev5: Scan id Type of scan progress 108842 active bitmap rearrangement fbn 35679 of 36059 w/ max_chain_len 3
... View more
thanks Scott - parent and 3 flexclone status below - can I suspend snaps to help things out? Volume stride: Scan id Type of scan progress 61342 active bitmap rearrangement fbn 31043 of 36319 w/ max_chain_len 3 Volume stridedbdev3: Scan id Type of scan progress 108832 active bitmap rearrangement fbn 34394 of 36059 w/ max_chain_len 3 Volume stridedbdev4: Scan id Type of scan progress 108837 active bitmap rearrangement fbn 34778 of 36059 w/ max_chain_len 3 Volume stridedbdev5: Scan id Type of scan progress 108842 active bitmap rearrangement fbn 34029 of 36059 w/ max_chain_len 3
... View more
Migrating another vFiler with flexclones - this time under 8.1 and latest Core and NMC 3.0.1 The baselines are done, but NMC is stuck "in progress" - job status reports warnings: Waiting for the deswizzling to complete on the parent volume 'flexcloneXYZ' of to be completed Will this eventually succeed or hang indefinately? thanks
... View more
We have one vFiler stuck in "Migrate Failed" state - I've tried dfm vfiler delete/add, but it comes back in the same state The vFiler is actually running just fine - DFM is out of sync with reality... any ideas? thanks
... View more
Looks like /etc/rc was updated correctly, but the route commands were permission denied. Luckily we have no static routes for our vFilers - just the default route This is feeling like a bug (which is not biting us - yet)
... View more
root is listed with no groups irt-na04> useradmin user list Name: root Info: Default system administrator. Rid: 0 Groups: irt-na04> useradmin role list Name: admin Info: Default role for administrator privileges. Allowed Capabilities: login-*,cli-*,api-*,security-* Name: audit Info: Default role for audit privileges. Allowed Capabilities: api-snmp-get,api-snmp-get-next,api-system-api-* Name: backup Info: Default role for NDMP privileges. Allowed Capabilities: login-ndmp Name: compliance Info: Default role for compliance privileges. Allowed Capabilities: cli-cifs*,cli-exportfs*,cli-nfs*,cli-useradmin*,api-cifs-*,api-nfs-*,login-telnet,login-http-admin,login-rsh,login-ssh,api-system-api-*,cli-snaplock*,api-snaplock-*,api-file-*,compliance-* Name: ndmp_role Info: Allowed Capabilities: login-ndmp Name: none Info: Default role for no privileges. Allowed Capabilities: Name: oracle Info: Allowed Capabilities: login-ssh,cli-snap* Name: power Info: Default role for power user privileges. Allowed Capabilities: cli-cifs*,cli-exportfs*,cli-nfs*,cli-useradmin*,api-cifs-*,api-nfs-*,login-telnet,login-http-admin,login-rsh,login-ssh,api-system-api-* Name: root Info: Default role for root privileges. Allowed Capabilities: *
... View more
At the tail end of this successful vFiler migration (the first of 16) I noticed this in the logs of the 8.1GA destination Permission denied, user root does not have access to route Thu May 3 14:43:15 PDT [irt-na04:tmig.step.completed:debug]: Step: 'Unbind_Source_Vfiler' of transparent migration completed at 32815 miliseconds. Thu May 3 14:43:15 PDT [irt-na04:tmig.step.started:debug]: Step: 'Configure_IP_Addresses' of tranparent migration started at 32815 miliseconds. Thu May 3 14:43:15 PDT [irt-na04:kern.cli.cmd:debug]: Command line input: the command is 'ifconfig'. The full command line is 'ifconfig na04-vif0-64 alias 171.65.64.100 netmask 255.255.255.0'. Thu May 3 14:43:15 PDT [irt-na04:tmig.step.completed:debug]: Step: 'Configure_IP_Addresses' of transparent migration completed at 32828 miliseconds. Thu May 3 14:43:15 PDT [irt-na04:tmig.step.started:debug]: Step: 'Configure_Static_Route' of tranparent migration started at 32828 miliseconds. Thu May 3 14:43:15 PDT [irt-na04:useradmin.unauthorized.user:warning]: User 'root' denied access - missing required capability: 'cli-route' Should I just allocate this capability to root? (why does root not have all capabilities?) what would be the useradmin command line incantation to fix this? thanks
... View more
No - I did not ask support about the 64 bit upgrade without adding disk but since we have the luxury of a standby cluster, I'm planning to test the secret commands without risk. Then do the supported workaround dance I'll tear down my 64 bit aggrs, build up the smaller (< 16Tb) 32 bit aggrs etc Its fun waking up in the morning with a potentially way to jump through these hoops to use the latest best features
... View more
I heard back from netapp support today that while snapmirror is supported from 32 -> 64 bit aggrs, there is an internal BUG currently preventing datamotion from being supported 32 -> 64 bit So as a workaround What I want to test next: datamotion 32 bit vfiler with live VM -> 32 bit aggr on standby cluster, then live upgrade the aggr and volumes to 64bit under 8.1 GA (this is supported by netapp) The steps are outlined in this thread (aggr 64bit-upgrade start aggr32b_to_upgrade -mode grow-all) If this works, then the plan will be to 1) tear down the standby 64 bit aggrs and on the standby 8.1 GA cluster 2) recreate them < 16Tb as 32 bit aggrs 3) datamotion all vFilers from current prod 8.1RC2 cluster (32 bit to 32 bit) 4) live upgrade the 32bit aggrs to 64bit (> 16Tb) 5) upgrade the 8.1RC2 cluster to 8.1GA 6) re-establish snapmirrors back thanks
... View more
What about this workaround (32->64 bit aggr and volume convert without adding disk) suggested by Robin back in Feb? Is this safe under 8.1RC2? thanks robin.mesotten at gmail Feb 9, 2012, 8:58 AM Post #5 of 5 (209 views) Permalink Re: Force 64bit aggr migration [ In reply to ] There is a way... But not supported.. 🙂 The only supported way is by adding disks... Commands below were executed in diag mode (priv set diag) ontap8-s*> vol create test aggr32b_to_upgrade 100g Creation of volume 'test' with size 100g on containing aggregate 'aggr32b_to_upgrade' has completed. ontap8-s*> df -h Filesystem total used avail capacity Mounted on /vol/test/ 95GB 536MB 94GB 1% /vol/test/ /vol/test/.snapshot 5120MB 0TB 5120MB 0% /vol/test/.snapshot ontap8-s*> aggr 64bit-upgrade start aggr32b_to_upgrade -mode check Checking for additional space required to upgrade all writable 32-bit volumes in aggregate aggr32b_to_upgrade (Ctrl-C to interrupt)...... Upgrading a volume to 64-bit consumes additional free space in the volume. The following table shows the space usage after each volume is upgraded to 64-bit: Volume Name Total Used Available Capacity ----------- ----- ---- --------- -------- test 100GB 5657MB 94GB 5% ontap8-s*> aggr 64bit-upgrade start aggr32b_to_upgrade -mode grow-all Checking for additional space required to upgrade all writable 32-bit volumes in aggregate aggr32b_to_upgrade (Ctrl-C to interrupt)...... Started 64-bit upgrade on aggregate aggr32b_to_upgrade and all contained writable volumes ontap8-s*> Fri Dec 16 13:58:59 GMT [ontap8-s:wafl.scan.64bit.upgrade.start:notice]: The 64-bit upgrade scanner has started running on aggregate aggr32b_to_upgrade. Fri Dec 16 13:58:59 GMT [ontap8-s:wafl.scan.start:info]: Starting 64bit upgrade on aggregate aggr32b_to_upgrade. Fri Dec 16 13:59:00 GMT [ontap8-s:wafl.scan.64bit.upgrade.start:notice]: The 64-bit upgrade scanner has started running on volume test. Fri Dec 16 13:59:00 GMT [ontap8-s:wafl.scan.start:info]: Starting 64bit upgrade on volume test. Fri Dec 16 13:59:01 GMT [ontap8-s:wafl.scan.64bit.upgrade.completed:notice]: The 64-bit upgrade scanner has completed running on aggregate aggr32b_to_upgrade. Fri Dec 16 13:59:01 GMT [ontap8-s:wafl.scan.64bit.upgrade.completed:notice]: The 64-bit upgrade scanner has completed running on volume test. ontap8-s*> aggr status aggr32b_to_upgrade Aggr State Status Options aggr32b_to_upgrade online raid_dp, aggr 64-bit Volumes: test Plex /aggr32b_to_upgrade/plex0: online, normal, active RAID group /aggr32b_to_upgrade/plex0/rg0: normal, block checksums
... View more