Hi, Provisioning Manager is being used to migrated several vFilers, but this one is repeatedly failing - can someone decipher why? Another issue is when this failure occurs, it  disables snapmirror on the head - killing other migrations and snapmirrors! (I end up having to issue an options snapmirror.enable on) Here is the excerpt from the vfiler_trans_migrate_log – can you tell me why the cutover of app-vf-01 repeatedly fails and why it sets options snapmirror.enable off causing all other snapmirrors to abort  ? thanks --------------------START OF NON-DISRUPTIVE MIGRATION------------------------------ SOURCE THE VFILER UNDER MIGRATION: app-vf-01 Snapshot create started Snapshot create completed took:2482 msecs Disable visibility started Disable visibility snapshot took:1252 msecs Snapshot disable started Snapshot disable completed took:0 msecs CUTOVER started : Thu Nov 11 20:49:24 PST 2010 Step:'Shutdown iscsi sessions running in vfiler:app-vf-01 context' took:0 seconds WAFL:Ref counts for volume 'appdata'          Volref count:1          snapmirror source WAFL:vol offline on 'appdata' failed error: CR_BEING_USED Multi-volume offline stats: Total volumes = 2, Volumes with errors = 1, Offline async msg complete = 312, Phase1 = 0, CP1 = 1166, Phase2 = 0, Phase3 = 0, CP2 = 468, Phase4 = 0, Name cache = 91, Phase5 = 3, Inode cache = 151, Phase6 = 21, CP switch = 1530, Phase7 = 121, Total Time = 3863 WAFL:vol online on 'appdata' failed error: CR_ALREADY_ONLINE Multi-volume online stats: Total volumes = 1, Volumes with Errors = 1, Time to online = 0 Multi-volume online stats: Total volumes = 1, Volumes with Errors = 0, Time to online = 1346 Snapshot deletion took :6073 msec ===MONITOR DUMP=== Total Heartbeats - 1 Frequency - 2500 millisecond [MON] WakeUp [MON] WatchDog = 120sec [MON] Abort [MON] Fallback Triggered on:app-vf-01 [FALL] Requested [FALL] Fallback Success [MON] Sleep to MEngine [MON] Sleep [MON] SleepToFallback [FALL] Sleep ===MONITOR DUMP=== ... View more

Hi, I've opened a case on this, but wanted to get the community's feedback Summary: Last night just after initiating 2 vfiler migrations, we experienced an outage on our new 3170 running 7.3.3 It dropped off the network around 10:50pm – I could not ssh to it – I logged in via the RLM and found the cf status said up, but all the network and NFS services were down. I ended up initiating a takeover from its partner, then a giveback after the problem head came up clean It was logging messages like these on the RLM console: ping: wrote 17.6.6.1 64 chars, error=No buffer space available na01> Wed Nov 10 23:00:07 PST [irt-na01: Java_Thread:info]: Lookup of time.school.edu failed with DNS server 17.6.7.77: No buffer space available.   syslogd: Could not forward message to host 17.2.65.2: No buffer space available Nov 10 23:00:07 [na01: Java_Thread:info]: Lookup of time.school.edu failed with DNS server 17.6.7.77: No buffer space available.   syslogd: Could not forward message to host 17.2.65.2: No buffer space available CPU was high before and after the outage NFS ops were low at the outage time Questions: 1) is this network related buffer space? 2) How can I track the buffer space usage? (is there an SNMP exposed metric?) 3) if its network buffers - is it related to a 10GigE bug? (I thought Netapp had worked those out for the most part) many thanks for any info / experience, http://vmadmin.info Fletcher ... View more

One of the useful reports "Volume Growth Rates" uses the Daily Growth rate to extrapolate "Days To Full" http://dfmserver/dfm/report/view/volumes-growth-rates?lines=20 It'd be more accurate for us if we could customize it to use the weekly growth rate (to smooth out daily outliers) is this possible? thanks, Fletcher. http://vmadmin.info ... View more

Hi, Does there exist a tool to make volume/vFiler migration suggestions based on optimizing overall performance and space goals of a set of filers? Analogous to how VMware DRS makes prioritized vMotion suggestions based certain criteria (CPU usage being primary criteria) - in the case of the Netapp tool the primary resources would be IOPs (observed average and peak) and storage space (observed current + growth rate). Inputs (all known by DFM): 1) aggregates: sizes, IO characteristics (how many IOPs can this aggr do at 5,10,20ms latency?) (based on type, number of disks) 2) volumes: sizes, IO characteristics (avg and avg peak IOPS) 3) administrative goals (similar to vmware DRS rules) where the admin can configure rules to force a vFiler/volume stick on a certain aggr, or vFilers/volumes not be located on the same aggr or cluster (eg for fault tolerance) Output: The system would use a contraints based system to perform combinatorial optimization on the inputs to  recommend migrations based on optimizing the volume/vFiler set of aggregates calculated ability to provide IOPs at 5,10,20ms etc Assumptions: - temp space (to rearrage volumes/vFilers on a fully allocated or  nearly fully allocated set of aggrs a certain amount of temporary space  will be needed) - all IOs are equal (the VMware concept of prioritizing via shares is not addressed directly) - instead the admin can dictate this volume/vFiler sticks here (eg This tool could work in different use cases: 1) fully allocated systems (suggest migrations to increase IO efficiency) 2) upgrades (how best to re-arrange volumes on new filer aggrs 3) presales sizing (inputs would be estimates of dataset IO and sizes) is there a tool like this in the works? thanks! Fletcher. http://vmadmin.info ... View more

Hi, we've got about 20Tb of data in 15 vfilers to migrate. I'd like to know how I can optimize throughput (eg will paring down the snapshots on the source before migrating help?) How can I best monitor the progress? I used "snapmirror status" on the last one, it was confusing because while the rate did not seem steady and it progressed up to 150Gb on a volume that was 85/500Gb full then completed. The main goal is to be able to estimate snapmirror initialization completion ETA so I can plan when I'll be doing the manual cutovers - I need to be watching all the clients during cutover, so can't let it happen automatically in the middle of the night... Is there a better progress monitor than snapmirror status? thanks ... View more

Hi, this one requires a little setup - I hope its clear: we are upgrading 2 campus area Netapp 3040 clusters to 3170s running 7.3.3 The snapmirror destination has already been upgraded to the 3170. We run all NFS served by vFilers.  Clients are VMware, Oracle, NFS logs, apache web content etc Our snapmirrors are all setup with the vfiler dr configure command line syntax. Our plan was to upgrade the remaining production 3040 cluster to the 3170 was to initiate a vfiler failover as we have practiced and documented several times in the past. (on 7.2.x) 1) Suspend VMware VMs, shutdown Oracle, tomcats, apache etc 2) vfiler stop (on 3040) 3) snapmirror update for all volumes 4) vfiler dr activate for all vfilers to "promote" the 3170 from snapmirror destination to production 5) re-animate VMware VMs, restart Oracle, tomcats, apache etc 5) upgrade the 3040 to 3170 6) re-establish DR vfilers and snapmirrors What we found when we did the steps we had documented from 7.2.x days (a small scale test with a small volume encapsulated in a test vfiler)  was the IP failover did not work as expected - we received duplicate IPs messages and soon after the VMs  in the test vfiler crashed when the ESX host got confused about the NFS datastore on the IP level due to duplicate IPs. We have had an open case with Netapp on this for a few weeks now without much progress. -- Last week we learned about the vFiler migration functions automated by Provisioning manager (PM). I tested them today and they worked flawlessly in offline and even online with a running VM - uninterrupted while the vfiler (NFS datastore) failover happened - I was particularly impressed to see the "converting to semi-synchronous snapmirror" messages. The IP failover seemed to be handled without the VMs or ESX boxes even logging a single timeout warning. However the PM vFiler  migration solution while seemingly free of duplicate IP issues, does not perfectly suit our vFiler failover goals since: 1) our vfiler and snapmirror relationships are already established and initialized.  In fact to re-initialized our terabytes of snapmirrors from scratch would take days - we want to use snapmirror update, not initialize 2) we want to end up not with the old vfiler being left in "needs cleanup" mode as the PM vfiler migration does, but we want to re-establish the snapmirrors in reverse from where we left them off (avoiding lengthy initialization times) So my question is: Does PM provide a method to discover existing vfiler DR relationships and provide administratively initiated failover automation ? Failing that, could we get the "script" PM is using to automate the vFiler migration and modify it for our needs? (PM already provides an input for a user customized script for vFiler migration) thanks for any feedback, Will summarize, Fletcher http://vmadmin.info ... View more

Hi, We are still burning in a new 3170 clusters running 7.3.3 - last night one of the heads logged: Emergency shutdown: Number of Failed chassis fans are more than tolerable limit. Shutting down now Shortly after midnight last night. When I went to check it this morning there were no fans failed (no amber lights on) and the head was sitting at the LOADER prompt. I typed 'bye' and it came up fine waiting for giveback. This is less than confidence inspiring - we want to cutover production to this cluster and having it flake out on a seemingly  non-existent fan issue is troubling. Anyone else seen this or anything like it? All HW diags appear green - do I have a faulty piece of hardware or not? thanks, Fletcher http://vmadmin.info ... View more