Hi, Unfortunately, unless you can move the data and delete the aggregate, you can't.  I've always found this a bit puzzling myself, but I've had occasion to have gotten "odd" size disks (a 250GB disk in an otherwise 500GB aggregate via 'vol/aggr add' and basically, the size of that disk in that aggregate will simply always remain the same. If the situation, for example, were reversed, and you had a single 1TB disk in a 2TB volume, when that 1TB disk failed, a 2TB replacement would only ever use 1TB of that disk. (The same is true for my case above with the 250GB disk, btw). Unless something has recently changed in ONTap, you basically just donated a 2TB disk for a 1TB job.  Experience is sometimes a painful thing, unfortunately. Good luck. ... View more

The discussion here is about standard maintenance routines that you will find in most databases.  "Vacuuming" or re-indexing or optimizing or defrag'ing are standard optimizing routines that many software packages require to work around "fragmentation" atrophy that happens as the data structures grow through insert and delete operations.  In this case it is not a "no-no".  The only thing you really need to remember is: Application first. Reallocate second. ... View more

Hi, Have you reviewed the messages file (attachment) that you received in your ASUP mails? (I really hope you configured at least this for your temp. solution).  That should give you a better idea of what is going on.  If you have had errors in the FC loops or from disks, that might explain the filesystem inconsistency.  If you have support, you probably should have received your spare disk long ago and should have swapped the bad one out.  You might even have made the problem worse by leaving the bad disk in the system causing FC errors that eventually lead to the WAFL problems. There may also have been disk firmware or shelf firmware issues that you should have give a bit of attention too that could have avoided the WAFL problems. The 24-hour issue has already been explained.  That shutdown should not, however, have caused any WAFL problems. Remove the bad disk. Insert the replacement. Start in maint mode and I think the reconstruction will comense.  A WAFL_check or wafliron is the only way out if you don't have enough disks to get an "emergency" root volume on-line.  If you really can't afford to lose the data, you should have NetApp Support looking at things with you.  ... View more

Hi, I have given all of this a bit more thought, although it probably isn't directly related to the choice of a storage solution.  The reason for this is that something here seems to be standing out on a design level.  I don't really have any idea what sort of application or data that your are working with (just the sizes and some basic structure), but basically, I think a I/O based application that has these sorts of I/O response time requirements is going to run into problems eventually because of physical constraints of using disk-based storage. I can't really comment on what your benchmark results mean, but I have a suspicion that something in the DAS test is lying.  There is no disk-based storage alive that can achieve 0.08 ms, except for perhaps SSD.  Basically, the filesystem cache or the underlying FS is telling you stories about the data actually being writing to disk.  Perphaps the DAS NVRAM is big enough to handle the benchmark I/O and that is where the data is, but that doesn't scale terribly well for larger operaitons. VMFS is not known as a terribly fast FS either.  So basically, because the hardware of today's world probably is going to disappoint expectations of < 1 ms, there are a few widely-used options to work around this: 1. For highly parallelized, real-time analysis, things like global file systems and NUMA clusters are the key.  They cost tons of money and are incredibly complex because of the high level of message passing and locking involved.  They also require complex software. 2. Databases. A releational database basically provides all of the buffering and indexing included to assure low response times and scalability. The write locking and buffering are already in the DB so you don't have to design your own.  Several of them work well with NFS as well (Oracle, Postgres, Sybase to some extent, etc).  Transaction logging also gives you a consistent picture of where you were when things went wrong. Clustering of databases (active-active) can also be found, but is also complex and doesn't scale linearly. 3.  Batch I/O.  Basically, I/O daemons and message passing to achieve sort of the same structure as a relational database, but some additional thought should be give to having optimized directory structures to decrease lookup/search times.  If you have a somewhat predictable processing operation, I/O daemons can prefetch and flush I/O without stopping processing and allow the worker daemons access to data sets in memory, which would meet your fast access requirements. The second two options (and partially the first) don't really include data protection in the form of hardware redundancy or FS protection necessarily, and backup is a whole different issue.  Depending on how disposable (or conversely, valuable) your data is, preventing its loss is an element that needs to be considered both technically and financially. I've worked at the edge of HPC setups (ca. 800 machines) for seismic data processing and one of the "low-tech" ways of getting the job done without the complexity of NUMA systems is batch I/O.  It limits cost as well because you don't need to scale for hundreds or thousands machines beating up the storage all of the time.  It's not as "cool" as having all of that infiniband equipment to play with, but it doesn't break as badly either. Anyway, all of this might be totally useless rambling, but I just had the feeling that something about the access time requirements just didn't feel right and that there might be a way to sensibly work around it. A method that sort of embraces the KISS principle and could be financially more sound at the same time.  I hope I haven't wasted your time. Good luck. 🙂 ... View more

I guess that is more a question of how much money you have.  Given enough PAM cards in a filer head to deal with most of your working data set, then you should be able to get constant access with very short response times.  I guess I would still collect as many facts as you can and see what NetApp comes up with.  NetApp NAS isn't always the clear speed winner, but you have a huge range of other functions/options that make it a better solution for situations where you need more than "dumb, fast disk" ... View more

Just regarding your one question: There will be times when reallocation is a good idea, even on dedup'ed volumes:  when you add new disks to an aggregate and such.  Otherwise, like I said,  highly and minimally deduped volumes will probably benefit in most cases. The rest is still more a matter of experience. There have been some imporovements in ONTap to alleviate some of the inherent problems with using WAFL and deduplication/reallocation with block-based storage allocation. 🙂 ... View more

I'm guessing that the culprit is to be found in the http interface.  It has probably made some assumptions about volumes that it shouldn't have.  If you can't find an active bug/fix then you might want to open a support case with all of the details. Otherwise, I guess I would familiarize myself with using the cli more often, hehe ... View more

Hi, Basically, there is probably too little information here to give a definitive result.  You haven't really stated what sort of servers you are going to virtualize, either. I guess I also disagree with the blanket statement that NFS wouldn't perform well enough.  See the specfs results for the previous line of NetApp products here: http://www.spec.org/sfs2008/results/sfs2008nfs.html   If you can live with 2-3 ms response times (easily comparable to SAN) then it could be a good choice.  You can also run your VMWare datastores on NFS.  IIRC, iSCSI and FC are also offered as "affordable" bundles. It all depends on more exact specification of performance characteristics and some idea of backup methodology, etc.  A NetApp sales agent/partner will be able to help you with sizing and prices. ... View more

Hi, I run reallocation on hundreds windows luns and a good handfull of luns with vxfs and zfs on top on the unix side. Reallocate as a command has been part of ONTap for probably 5+ years (longer as wafl scan reallocate, iirc). You said you use deduplication on "all vols".  Even for the Exchange databases?  I guess without some PAM card, I would have never been so brave. As I previously suggested in the other thread, see the System Administration Guide... and probably the Block Access Guide as far as interaction between deduplication and reallocate. There are also some other hints there at getting more performance out of Exchange.  High and low levels of deduplication with probably reallocate well and have advantages.  Those in the middle may very well be a matter of experience.  The two data manipulation mechanisms have sort of tangential purposes: space savings v. performance.  Dedup'ing Exchange probably isn't a good idea if you really need performance, at least not without a PAM card.  Reallocate'ing a dedup'ed volume might not help much at all.  Where did you read that you shouldn't run reallocate? There has been a good deal of mumbling about a more definitive Best Practices guide for deduplication and reallocate, but nothing has emerged from NetApp yet, afaik.  For now, try using a newer 7.3.x release and things will work better all around for deduplication and reallocate. ... View more

You can read more about reallocation in the ONTAP System Administration Guide for your release on the NOW site, under "System performance and resources"... i.e.: http://now.netapp.com/NOW/knowledge/docs/ontap/rel7351/html/ontap/sysadmin/frameset.html ... View more

Hi, You will probably need to run defrag on your Exchange databases with some frequency or Exchange will show its true face and slow to a creep and you'll be in trouble with pretty much everyone.  Microsoft default settings for defrag are very aggressive and basically they don't think that your storage has anything else to do than optimize their concept of mail storage all night, every night. The best plan is to probably find out how often it really needs to be done (there is probably some tool or log that will help here, I'm not an Exchange admin) and then to run 'reallocate' jobs on the NetApp side after the Exchange defrag is done.  Then you will have the most optimal layout. ... View more

How did you destroy your volumes?  (cli vs. http interface). Normally there should not have been any shares pointing to the "old" aggregate.  I had the impression that one couldn't destroy a volume with a CIFS share attached to it (but I may be confusing vfiler delete here) without forcing (-f) it. ... View more

Hi, It took me about 30 seconds to find this KB article: https://kb.netapp.com/support/index?page=content&id=3012279 There are additional references in the KB article.  It is not so easy to find in the TR's unfortunately.  I would hope that someone in NetApp would correct this since it is a pretty common question. ... View more

Hi, I was sort of hoping that this was NFS, then the situation is easy.  With Windows, there is no active-active sharing of LUN's anyway unless you use Veritas Cluster Server.  Windows 2008 can do NFS, so perhaps this is something to play with.  I can't attest to the quality of their NFS implementation, however. Setting up a CIFS share to be access by a system user isn't that difficult, shares just need to be mapped with a crude "net share" batch script and the filer needs a few lines of configuration.  It would be an advantage if it was on an isolated/private (v)lan segment, of course.  We run a couple of pretty loaded Notes servers using CIFS shares and it is amazingly stable.  There used to be some documentation on NOW on how to set it up, but last I looked I couldn't find it.  I have a working implementation, so I have my documentation, in a way. ... View more

Hi, Here is the information... after some delay: [root@dfm ~]# dfm volume list -a | wc -l 17894 [root@dfm ~]# dfm volume list | wc -l 1146 [root@dfm ~]# dfm qtree list -a | wc -l 20926 [root@dfm ~]# dfm qtree list | wc -l 1214 [root@dfm ~]# dfm lun list -a | wc -l 8149 [root@dfm ~]# dfm lun list | wc -l 683 I am not sure, however, how you conclude that this has some effect on the size of a snapshot backup.  If taking a snapshot simply means setting the database in "backup mode" (however this is done with Sybase), quiescing the filesystem, and taking a snapshot, why should the snapshot balloon? The performance data is on a different NFS mount (I think in the day, we just moved the database back to local storage... ).  In any case, the details fail me now and there was never any real solution from NetApp support, so we gave up and used our limited time on things that gave us results. Sybase should really work on an NFS mount (and has worked for us internally in testing), but that doesn't seem to be supported Now, there are some KB articles on taking snapshot backup, but it doesn't seem to be included in the standard documentation.  Even if it were, having the database on one of the systems that one is trying to manage can be a complicated (or just terrible) solution.  It seems to me that if I had a database solution that was offline for 30-45 minutes daily because of poor design choices, then I probably couldn't say that I have an enterprise class solution. Either the method of sampling data or the backup method needs to be changed. ... View more

Hi, Basically, you want to try to avoid that both the disks and the controller stop a the same time.  This makes it easier (possible) for the OS to make decisions concerning a fail-over.  When everything goes at the same time, you get a "split-brain" situation.  This basically means that the partner disappears and the remaining controller has no information about what happened so it waits for human intervention before taking over the partners disks to avoid corrupting data... or having two mirror copies that can't be united without loss of data. The situation that latecoere suggest is more ideal.  Make sure that either the controller or the disks fail first (with perhaps 30 seconds head start) before the remaining element.  This way, a fail-over will occur as expected, because there will still be enough information for the surviving head to make such decisions. Of the two, I guess I would prefer that the controller fails first. ... View more

Hi, I'm guessing some of the latency is simply coming from how and where it is measured.  I guess you would need to now how deterministic the sampling rate is for perfmon on the Windows side.  I don't know what stats you are collecting on the filer either.  Response times from LUN stats on the filer may point to activity much farther down in the system than what is going out over the wire (fiber). From that which I have collected from the Windows gurus here, there are a lot of patches to the SAN stack in 2008 as well.  You might want to also test a up-to-date R2 build as well. Are you using ALUA? ... View more

Hi, You probably should read the File Access and Protocols Management Guide for your release on the NOW site.  If you have specific questions after that, then ask them here. ... View more

I think you need to restate your question more exactly if you expect a useful answer. ... View more

First, I think you mean 7.3.5.1P2.  Second, there are a number of changes that will make the upgrade easier and any reversion back to a 7.x (7.3.5.1) much easier incase the upgrade to 8.0.1 goes wrong. You can easy view the bugs fixed between 7.3.1 and 7.3.5.1P2  on the NOW site. Follow NetApp's advice and your upgrade will be a lot less painful.  Since this is a major version upgrade, it is important to get it right the first time.  Reverting back to the previous release is a major PITA. ... View more

You should be able to get interfaces which support most of the 1000BaseX specs, including the fibre based ones (SX, at least) You will also be able to get 10GbE from 10GBaseSX cards and FCOE cards. ... View more

Hi, You might want to experiment with smaller write sizes more tuned to your ethernet frame sizes.  I would hope you are using jumbo frames.  I'd have to look at some earlier work of mine to see how such tests went in an "earlier life" I guess I would try to watch sysstat running during your test as well and see if you can get some useful stats out of 'nfsstat -h '. Zero the stats between runs with '-z'. The only other thing would be to try a number of hosts if the sysstat is showing that you aren't maxing out your disks, then it would be good to see how many clients and how much traffic you could push before you saturate the filer.  It would eliminate the filer as a problem area and leave you to tune the linux nfs and tcp subsystems. Turning on 'no_atime_update' on the filer volume would probably get you a bit more performance already, if you haven't done so.  Is this ONTap 8.x or 7.x? I know I'm not much help, but tuning interests me a bit. ... View more

Dude, on an ONTap  7.3.2 or new version system (iirc) try: priv set advanced; qtree You will see that it has been possible to delete qtrees from within OnTap for about 2 years now... 😉 ... View more

We have a setup that is probably half the number of controllers and a lot few qtrees, but we see the same behavior during backup.  We basically lose almost all DFM functionality during backups.  Is there any on-going work to fix this? We tried using snapshot backup once, but it seemed to balloon the storage requirements immensely.  Our local NetApp partner opened a case, but nothing ever came of it.  In fact, we've almost never gotten any results for DFM support... (Sorry to piggy-back your issue... but we've thought of splitting up monitoring as well, just haven't gotten that far) ... View more