We are getting towards the pointy end of the procurement phase of an infrastructure refresh at our SME. Part of this is replacing an aging (badly) FAS2050 + a couple of DS14Mk4 shelves with a new NetApp filer. The FAS2240-2 was an obvious candidate, as even an entry-level system such as this provides more than enough performance and capacity for our current and planned needs for the next few years. However, a key part of our upgrade plan was implementing FlashPool, and we only discovered late in the piece that the FAS2xxx series has a 400GB usable FlashPool limit per HA pair. This is/was cause for concern around whether this would be sufficient for us given a pending VDI deployment, and led to a flurry of research into alternatives, quotes on alternative systems, etc. Where we have "landed" is that we have been quoted on two options - the original FAS2240-2 and a FAS3220*, with otherwise identical disks/software licenses/etc. The interesting part is that the FAS3220 quote is only fractionally more expensive than the 2240-2 (think 5% or so). I am being told that the reason for the close-to-price-parity is a sharp deal on the FAS3220, whereas the FAS2240-2 is close to being a fixed price / fixed margin unit. While I have not yet verified the truth of the above (that the reason for price parity is a cheap 3220 quote, not an expensive 2240-2 quote), if we take it at face value that this is true and hence can buy a FAS3220 for essentially the same price as a FAS2240-2, can anyone think of any good reason why we wouldn't take the FAS3220 (remembering it is overkill for us in every way except the FlashPool limit)? All I can think of is that ongoing support / warranty will be a bit more expensive, and any additional software licenses purchased later will cost more (can anyone quantify approx how much?). I think the former is worth the cost, and the latter isn't too much of a concern as we barely use any separately licensed features now and I don't envision this changing in the future (except perhaps for SnapMirror - we may implement that). Thoughts from the gurus? And thanks, in advance. * we actually have two FAS3220 quotes, which are essentially the same price and spec except that one provides 7 x 200GB SSDs for 400GB + 200GB of FlashPool, and the other provides 2 x 512GB FlashCache cards. Secondary question here: for a small environment like ours that will only have one aggregate per controller, I'm thinking the FlashPool options makes more sense, but would like sanity check here - thoughts?
... View more
> This is a tough one. Reallocate likely would help with I/o but we shouldn't run it with Asis. I would run this by support and post back. Thanks for the additional response. I've just opened a case with support and will post back what they say. Also, do you happen to remember where it says you shouldn't run reallocate on a de-duplicated volume? I know I've read it to, but I can't find the reference now.
... View more
> Physical reallocation will not cause snapshots to grow, that's fine. OK, thanks for confirming that. So we just need to find out about de-dupe.
... View more
Hi Scott, > Are you running Asis? With dedup reallocate can be a challenge...not recommended. On 8.1 the release notes list fixes with this but I heard there are still burts about it. Yes, we're running dedupe: "This aggregate contains a bunch of flexvols used exclusively to store VMDK files for our VMware environment (provided over NFS), and has both snapshots and deduplication enabled on all volumes.". You say not recommended - as in there is no "safe" way to do it? Or we just have to be careful to run the correct command (as per my original question)? > If no asis reallocate make sense if disk I/o will be an issue after a small disk addition. I thought that after adding two disks to a very full aggregate a reallocate of some sort was pretty much mandatory, dedupe or not? Thanks.
... View more
Hi Radek, > > I think what we need to do is run "reallocate start -f -p <volume_name>" against each volume in the aggregate, but I'm not sure. > Yes, this is correct. More details could be found here: http://now.netapp.com/NOW/knowledge/docs/ontap/rel733/html/ontap/cmdref/man1/na_reallocate.1.htm Thanks. I did read that document but it didn't make things completely clear for our specific situation. Are you confident that using the above command won't "un-deduplicate" our de-duped data / cause snapshot space to balloon, as per my concerns? > > Can anyone give some (broad) guidelines as to how long we can expect a FAS2050 to take to reallocate c. 1.6TB of data across the newly-expanded 18-disk RAID-DP group (144GB 15K drives if it matters), assuming no other load on the filter? >I have no idea to be perfectly honest. FAS2050 is a rather limited box in the terms of its CPU & memory. On the other hand reallocate will not shift the entire 1.6TB from one place to another, only a portion of it required for proper re-balancing. > As a plan B, you can quiesce reallocate job, should it still be unfinished when the maintenance window ends. OK - thanks for the info anyway, and the idea re quiescing the reallocate job.
... View more
Hi all, Hoping the NetApp community can provide some assistance to this relatively novice NetApp administrator. Firstly, to describe the environment / scenario: - FAS2050 running DOT 7.3.5 - Head #1 (which this question relates to) has all 20 "internal" disks assigned to it - Currently, 16 of these 20 disks are in a single aggregate, all in a single RAID-DP raid group. This aggregate is c. 90% full at present. - This aggregate contains a bunch of flexvols used exclusively to store VMDK files for our VMware environment (provided over NFS), and has both snapshots and deduplication enabled on all volumes. - Out of the remaining 4 disks, 2 are hotspares and 2 are not being used at all (freed up after some recent changes involving head #2) - We want to add the 2 "unused" disks to the existing raid group / aggregate, increasing the raid group size to 18 in the process. I understand that we will need to run a "reallocate" after adding the 2 extra disks to the existing raid group / aggregate or else we will end up with "hot spots" on these new disks. From some research it appears that we need ensure DOT only does a physical redistribution of the blocks across all disk in the aggregate and does not try to otherwise rearrange them, otherwise we risk "un-deduping" our deduplicated data and/or greatly increasing the space consumed by our snapshots. Both of these would result in big problems for us as we'd rapidly run out of space in the aggregate - we need to perform the physical reallocation across the new disks without increasing space usage. Therefore, what I need help with is: Again from research I think what we need to do is run "reallocate start -f -p <volume_name>" against each volume in the aggregate, but I'm not sure. Can anyone confirm if this is correct, and provide the correct syntax if it is not? Also, a bonus question: I'm assuming the reallocate will put a significant load on our filer and hence should be run outside of peak hours. Can anyone give some (broad) guidelines as to how long we can expect a FAS2050 to take to reallocate c. 1.6TB of data across the newly-expanded 18-disk RAID-DP group (144GB 15K drives if it matters), assuming no other load on the filter? Thanks!
... View more
Hi Shane, > Allocating full shelves is probably best practice and if you can i would suggest you do, But i've seen and done 1/2 shelves before with no decernable issues. If best practice says do it X way, then all ways try and do it that way. > FC and SAS drives are effectively the same physical spindle in most cases so i doubt there would be any problems. OK, thanks for the clarifications. > With your RG question, in one you've got 2x12 disk stripes and the other you've got 1x 26 disk so theortically the single stripe should have the greatest theoretical throughput. There is a lot more to the discussion that that, failure domains is one of them the chance of a 3 in 28 failure is higher than the chance the chance of 2x 3 in 14. Thanks for the extra info, but I'm still not clear on the performance aspects of this - sorry. With, e.g., 2 x 13 disk stripes, are we losing half of our performance compared to 1 x 26 disk stripe? Or will both options deliver essentially the same performance? Thanks, Matt
... View more
Hi Shane, > I wouldnt be too fussed about where the disks are physically located, as soon as you start getting failures they will start moving around anyways. What are you trying to achieve? given your shelf count you're not going to buy any more redundancy. I was under the impression that allocating whole shelves to a controller (so all disks in each shelf are owned by only one controller) was best practive, and was simply trying to folllow that. Is that incorrect? Or just not an "important" best practice? > As for RG sizes i would stick to smaller ones if you can (i.e. 12-18 disk) It seems to be the sweet spot from what i've seen. > The raid group layout and the disk assignment is dependant on how you want to use the filers. i would try and layout my data/disk equally between both controllers it allows you to better utilise your storage investment. > Option 2 seems fine, i would proabbly be tempted to use 2x13disk RG's rather than 1x26 but thats just me. I also wouldnt be too fussed about what disks were being used where assuming the SAS and FC are both 15K drives you'll be fine. OK, so even going so far as to mix SAS and FC (both are 15k RPM) in the same RG is also OK? I know NetApp allows it, but wasn't sure if it had any performance (or other) implications. Also, are you possibly able to clarify the following from my OP for me? Can someone clarify something for me to do with RG sizes and performance please? If we were comparing performance of 2 x 14 disk RGs vs 1 x 28 disk RG, would the former (ignoring parity disks) only give us the performance of a 14-disk stripe (the size of each RG), or would it give us the performance of a 28-disk stripe (the size of the two RGs combined)? Thanks for your input, greatly appreciated. Cheers, Matt
... View more
Hi Eugene, > Option #3 ? > Ctrlr1/Ctrlr2: > 24 disks each. > 22 disks each in 22 disk raid groups. > Disks split evenly across all shelves. > Use the 'disk replace' command to juggle out the existing disks to the new shelves. So what you are proposing is that we split all shelves evenly between each controller - i.e., half of internal shelf, half of FC shelf #1 and half of FC shelf #2 on controller #1, and the other half of each of these shelves on controller #2? If so, isn't this violating one of the best practice recommendations re allocating whole shelves to a single controller? > Consider the hot spot created when adding the few disks to the existing aggr - maybe do a 'reallocate'. Yes, we will do a reallocate if we add any disks to the existing aggregate on controller #1 - thanks though. Thanks for the input. Cheers, Matt
... View more
Hi all, We're in the process of adding some extra disks to our FAS2050A and I'm having a tough time figuring out the "best" way to allocate these disks (and hand-in-hand with this, choose the "best" RAID group sizes). Current system is a 2050A with 20 x 144GB disks - 16 data/parity + 1 spare presently allocated to controller #1 (currently out "active" controller), 2 data/parity + 1 spare to controller #2 (currently acting only as a "passive" controller). Additional disks are 28 x 144GB in two DS14MK4 shelves. As noted above, I'm trying to plan how we spread these disks across the two controllers, taking into account both our current setup and also possible future expansion. A few possible possible configs I've come up with - all RGs would use RAID-DP: Option 1: Controller #1: Data: 16 disks (internal shelf) in one 16-disk RG Spare: 2 disks (internal shelf) Controller #2: Data: 28 disks (DS14MK4 shelves) in two 14-disk RGs or one 28-disk RG Spare: 2 disks (internal shelf) Option 2: Controller #1: Data: 18 disks (internal shelf) in one 18-disk RG Spare: 2 disks (internal shelf) Controller #2: Data: 26 disks (DS14MK4 shelves) in two 13-disk RGs or one 26-disk RG Spare: 2 disks (DS14MK4 shelves) Some questions arising out of this: 1) Any other configurations I should consider? 2) Thoughts on RG size for controller #2 - 13/14 disks in below the default, 26/28 is approaching maximum. Obviously the choice is a trade-off of performance & efficiency vs reliability, but can anyone offer some insight into the "best" choice here? 3) Can someone clarify something for me to do with RG sizes and performance please? If we were comparing performance of 2 x 14 disk RGs vs 1 x 28 disk RG, would the former (ignoring parity disks) only give us the performance of a 14-disk stripe (the size of each RG), or would it give us the performance of a 28-disk stripe (the size of the two RGs combined)? 4) The reason for considering option 1, where controller #1 only uses 16 data/parity disks and controller #2 has its spares in the internal shelf, is to do with future expansion. Assuming we were to add another 28 disks in the future (2 more DS14MK4's) and wanted to add these disks to existing aggregates then I believe option #1 would be better. By adding a whole shelf to controller #1 and reclaiming the two internal-shelf spares assigned to controller #2 we could create a 2 x 16 disk RG aggregate with 2 internal-shelf spares on controller #1, giving us balanced RG sizes and following the best practice recommendation of assinging whole shelves to a single controller. Likewise, the other additional shelf could be added to controller #2 and would provide for 2 replacement spares + expanding controller #2's RGs to 2 x 20 or 1 x 28 and 1 x 12. We would be mixing SAS and FC in one of the controller #1 RGs though... I know we can do this, but not sure if it has performance implications? On the flip-side, I think option 2 is a better design for us now - it maintains the best-practice recommendation of assigning whole shelves to a single controller with the present setup, and does a slightly better job of balancing the number of disks per controller. It does make further expansion - e.g. adding another 28 disks - tricker though, as we'd need to either go for unbalanced RGs on controller #1 (18 + 14) or break the assign-whole-shelves-to-single-controllers best practice recommendation (2x18 disk RG on controller #1, with the second RG taking a whole shelf + part of another). I'm guessing the "right" choice here probably comes down to our specific environment + timeframe for additional expansion, but does anyone have any other thoughts on the matter? Or am I over-thinking this, with there being very little real-world difference between the two options? Thanks all, really appreciate any advice that is forthcoming. Cheers, Matt
... View more
> Found the document: > > http://now.netapp.com/NOW/public/knowledge/docs/hardware/filer/210-03951.pdf > > "Note - You do not need to ground your system" Thanks. I guess that clears up grounding (or lack thereof) of the FAS2050 to the rack, but sounds pretty ambiguous with regards to the need to ground the shelves to the FAS2050. Would you agree? Or do you think "You do not need to ground your system" means that grounding the shelves to the FAS2050 is also not required?
... View more
Hi Eugene, Thanks for the response 🙂 > Use any of the screws on the 2050 chasis you can reach to attach the grounding straps. That's the problem... we can't find any screws / screw holes on the 2050 that look like they could be used as a mounting point for the grounding strap(s) :-S > Use two straps, for redundancy, if you have them. Will do - thanks. > You should already be grounded to rack with the mounting screws. Do you mean we should already be grounded to the rack simply by having mounted the filer / shelves in it? Not sure if that would be the case with our racks as they are covered in some sort of black powder-coating that doesn't look very conductive to me. > I did not go look at the hardware installation guides for these answers, but there probably is documentation on this. If there's documentation then I can't find it 😞 The DS14MK4 guide tells you to use the grounding straps but only cover the connection point(s) on the DS14MK4 itself, and there doesn't appear to be any information about grounding straps in the FAS2050 documentation. Cheers, Matt
... View more
Hi all, We're in the middle of adding two DS14MK4 shelves to our FAS2050C. One of the steps in the installation instructions reads: "7) Connect the grounding strap connecting the disk shelf to the other disk shelves or your storage system." Three things we're confused about here, and can't seem to find answers for in the documentation: 1) The location to attach the grounding strap to the DS14MK4's is obvious, but we can't figure out where to attach it to the FAS2050. Any idea where it attaches? 2) There are two locations on each DS14MK4 to attach a grounding strap - immediately to the left and right of the ESH4 modules. Do we need to attach grounding straps to only one of these two locations per shelf (so one strap connecting each shelf / filer), or do we need to attach both? 3) Just confirming, we don't need to create a loop with the grounding straps, or attach them to anything except the shelves and the filer (no connection to the rack itself, or to any external earth / ground)? So it's just Shelf1 <-strap-> Shelf2 <-strap-> Filer? Thanks, appreciate any info / advice. Cheers, Matt
... View more
> However, I want to point out there is a tool on the NOW Utility ToolChest that estimates the transfer time and how much data is to be transferred Thanks, looks to be a useful tool.
... View more
> I posted in that thread with more details, but if you are sketchy on your WAN link, check into Riverbed Steelheads. I can post some data reduction screenshots for SnapMirror traffic a bit later. It's pretty amazing what it can do to your replication traffic. Interesting info, thanks for contributing. Have you compared how the Steelheads perform vs native DOT compression of SnapMirror transfer sets? Would be very interested to the see the data reduction info you referred to if you don't mind posting them.
... View more
> From my experience the FAS2050/FAS2020 can serve data well as long as it doesn't have to doing anything else. These are single CPU storage controllers and red line CPU easily when additional tasks are given to them. I can certainly attest to that as well - very easy to max the CPU on our 2050. > The FAS2040 is more powerful so I'm not sure of performance when having to handle multiple tasks. I'm pretty sure the 2040 is dual-core, so it should handle multiple concurrent tasks much better. One of the many reasons I wish it was around when we bought our 2050 :-S > I had problems with the following until I upgraded > > FAS2050 with two shelves of 300 FC - Production > FAS2050 with 14 1TB SATA - DR > > We deployed 40 Virtual Machines on 4 Volumes runing on Hyper-V > Exchange CCR Volumes > Multiple SQL DB Volumes > > We were updating snapmirror hourly for databaes and daily for Virtual Machines volumes > We were also deduping virtual machine data daily > > The result was 75% CPU utilization for production and 90% disk utlization for DR. Wow, SnapMirror must produce a massive amount CPU and disk load then. Our current environment is vaguely similar the one you describe (2050, 40-odd VMs, SQL DBs, daily dedupe, etc) except we don't use SnapMirror, and our 2050's average CPU load wouldn't be any higher than 20%.
... View more
> Check out the pics, I'm working on setting up my rack with them right now. Looking good. I assume you're putting the 2020 on your local LAN to getting everything config'ed right, and will then move it to your DR site later? > Here's the deal, I BARELY, and I mean BARELY was able to squeek out the funds to do all of this anyway. My justification for even buying the 2020 was, "If we go with EqualLogic we're going to need to spend ~20-30k on a backup infrastructure, so why not let me take that 30k and grab a device that will act as both a backup environment but also the foundation for a DR site as well." If the 2020 wasn't such a fantastic value right now this would have all come apart. I knew I was going to have to deal with being "stuck" at the 7.* DOT, but I'll take that functionality over anything from the main competitors. OK, fair enough then - completely agree that being "stuck" at DOT 7.* isn't exactly an awful situation to be in 🙂 Just wanted to make sure you were aware of 2020's limitation and that you weren't going into it "blind". > Aside from backups our total environment should require\generate <1000 IOPS for the most part. Goodo - I'd estimate you'll be fine then. > Booya, this is what I was looking for. That's exactly how I drew up the mix before, contr1 controlling 10 disks set to raid-DP, all one agg, and contr2 controlling 2 disks set to RAID4. Yeah, I had trouble getting my head around how to best utilise our disks when we first installed our 2050 too... happy I could help 🙂 > I guess it would be prudent to use at least one spare. Again, I'm going to do my best to get it rolling like this and then go back for another shelf if need be. It depends on your data / business, but I reckon I'd be inclined to skip the spare for now and get the most possible disks into "production". If/when you add another shelf you can then use one of those disks as a spare (or ideally two, as this enables the filer to put disks into the "maintenance centre" if required - read up on this in the manuals for more info). With the combo of low disk count, RAID-DP, fast NetApp warranty replacements AND your backup 2020, the chance of data loss, even without a spare, is very, very small. > We will be using SnapManager for SQL. I just attended the "Database Bootcamp" training session yesterday, and our instructor mentioned that NetApp just released (I believe an update to SMFSQL) which will allow everything to work correctly with SQL databases mounted on VMDKs on NFS. Being able to stick everything on NFS makes it all simpler. Ahh, that's excellent news for you then - as you say, NFS is incredibly simple, and from our benchmarks it performs just as well as iSCSI (with NetApp filers anyway). Have you read about / checked our SnapManager for Virtual Infrastructure (SMVI)? Great idea for Netapp+VMware farms. > I'm also not entirely sure that we won't have to upgrade our WAN link to handle the traffic Just FYI, this thread I started a while back may be helpful: http://communities.netapp.com/thread/7320 > 13 total NICS (including iLO) per Host (huge pain to cable) Our network design is a little different to yours, but I feel your pain. 11 NICs per host here :-S
... View more
> Matt the way you describe your path to your current infrastructure closely matches what I've been doing this year. In January I was planning on buying an HP\LeftHand based solution, then I almost went with EqualLogic, but after relentless digging and scouring I realized that if I could go NetApp I should. Yet another similarity between our respective paths / infrastructures 🙂 When researching SANs I rapidly narrowedthe choice down to EqualLogic, LeftHand and NetApp, and in the end chose NetApp. > The 2040 is a dual controller model. The 2020 was actually supposed to be, but a single controller model was slipped in. > Architecturally it should still suffice, but I'm working on getting my second controller for it because I want the speed and redundancy for my backup environment as well. Ahh good (that the 2040 is dual controller) - sorry, wasn't clear from your post. Personally I wouldn't spend the money on a second controller for the backup unit - it won't make a lick of difference to speed given it would just be acting as a "passive" backup controller (due to your low number of spindles), and given the speed / quality of NetApp's warranty I'd argue you don't need redundancy in what is already a backup SAN. Up to you though, of course! One other thought: have you bought the 2040 and the 2020 yet? If not, you might want to consider upgrading the 2020 to another 2040 as well. Reason being, the 2020 (and our 2050) won't run anything higher than Data ONTAP 7.x, as DOT 8 is 64-bit (only) and the architecture of the 2020 and 2050 is 32-bit. This will limit your 2040 to only being able to run DOT 7.x as well, because you cannot SnapMirror between a filer running DOT 8.x and a filer running DOT 7.x. Not the end of the world - DOT 7.x is fine - but you won't be able to take advantage of any new features / performance improvements / etc that are introduced in DOT 8 (and later versions). > As to I/O, it's actually pretty low. Between the VMWare capacity planner and some perfmon captures we determined that our current peak I\O is <2500 IOPS for ALL our physicals, and that's while running backups (no longer a concern once migrated to the new environment). It still sounds like you're still going to be pretty "close to the line" performance-wise - you have to remember that with your dual controller 2040 you're only going to have, at most, 8 active spindles (more on this below). Any idea what your peak IOPs will be without the backups? At the end of the day I guess it can't hurt to just "suck it and see", especially if you'll be gradually migrating load onto the SAN, but you might want to make sure you have the budget to add another shelf of disks if required (or possibly down-spec the 2040 to 300GB disks and buy another shelf up front). >I know it is technically best practice to have a dedicated system aggregate, but it wastes too much space on a very small system like your 2040 or our 2050. You only have 12 disks to work with, and with your current design you're losing four of them to parity and your system aggregate. >I'd scrap the system aggr and just create a single 12-disk RAID-DP aggr (or perhaps 11-disk + 1 spare), and use that to host both vol0 and your production vols. This is >what we did with our 2050 (although we also had to dedicate some disks to the second controller, which is unavoidable). > Honestly I didn't do the 2-disk system agg because of best practice, but more because I knew I had to give one controller a few disks so it seemed a logical way to accomplish that. I really wish I could just create one 12-spindle aggregate for speed, but my understanding is that I cannot. Right, I understand what you mean now - I think we just got our wires crossed on the terminology you used. You're basically correct: in a NetApp dual controller setup each disk is assigned to only one controller, and each controller needs a minimum of two disks assigned to it to hold that controller's system aggregate and root volume. This unfortunately means you "lose" a high percentage of your disks in a low-disk setup like the base 2020, 2040 and 2050. With your 2040, the highest performing and most space-efficient setup you can choose will be "active / passive": - Assign 10 disks to controller #1, and create one RAID-DP aggregate out of these. This controller / aggregate is your production data-store. Best practice says you should have 2 spares, but that is a lot (more) disks to "waste" - with the low disk count you probably want one spare at most, or possibly even none. - Assign the remaning two disks to controller #2, and create a two-disk RAID-4 aggregate out of these. This controller / aggregate will do nothing except sit around waiting to take-over from controller #1 in case of a problem - you could put some data on this two-disk RAID-4 aggregate, but it'll be slow and you don't have a lot of redundancy here. Given the lack of production data you would use no spares on this controller to save disks. Unfortunately, even with this setup you'll be using only 6 - 8 of your disks to store and serve actual data, depending on how many spares you use: 12 - 2 (for controller #2) - 2 (for RAID-DP parity on controller #1) - (0 to 2) (for spares on controller #1). This would be another reason to consider reducing your disks to 300GB each and buying another DS14MK4 or DS4243 upfr-ton. > From my research I was lead to believe that you needed iSCSI for SQL databases in order to properly quiece SQL for snapshots, or something to that effect. If SQL will run just peachy on my NFS store with everything else, I'd much prefer the simplicity. Is this when using SnapManager for SQL? If so, that is possible, I don't know anything about SnapManager for SQL. Otherwise, AFAIK quiescing works the same regardless of whether you're using NFS, iSCSI or FC. Additionally, you could just do what we do for an extra level of protection for our SQL databases, and continue to run SQL-based disk backups on a regular basis. Gives you better point-in-time SQL recovery options than VMware / NetApp snapshots too. > I suppose my best bet would be to ask the community just where I would find the guides\scripts\resources I need to plug a console cable into an unconfigured 2040\2020 controller and get it up and running. I've pulled countless design\best practices docs from the NOW site, but I've yet to locate the specific instructions for that task. Have you read the actual Data ONTAP manuals? It may not be in a "do this, then this, then this" format, but the manuals cover everything you need to know. In our case a NetApp engineer came out and spent about an hour with us to get the system up and running and do some very basic config - this service was included with the purchase of the unit. After that, it was just a matter of reading manuals and experimenting - once you get your head around the core concepts it's not too hard, and the FilerView GUI is decent for basic admin / setup tasks. Also, setup SSH / the BMC as soon as you can, so you can ditch the console cable 🙂 > Thanks again for the feedback, really helpful stuff. My pleasure, happy to help.
... View more
> Which HP switches are you using and how do you have that particular piece of the puzzle set up? We are using ProCurve 1800-24Gs for our dedicated storage network (a bit underpowered for the job, but they work), and ProCurve 2810-24Gs for our main network core. In both cases the two switches are configured as an independant but redundant pair to provide "failover" redundancy - we're not using MPIO. For each pair, we connect the switches together using a two-port LACP channel, connect a network port from each of our SAN controllers (and VMware hosts) to each of the two switches in the pair (so one connection from each controller / host to each switch), and configure Data ONTAP / ESXi / whatever to use fail-over redunancy between the two network ports. In DOT this involved creating a single-mode VIF out of the two network ports connected to each of the switches, and in VMware it's a function of assigning the two pNICs to the right vSwitches, configuring the role of each pNIC, etc. > I can find all sorts of documentation on setting up Cisco switches for MPIO and redundancy, but not having quite as good of luck finding proper information about my HP ProCurve 3500yls. A lot of the info out there relates to Cisco EtherChannel and stacked Cisco switches, which let you do things like connecting multiple NICs from one device to the switches, and having the multiple network links act as a single, cross-switch, aggregated and redundant link. This is nifty, but not required - you can achieve the same goal without EtherChannel / switched switches, you just have to let the devices (SAN controller, host, etc) handle the redundancy rather than the switch. Hope that helps, despite being a bit general - happy to answer more specific questions if you have any.
... View more
> Actually I had situations in the past when ndmpcopy didnt't work as expected & plainly missed some files / directories when moving root vol - in manifested e.g. in a form of FilerView refusing to work. > Interestingly enough volcopy always worked without the hitch & no one was able to give me a credible explanation why. Sounds like more votes in the "vol copy" column than the "ndmpcopy" column, so I might stick with with "vol copy". Thanks for the info.
... View more
> Probably no. Actually, ndmpcopy is definitely more flexible. OK, will stick with ndmpcopy then 🙂 > http://now.netapp.com/NOW/knowledge/docs/hardware/NetApp/diag/html/index.htm Thanks for the link, I have some reading to do... > New disks are prezeroed so to actually zero them you'll first need to "dirty" them. Thanks for the tip. As I mentioned in my OP the shelves / disks are actually used/refurb items, but I would assume that the vendor zeroed them and hence we will need to "re-dirty" them again as you suggest. Cheers, Matt
... View more
> 1. NetApp recommends allocating the whole shelf to a head. That is not absolute set in stone requirement. Mixing FC and SAS in one aggregate is supported. In any case – the first priority is sizing requirement – how many disks you application needs. We're a relatively small business, so no single application has huge requirements for either performance or size. The FAS2050 is providing storage to our VMware farm, which runs about 35 low-to-mid load VMs. This means we can afford to lay out our storage according to general best practice guidelines and then balance performance / size requirements of applications across the two controllers by simply moving VMs between them as needed. Given the above, if NetApp recommend allocating whole shelves to a head then it sounds like we are best off allocating both new shelves to controller #2, and all internal disks to controller #1. > 2. Yes, the procedure to move root volume is correct I’d use vol copy for whole root volume, but it is my personal preference. Thanks for the info - I'll look into vol copy for moving the root volume. Is there any particular reason vol copy is better than ndmpcopy (ndmpcopy is recommended in the NetApp docs). > 3. To be honest, I never actually understood how reallocation is supposed to interoperate with A-SIS; hopefully someone can chime in here. I've been researching this further since my OP. Based on what I've read so far, it sounds like reallocation will completely ignore deduplicated blocks (i.e. will not move them) unless you run reallocate with the "-p" switch. In that case, reallocate will physically redistribute a volume accross all disks in the aggregate, but won't change the logical layout of the volume. Likewise, I think that reallocate with the "-p" switch has no effect on snapshots, but reallocate without the "-p" switch will cause greatly increased snapshot space usage. I think what we to do is expand the aggregate, run "reallocate -p" against each volume to spread the data out accross the new disks, and then run "reallocate -A" against the aggregate to optimise free space. I'm certainly not 100% on all of this though, so if anyone else can clarify that would be much appreciated. > 4. I am not aware of any tests except built-in diagnostics. But it is offline so you lose redundancy for the whole test duration. Which diagnostics are you referring to here? Sorry, my NetApp knowledge is a bit patchy in places. > Other possibility would be create aggregate, destroy it and start spares zeroing. It is not real burn-in, but at least it does write loads all disks. We'll definitely zero all the new disks before we use them. I was thinking we could try running a disk burn-in / testing tool on a separate box and pointing it at the new disks over NFS, but I'm not sure if we'll be able to generate sufficient load to do proper testing this way. Hence, was hoping for something built into DOT. Thanks for the response / info, greatly appreciated. Cheers, Matt
... View more
Hi all, Currently we have a FAS2050C (running DOT 7.3.4) with 20 x internal 15k RPM 144GB SAS drives (part number X286) and no external shelves. To make best use of our limited spindles, we operate the two controllers as a quasi active / passive pair, with 17 disks assigned to controller 1, and the remaining 3 disks assigned to controller 2. All 17 disks assigned controller #1 are in a single RAID-DP RAID group / aggregate, and support our production workload. The 3 disks assigned controller #2 are in a single RAID-4 RAID group / aggregate, and do nothing except sitting there waiting to take over from controller #1 if required. We're about to add two DS14MK4 shelves to the filer, both fully populated with 14 x 15k RPM 144GB FC drives (part number X278A-R5), and I'm hoping for some assistance with a few questions: 1) Once the extra shelves are installed we want to assign more disks to the second controller and start using it "properly", rather than just having it available as a "passive" partner. This decision was taken because we run into a CPU bottlekneck far more often than we run into a disk I/O bottlekneck, so we figure that the extra disks will be best used if the load is spread across both controllers. This leads to my first question: are there any guidelines / best practices about how to best allocate disks to each controller, both in a general sense and also taking account the two (slightly) different disk formats and different disk interfaces? i.e. put all 20 internal disks on one controller and all 28 external disks on the other, or split then exactly down the middle (half of each shelf + half of internal per controller), etc? 2) If the "best practice" is to put all internal disks on one controller and all external disks on the other (which I'm guessing might be the case), this is going to require us to move the root volume for controller #2 onto the new disks. From reading NetApp documentation I believe the process to do this is: - attach new shelves, assign new disks to controller #2, create new RAID groups / aggregate(s) - create a new root volume on the new disks - use ndmpcopy to copy /etc from the old root volume to the new root volume - set the root volume option on the new root volume - reboot Is that all that needs to happen / is that all correct? And once I've done this, I should then be able to destroy the old root volume / underlying aggregate / 3-disk RAID-4 RAID group, and reassign these three disks to controller #1 to add to its RAID group / aggregate? 3) I understand that after adding more disks to an existing aggregate we should manually run a WAFL reallocation against any volumes contained in that aggregate. Given that we use dedupe and extensive snapshots, will this reallocation interfere with existing deduped data and/or snapshots in any way? i.e. will it "un-dedupe" our volumes, or cause snapshot sizes to blow out, as part of the process? 4) The shelves / disks that we are adding are used - any suggestions for methods / tools to run appropriate burn-in testing before we put them into production? 5) Any other gotchas I should be aware of during the process of adding these extra shelves / disks? Thanks all, appreciate any information / advice. Cheers, Matt
... View more
> Good morning all, this is my first post here, and I'm hoping to seek a bit of guidance on my design. Minus the DR stuff, your design is remarkably similar to one I rolled out at my company two years ago. 🙂 The specific models are different - e.g. we are using IBM servers, HP switches and a FAS2050 - but the "entry level fully redundant" concept, implementation of this idea, and selection of hardware to do it is almost identical. I'm far from a NetApp expert, but based on my experience over the past few years, a couple of comments: > I have, in my data center right now: > 1: FAS2040 loaded with 450gb 15k SAS disks (Windows software bundle if I remember correctly) Is your FAS2040 single or dual controller? If the former, that's one big weak spot in your design / goal of having redundant everything - single point of failure at the SAN. Also, any idea how much disk I/O your VMware farm is going to produce? You don't get a lot of spindles in a base 2040, so you may run into performance issues. I'm not saying you will have issues - our 3 server / 30 VM farm runs just fine off 17 spindles in the 2050 - but you may, depending on the I/O that you generate. > For the network, I’ll be using 2 of the switches stacked for production\management traffic, and two stacked, set to jumbo frames, and used as an isolated storage network. I’ve drafted a design for the storage network based on the “VMWare on NetApp storage Best Practices***” guide, and thankfully I have some actual CC** types that are going to handle the config of those. Had a quick glance and the network design looks good to me - and again, pretty closely mimics what we implemented. > 2 Disks set to RAID4 to be used as the System Aggregate > 10 Remaining disks set to RAID-DP as the Production-Storage Aggregate (all NFS FlexVOLs Deduped) I know it is technically best practice to have a dedicated system aggregate, but it wastes too much space on a very small system like your 2040 or our 2050. You only have 12 disks to work with, and with your current design you're losing four of them to parity and your system aggregate. I'd scrap the system aggr and just create a single 12-disk RAID-DP aggr (or perhaps 11-disk + 1 spare), and use that to host both vol0 and your production vols. This is what we did with our 2050 (although we also had to dedicate some disks to the second controller, which is unavoidable). > NFS FlexVol for Production VM Datastore > NFS FlexVol for Development VM Datastore > NFS FlexVol for VM Pagefiles (No Snapshots) > NFS FlexVol for VSWAP files (No Snapshots) Very similar to what we did - although instead of using Prod / Dev datastores, we created our datastores based around the type of data they store (OS, temp/transient, apps, data, etc). Good choice going with NFS too, it is very flexible and simple. > iSCSI LUN for SQL DBs Any particular reason you're using iSCSI for SQL DBs and NFS for the rest? We just used NFS for the lot. > I’ve managed and been on the deployment of a few older FAS270 and R200 devices (~5-6 years ago), and I have a decent understanding of the concepts and admin of NetApp storage (when everything but filerview seemed to be CLI), but I haven’t actually set up or administered any modern NetApp device. > I would LOVE to set this infrastructure up myself, and I’m hoping that between the community and documentation on the NOW site I can do so. Everyone I’m talking to insists that you need to drop 10-20k on consulting to bring a few NetApp FAS' up, I’m hoping that isn’t the case. ... > So, am I insane for attempting to create the above myself? Or has everything on the newer FAS' really become wizard and menu driven enough that I should be able to tackle it with a bit of guidance? The attached sketch is a bit simplistic but it gives a rough idea of how everything will be connected. Insane? No 🙂 But whether it's a good idea of not does depend on how quickly the new infrastructure is needed, how much time you have, how interested your are, how quickly you pick up new concepts, your current level of experience, etc. As I mentioned at the start of the post, I researched, designed and implemented a very similar setup on my own, from scratch, without any direct help or prior experience with VMware, NetApp, networking (beyond the basics), etc. This infrastructure has now been operating for ~2 years without a single issue, so it certainly is possible to do it on your own. On the flipside, however, it was a very time consuming process, and I don't think it would have been possible if I didn't find the technology I was learning / working with fascinating. I spent months reading and learning before I even ordered the first part, and I had the luxury of a loose implementation timeframe and the available time for me to learn as I went along. The whole process was a great experience - I found it very interesting / challenging, learnt a massive amount, etc - so if you feel that you're up to the challenge and have the time, inclination and interest, I say give it a go. Good luck! Cheers, Matt
... View more