Re: Netapp Head swap iSCSI configuration

dorseyfoga · ‎2011-05-05

I recently did a head swap between a FAS2050 and a FAS3210 without too much of a problem. I used the root volume from the FAS2050 and everything came up fine with the exception of the iSCSI configuration. All of the LUN's came up, but none of the mappings came up and I had to re-create it manually. This was not a huge deal for this swap since it had a total of two connections, but the next two I do have somewhere in the area of about 20 mappings each.

Is there any way I can avoid having to re-do this all manually or did I miss something with the iSCSI setup when I did my first migration so that the information didn't copy over? The same network address is being used, so I'm hoping that most of the initiators can just pick up on the setup.

Thanks.

shaunjurr · ‎2011-05-05

Hi,

I'm assuming that with "mappings" you are referring to some part of the igroup and lun mapping configuration. Was it just the lun maps? What ONTap version?

The only migration problems that I have run onto are problems with the changing iscsi nodename with headswaps (you can set it in one of the priv modes, iirc) and that on certain upgrades, LUN serial numbers have changed. Having a recent ASUP mail will save you in a lot of the unexpected situations. Scripting known changes will save you time.

Good luck.

dorseyfoga · ‎2011-05-05

The version is 7.3.5.1. I think I answered the rest of the questions in my other reply.

scottgelb · ‎2011-05-05

If you keep the same root volume after controller swap, all of the configuration of luns, igroups, lun mappings to igroups, ips and iscsi nodenames should move over with no issue. Sometimes you have to modify /etc/rc and /etc/hosts if port names change (e0c to e4a after swap for example).

Did you do a complete head swap or did you use the existing root volume on disks that shipped with the new controller?

dorseyfoga · ‎2011-05-05

We went from a FAS2050 that had 5 shelves and added them to a FAS3210 that had two shelves with it that were pretty much blank. We used the root volume from the FAS2050 and initialized it once it was connected to the new head and all of the configurations came with it with the exception of the iSCSI setup. (Snapmirror, network, etc were all fine.)

When I checked the LUN's it listed no mappings at all and had no initiators, igroups or anything listed. I had to re-create them.

aborzenkov · ‎2011-05-05

We used the root volume from the FAS2050 and initialized it once it was connected to the new head

What exactly do you mean under "initialized it"?

dorseyfoga · ‎2011-05-06

We onlined the aggregate by doing aggr online aggr0 and then vol options vol0 root and then rebooted. I renamed the aggr and vol0 on the FAS3210 head before I shut everything down. When it came back up, vol0 that was used on the 2050 was the one in use.

aborzenkov · ‎2011-05-06

Where and how igroups and mappings are stored still remains a great secret, apparently protected NetApp above everything. But one thing for sure – when you online volume (which is implicitly done when you online aggregate) NetApp rescans storage for LUNs. If your LUNs were located on the same aggregate it could have happened that any igroup mapping was removed from LUNs (assuming they are stored somewhere with a LUN).

I really wish someone from NetApp would finally explain in simple words where igroups and mappings are stored.

In general, what you have done is the wrong way to do upgrade. You normally just connect old shelves to new head, re-assign disks and come up with exactly the same config as before. Then – if I have reasons to think newly delivered system has own root volume – I connect new shelves online and simply destroy new root volume/aggregate if it appears.

dorseyfoga · ‎2011-05-09

The new NetApp was up and runing for a week or so due to us having internal storage on the head and it needing to be snapmirrored off. So it had it's own setup and everything. I created vol0 on a different aggr then where the snapmirror destinations were. So after moving the shelves over, changing the disk assignments and then setting the vol0 from the old head as the main vol0, everything came up fine with the exception of the lun's. I did it by using the directions I found at this place:

http://now.netapp.com/NOW/knowledge/docs/hardware/NetApp/cs_migration/FAS2xxx/FAS2040_internal_storage_to_FAS32xx_single-controller/GUID-55D5319A-2319...

jmmorrell · ‎2012-03-10

I have run into this issue on almost every controller upgrade I've performed. I've talked to NetApp support & PS engineers, they've both told me they've never seen this happen. I'm trying to determine where this information is stored so I can back it up. The only way I've been able to save the config is doing a "igroup show" & "lun show -m" and then manually recreate the igroups & maps.

I always use the original root volume from the older controller.

The only thing I can think of is the time at which I online the "foreign" aggregate which resided on the older disks.

1. Cable all disk shelves

2. re-assign disks from old controller to new controller

3. Boot into ONTAP

4. Online the aggr, which onlines the volumes. At this point I see a bunch of messages similar to the one below:

Sat Mar 10 16:43:21 EST [netapp01a: lun.newLocation.offline:warning]: LUN /vol/data/lun3 has been taken offline to prevent map conflicts after a copy or move operation.

Do I need to online the aggr's in maintenance mode?

aborzenkov · ‎2012-03-10

I've talked to NetApp support & PS engineers, they've both told me they've never seen this happen.

I have performed a couple of head swaps and I've too never seen this happen. So you must be doing something differently. E.g. dorseyfoga booted with new root first which apparently was the reason NetApp had lost existing mappings, as they are kept in root volume. You seem to have offlined aggregates with LUN before doing head swap. So it is quite possible that NetApp removed existing mappings because no LUN was found when you booted for the first time after head swap.

Why did you need to offline anything in the first place? Doing head swap is really simple. Just connect old shelves, reassign disks in maintenance mode and boot.

jmmorrell · ‎2012-03-10

When you assign the disks in maintenance mode, the aggregates are imported as “foreign aggregates” & offlined. So it sounds like you need to online the aggr in maintenance mode before booting into ONTAP. Can you change the root volume in maintenance mode?

Also do I need to offline the existing root volume on the new controller? All of my migrations typically have new shelves & the systems are powered up initially to burn them in. It is frustrating that NetApp does not document the steps to prevent this.

Should you not have the “new” disks connected until you assign the old disk and use the old root volume?

Thanks,

Jim

aborzenkov · ‎2012-03-10

When you assign the disks in maintenance mode, the aggregates are imported as “foreign aggregates” & offlined.

No. I have never seen this happened. You must be doing something entirley differently.

Please describe step by step procedure you are following. From the very beginning. Every single command invocation.

All of my migrations typically have new shelves & the systems are powered up initially to burn them in.

This starts to sound like previously described issue. So you have new root first. Then you add old disks. This effectively means LUN mapping present on old disks is lost.

The correct precedure is to do pure head swap first using old disks and then add new disks online. You will get foreign vol0 which you now can happily destroy. This will preserve complete configuration.

OK, just to avoid misinterpretation if someone reads this out of context. Things related to hardware - LUN serials (tied to sysid), HBA WWNs will or may change. But software configuration will not change.

jmmorrell · ‎2012-03-12

My steps got truncated by when I replied by email. But I have found the answer to this issue. The below KB as an explanation as to why this occcurs & how to resolve.

https://kb.netapp.com/support/index?page=content&id=1013497

Description

Storage controller motherboard replacements and head upgrades, when performed
improperly can result in loss of the controller’s SAN identity information
including Fibre Channel WWN, WWPN and iSCSI IQN. It can also result in existing
LUN serial numbers being changed or the loss of LUN to initiator group mappings.

Data ONTAP 7G and Data ONTAP 7-Mode store SAN identity and LUN mapping data
in the controller’s root volume. Therefore, the root volume and aggregate must
be preserved during a storage controller motherboard replacement or head
upgrade.

LUN serial numbers are automatically changed in the event the aggregate
volumes are reassigned from one storage controller to another. The only
exception to this is when Data ONTAP determines that a motherboard replacement
has occurred. The motherboard replacement detection occurs when all aggregates
attached to the storage controller are determined to come from another storage
controller. If foreign aggregates are added to a storage controller with
existing aggregates and owned disks then the motherboard replacement detection
will fail and any LUNs with in FlexVols on the foreign aggregates will receive
new serial numbers.

aborzenkov · ‎2012-09-20

I think this message I received after doing "aggr XXX options root" in maintenance mode pretty much explains what happened

*> aggr options system root

aggr options: Could not copy the FCP, ISCSI and igroup configuration

information from the previous root 'aggr0' to the new

root 'system'. Use the fcp, iscsi, igroup, lun and vfiler

commands to verify the configuration information and recreate it if

needed. Contact NetApp Global Services for further assistance.

Aggregate 'system' will become root at the next boot.

*>

So when you issued "vol ... options root" under running system, it overwrote definitions on old root.

I wish there were easy way to save and restore block-protocol related settings, or at least that how they are stored were documented in more details.