How to migrate 32bit volume with qtree and non qtree data to 64 bit volume

[JLundfelt]

Hi

I have a 11TB CIFS based NTFS volume with 4 qtree's on 7.3.7 controller, that I need to migrate to another controller, the problem I am having is that

if I use QSM-

Invoke-NaSnapmirrorInitialize -Source 172.16.1.1:/vol/Groups/- -Destination irv-gdc-san1a:/vol/Groups/qtree

I only get the base contents of the volume, and not the other qtree's that are co-mingled with folders in the base of the cifs share local path (/vol/Groups)

if I use VSM-

The volume would be replicated with all data / qtree's. but would remain 32bit.

Part of the issue I have is that this is over 10TB / 10 million files. I already have the QSM relationship for the base contents of the volume. If I setup additional QSM's, they would be adjacent to the base contents, and a single cifs share could not access. As part of the cutover plan, I am going to be aliasing the IP address since we have both Windows and Unix clients accessing this data from god knows where. Anyone have any suggestions? I am trying to make this a single cutover event, where the outage is less than an hour, and from previous expireance, if I move the contents from the base QSM  (irv-gdc-san1a:/vol/Groups/qtree --> irv-gdc-san1a:/vol/Groups) it could take hours, and I wouldn't be able to maintain permissions unless I used robocopy with the /MIR and /SEC switches.

Any suggestions? TIA!

-jon

[richard_payne]

Ouch, that sounds like one busy toaster. If it's not CPU it could easily be disks - sysstat -x 1 will give you some idea (though the Disk util is the busiest disk not an average).

If you're running dedup you may also want to look at schedules. I know Netapp calls it a background process, but it still drives I/O on the disks and we sometimes see an improvement in filer performance by stopping the scans until the load dies down.

--rdp

[JLundfelt]

So no dedupe, here are the metrics-

IRV-SAN-NA1*> vol status Groups
Volume State Status Options
Groups online raid_dp, flex maxdirsize=20971, guarantee=none,
Containing aggregate: 'aggr3'

IRV-SAN-NA1*> df -g Groups
Filesystem total used avail capacity Mounted on
/vol/Groups/ 8243GB 6986GB 1256GB 85% /vol/Groups/
/vol/Groups/.snapshot 3532GB 3342GB 189GB 95% /vol/Groups/.snapshot

IRV-SAN-NA1*> wafl scan status Groups
Volume Groups:
Scan id Type of scan progress
28 active bitmap rearrangement fbn 68485 of 105086 w/ max_chain_len 5
1151663 container block reclamation block 6337 of 105087 (fbn 43637)
1151664 block ownership calculation block 12495 of 105086

Everything was working after the initial baseline-

dst Mon Jul 20 16:29:57 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Initialize)
dst Mon Jul 20 16:30:17 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Thu Jul 23 19:00:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (10752351408 KB)

and the first differential-

dst Fri Jul 24 00:10:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 00:10:20 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 13:55:23 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (44940072 KB)

and then it was running all weekend every 10 minutes-

dst Fri Jul 24 13:56:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 13:56:21 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 15:53:05 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (2257532 KB)
dst Fri Jul 24 16:00:21 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 16:07:52 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (2129884 KB)
dst Fri Jul 24 16:13:37 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 16:15:07 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (81048 KB)
dst Fri Jul 24 16:20:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 16:20:22 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 16:21:24 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (39652 KB)
dst Fri Jul 24 16:30:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 16:30:19 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 16:31:21 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (84688 KB)
dst Fri Jul 24 16:40:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 16:40:15 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 16:41:38 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (57788 KB)
dst Fri Jul 24 16:50:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 16:50:19 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 16:53:42 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (767740 KB)
dst Fri Jul 24 17:00:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 17:00:17 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 17:01:40 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (70368 KB)
dst Fri Jul 24 17:10:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 17:10:15 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 17:10:58 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (45224 KB)
dst Fri Jul 24 17:20:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 17:20:13 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 17:20:40 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (27008 KB)
dst Fri Jul 24 17:30:01 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 17:30:13 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 17:30:40 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (50456 KB)
dst Fri Jul 24 17:40:01 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 17:40:16 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 17:40:37 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (20816 KB)
dst Fri Jul 24 17:50:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 17:50:16 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 17:50:43 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (1952 KB)
dst Fri Jul 24 18:00:02 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 18:00:15 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 18:00:29 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (1344 KB)
dst Fri Jul 24 18:10:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 18:10:22 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 18:10:35 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (644 KB)
dst Fri Jul 24 18:20:01 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 18:20:15 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 18:20:27 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (3504 KB)
dst Fri Jul 24 18:30:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)

Until it started having larger deltas / transfer times-

dst Fri Jul 24 18:30:09 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Fri Jul 24 23:47:12 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (31132 KB)
dst Fri Jul 24 23:48:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Fri Jul 24 23:48:17 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Sat Jul 25 06:51:44 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (277864 KB)
dst Sat Jul 25 06:52:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Sat Jul 25 06:52:22 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Sat Jul 25 10:14:58 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (267992 KB)
dst Sat Jul 25 10:15:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Sat Jul 25 10:15:18 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Sat Jul 25 15:10:28 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (197624 KB)
dst Sat Jul 25 15:11:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled) 

dst Sun Jul 26 20:10:14 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Sun Jul 26 20:10:34 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (424 KB)
dst Sun Jul 26 20:20:01 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Sun Jul 26 20:20:15 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Sun Jul 26 20:20:29 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (1600 KB)
dst Sun Jul 26 20:30:02 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Sun Jul 26 20:30:19 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Sun Jul 26 20:30:33 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (6424 KB)
dst Sun Jul 26 20:40:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Sun Jul 26 20:40:08 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Sun Jul 26 20:40:20 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (5776 KB)
dst Sun Jul 26 20:50:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Sun Jul 26 20:50:09 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Mon Jul 27 00:59:01 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (13704 KB)
dst Mon Jul 27 01:00:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Mon Jul 27 01:00:19 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Mon Jul 27 06:13:28 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups End (318104 KB)

dst Mon Jul 27 06:14:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Request (Scheduled)
dst Mon Jul 27 06:14:14 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Start
dst Mon Jul 27 11:01:00 PDT 172.16.1.1:Groups irv-gdc-san1a:Groups Abort (replication transfer failed to complete)

Cannot do a snap list, snap delta or delete snaps on this volume, the first two time-out the latter produces-

IRV-SAN-NA1*> snap delete -V Groups '2015-07-28 09_00_11 hourly_IRV-SAN-NA1_Groups'
Snapshot 2015-07-28 09_00_11 hourly_IRV-SAN-NA1_Groups is busy because of snap delta

[JLundfelt]

Just a quick update. I was ab;e to resove the issue after opening a NetApp case, and getting three layers deep with tech support. They gathered a bunch of data per usual, but what ended up finally fixing the issue, where the source volume was no longer 'busy' and giving me the snap delta, in use warning when trying to delete a snapshot, and 5-6 hours for snapmirror to even start transferring data, or fail, was when I stopped the snapmirror service on the target controller. As soon as I did that, the source created a new snapvault dependent snapshot, and removed the dependencies on the old snapshots, and cleared the remaining from being 'busy'

Thanks for everyones help.  Have a good weekend  😃

-Jon