topic Data inconsistency in using robocopy for CIFS migration in Network and Storage Protocols

Data inconsistency in using robocopy for CIFS migration

kalki1 — Sat, 12 Apr 2025 16:41:44 GMT

Hi, I have used robocopy for CIFS data migration from old ONTAP (9.1P2) to new (9.11.1P7). After initial copy and incremental, I see the volume on the new system is taking almost 3.5 TB of extra space. Also the number of files in source volume is more than that in destination volume. There are no errors in robocopy and the summary shown everything copied. I have done random checks on folders on both sides they look good. This is really strange!

----------------------------------------------------------- SOURCE USAGE ----------------------------------------------------------------

source::> vol show-space cifs_data

Vserver : source_cifs
Volume : cifs_data

Feature Used Used%
-------------------------------- ---------- ------
User Data 4.24TB 28%
Filesystem Metadata 5.68GB 0%
Inodes 2.05GB 0%
Snapshot Reserve 768GB 5%
Deduplication 34.97GB 0%
Snapshot Spill 350.4GB 2%
Performance Metadata 580.3MB 0%

Total Used 5.37TB 36%

Total Physical Used 5.19TB 35%

source::> vol show -vserver source_cifs -volume cifs_data -fields files
vserver volume files
-------- --------- --------
source_cifs cifs_data 31876696

------------------------------------------------- DESTINATION USAGE ----------------------------------------

dest::> vol show-space -volume cifs_data

Vserver : dest-cifs
Volume : cifs_data

Feature Used Used%
-------------------------------- ---------- ------
User Data 7.63TB 51%
Filesystem Metadata 3.01GB 0%
Inodes 1.50GB 0%
Snapshot Reserve 768GB 5%
Performance Metadata 42.77GB 0%

Total Used 8.42TB 56%

Total Physical Used 8.23TB 55%

dest::> vol show -vserver dest-cifs -volume cifs_data -fields files
vserver volume files
--------------- --------- --------
dest-cifs cifs_data 21251126

---------------------------------------------------- ROBOCOPY command used --------------------------------------------------

robocopy /e /mir /copyall /r:0 /w:0 /ETA /mt:32 /sec /secfix /dcopy:t \\source_cifs\cifs_data$\ \\dest_cifs\cifs_data$\

Is storage showing wrong usage ? or is there any bug or issue with this ?
Any help is appreciated. Thanks!

Re: Data inconsistency in using robocopy for CIFS migration

Sanaman — Mon, 14 Apr 2025 04:48:41 GMT

If copying/migration from NetApp to NetApp I have always used snapmirror, it is quick as it utilizes inter-cluster network.

Re: Data inconsistency in using robocopy for CIFS migration

kalki1 — Mon, 14 Apr 2025 05:08:25 GMT

snapmirror cannot be done as the ONTAP versions are not compatible and the source storage is old enough that it cannot be upgraded from 9.1.

Re: Data inconsistency in using robocopy for CIFS migration

Sanaman — Tue, 15 Apr 2025 05:02:36 GMT

Ok, didn't know source OnTap is that old. Have you tried XCP?

NetApp XCP Overview

NetApp XCP is a client-based software tool designed for fast and reliable data migrations to and from NetApp systems, as well as for file analytics. It supports both any-to-NetApp and NetApp-to-NetApp migrations, making it versatile for various data management needs.

Key Features

High-Performance Migrations: Utilizes parallel streams of asynchronous I/O requests to significantly speed up the transfer of large volumes of files, even in the millions or billions.
File Analytics: Provides insights into file systems by scanning file shares and analyzing data, applicable to both NetApp and non-NetApp systems.
Customizable Reporting: Generates detailed reports in various formats (CSV, HTML, plain text) based on file attributes like access time, owner, and size.
Incremental Updates: Supports syncing for incremental updates to ensure data consistency.

Re: Data inconsistency in using robocopy for CIFS migration

kalki1 — Tue, 15 Apr 2025 13:12:01 GMT

I am aware about XCP. we chose robocopy for this migration. Unless, we know for sure the discrepancy is because of the migration tool, we do not want to change it yet.