topic Re: Fileserver Migration with robocopy in ONTAP Discussions

Fileserver Migration with robocopy

hzickner — Fri, 22 Apr 2016 21:14:22 GMT

I want to migrate data from am Windows2012 server to a NetApp CIFS share.

robocopy \\winserver\source \\vserver\destination /MIR /COPYALL /XO /FFT

It does the initial copy without errors. In the second run I ant to copy only changed files, but robcopy incorrectly recognizes everythig as changed and does the whole job again. Between two Windows servers robocopy works as expected. Only modified files are transferred in the second run.

Anybody else with this problem?

Holger Zickner

Re: Fileserver Migration with robocopy

Robert_EE — Sat, 23 Apr 2016 02:54:49 GMT

I read an article on an unrelated issue that says for copying updated files and changed permissions with robocopy, the following syntax should be used.

> ROBOCOPY <source> <target> /MIR /SEC /SECFIX

https://blogs.technet.microsoft.com/filecab/2008/07/31/robocopy-mir-switch-mirroring-file-permissions/

I am not a robocopy expert, so I cannot confirm this will fix your problem.

Thanks,

Re: Fileserver Migration with robocopy

notermanjeroen — Sat, 23 Apr 2016 19:07:04 GMT

Hi,

For the initial sync we use:

robocopy \\source_share\folder \\destination_share\folder /E /V /ZB /R:1 /W:1 /NP /COPYALL /LOG:c:\robocopy.log

Explanation:

/E - copy subdirectories, including Empty ones.
/V - produce Verbose output, showing skipped files.
/ZB - use restartable mode; if access denied use Backup mode.
/R:1 - number of Retries on failed copies
/W:1 - Wait time between retries: default is 30 seconds
/NP - No Progress - don't display percentage copied.
/COPYALL COPY ALL file info (equivalent to /COPY: DATSOU)
/LOG - LOG location

After the initial sync you can use the following command to copy only the changes:

robocopy \\source_share\folder \\destination_share\folder /MIR /V /ZB /R:5 /W:5 /NP /COPYALL /LOG:c:\robocopy.log

Explanation:

/MIR - MIRror a directory tree (equivalent to /E plus /PURGE).

Kind regards,

Re: Fileserver Migration with robocopy

Robert_EE — Sat, 23 Apr 2016 23:06:38 GMT

Yes, so my assertion is that the /copyall switch is what the problem may be.

Re: Fileserver Migration with robocopy

ChristianSmet — Mon, 31 Oct 2016 12:54:05 GMT

Hi,

I really would like to see if somebody had found a solution.

After the copy baseline towards netapp cdot 8.3.1 with robocopy the update "marks" every file as changed.

This creates a huge logfile and extends the robocopy update job.

Doing excactly the same copy to a windows destination is not showing existing files not as changed.

br,

Christian Smet

Re: Fileserver Migration with robocopy

Premkumar — Tue, 01 Nov 2016 01:17:58 GMT

Hello,

I have tried with the below option with full permission on source and destination and its works. I copied near by 6 TB of data without error. Can you try this below option instead of using other options?

> ROBOCOPY <source> <target> /MIR /SEC /SECFIX

/MIR will replicate data and security (as /SEC is specified) for changed files, and /SECFIX will update just the security for unchanged files. Add /V to the command line if you want to see which files are having their security “fixed” without having their data copied.

Thanks,

Re: Fileserver Migration with robocopy

ChristianSmet — Wed, 02 Nov 2016 06:32:27 GMT

Hi,

thanks for the reply.

We are using below options.

Options : *.* /V /S /E /DCOPY:DA /COPY:DATS /SECFIX /PURGE /MIR /R:0 /W:0

The problem is that the same options act different on a Windows destination compared with our current Netapp destination.

Above options tread every file as changed. So there are no files skipped on the incremental.

When we do the same on a windows destination it skips the files. Only some dirs.

Below incremental on a Netapp destination for a 5TB volume that has already had a first baseline.

So it took almost 100hrs just to check 5TB on changes?

------------------------------------------------------------------------------

               Total    Copied   Skipped Mismatch    FAILED    Extras
    Dirs :    571018      7366        14         0         0      3822
   Files :   4954222   4954204         0         0        18     36030
   Bytes :   5.770 t   5.770 t         0         0     5.5 k 83.337 g
   Times : 97:32:00 83:04:45                       0:00:00 14:27:15

   Speed :            21212801 Bytes/sec.
   Speed :            1213.806 MegaBytes/min.
   Ended : 30 October 2016 13:41:44

br,

Christian Smet

Re: Fileserver Migration with robocopy

ChristianSmet — Mon, 07 Nov 2016 10:34:37 GMT

Found and solved....

http://www.noelis.be/?p=173

Re: Fileserver Migration with robocopy

BRECHTMON — Fri, 17 Mar 2017 09:28:41 GMT

We're having a similar issue where robocopy marks all files as "modified" on a subsequent run.

/fft parameter makes no difference. File size, modified time etc are all equal when checked in windows explorer

however, using robocopy on server 2008 instead of server 2012 does correctly recognise files as unmodified, resulting in a much quicker subsequent run.

I don't know if the settings you mention in your blog post are active or would make a difference.

Re: Fileserver Migration with robocopy

ChristianSmet — Fri, 17 Mar 2017 09:40:51 GMT

Hi BRECHTMON,

We have already finished everything by the procedure provided.

Another workaround you can try is... the parameter /NOOFFLOAD with ROBOCOPY.

I don't know why I didn't see that before. It's worth a try 🙂

br,

Christian Smet

Re: Fileserver Migration with robocopy

CHMOELLER — Fri, 07 Apr 2017 14:29:01 GMT

Hey guys

We're having this very issue right now, same behavior - Robocopy marks every file as modified when we transfer to a NetApp System (or between them). It's all good when only Winodows is involved.

Via robocopy /debug and some subsequent testing I came to realize it's something about permissions. When I do /COPY:DAT it's just fine, when I do /COPY:DATS (to include ACLs) it all goes wrong. I don't think the files are actually copied (it's too fast for that) but I guess Robocopy tries to fix ACLs or something - and fails, because they are of course identical. (The ACLs and SDDLs look the same on source and destination). Still it's much slower than when it dosn't really do anything.

BRECHTMON is pretty much on the money though - I can confirm that this does NOT happen with a 2008 R2 server and its robocopy.

I cannot be related to ODX because in our case it happens with 7-mode systems also (with no cDot involved).

Does anyone have more info on this?

Re: Fileserver Migration with robocopy

Netapp2017 — Tue, 02 May 2017 15:56:37 GMT

Also having this exact issue. If I robocopy to Windows, each directory without changes it only displays the directory name and local file count. If I robocopy to the Netapp, it lists every file but does not say "Newer or New File" next to the unchanged files.

I ran a test with a 3GB ISO file which confirms suspicions that the files are not getting copied each subsequent time. The problem is that it doesn't seem like they're getting skipped either which is increasing the total time it takes to complete.

I have already tried the ODX disable fix but that didn't work.

Anyone other insight?

Re: Fileserver Migration with robocopy

CHMOELLER — Fri, 02 Jun 2017 11:25:55 GMT

Hey guys,

well I just did some more tests and found out something. I'm not sure whose "fault" it really is - but I'd say there is a good chance it might be on NetApp's side.

Clearly something changed in Robocopy. Which is easy to prove since the copy works just fine with older versions of it but not with more recent ones. Now the base-problem however might be on NetApp's side - who seem not to imitate an actual Windows Fileserver quite closely enough. Which again is easilsy proven - because the problem does not occur when copying between (or to) Windows fileservers. So maybe newer Robocopy version rely on a feature not present or broken in NetApp's CIFS implementation?

So here's what I did. I copied a simple folder with a few PDFs with Robocopy. I copied from a cDOT Filer to a 7mode filer here, but copying the same folder from a "real" Windows fileshare shows the exact same behavior.

I did an initial robocopy /MIR /COPYALL - which as expected creates a duplicate ot the folder and files. Every subsequent run of /MIR /COPYALL shows the problem we have - all files are marked as "Modified" although they are not. However as established, the files do not get copied.

So another run with the /debug parameter:

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robustes Dateikopieren fr Windows                              
-------------------------------------------------------------------------------
  Gestartet: Freitag, 2. Juni 2017 12:18:40
OSVersion : 6.3 (9600) 25800306
   Quelle : \\?\UNC\SOURCE-FILER\testfolder\
     Root : "\\SOURCE-FILER\fpm\"
  VolName : "fpm"
   Serial : 80000415
  MaxName : 255
   FSFlag : 000400CF
   FSType : "NTFS"
   IsNTFS : 1
     Ziel : \\?\UNC\TARGET-FILER\testfolder\
     Root : "\\TARGET-FILER\data\"
  VolName : ""
   Serial : 3B02D496
  MaxName : 255
   FSFlag : 0004004F
   FSType : "NTFS"
   IsNTFS : 1
    Fudge : 0
    Dateien : *.*
	    
  Optionen: *.* /S /E /COPYALL /PURGE /MIR /DEBUG /R:1000000 /W:30 
------------------------------------------------------------------------------
\\?\UNC\SOURCE-FILER\testfolder\tr-3197_TEchnicalOverviewofSnapDrive.pdf: Create: 1cd4ee6369beb3e LastA: 1d2db899501cfd8 LastWr: 1cd49f3a4f22f93 Change: 1d093933059611c
\\?\UNC\SOURCE-FILER\testfolder\tr-3483_ThinProvisioningInANetAppSANorIPSANEnterpriseEnvironment.pdf: Create: 1cd4ee636a428c6 LastA: 1d2db8995068ac8 LastWr: 1cd486bafe9d4dd Change: 1d093933059af46
\\?\UNC\SOURCE-FILER\testfolder\tr-3563_ThinProvisioningIncreasesStorageUtilizationWithOnDemandUtilization.pdf: Create: 1cd4ee636a895aa LastA: 1d2db89953c1968 LastWr: 1cd49f39598b963 Change: 1d093933059fd5c
\\?\UNC\SOURCE-FILER\testfolder\tr-3702_BestPractivesForMicrosoftVirtualizationAndSnapManagerHyperV.pdf: Create: 1cd4ee636b19682 LastA: 1d2db8995601bf6 LastWr: 1cd4a3269d656a4 Change: 1d09393305a4b86
\\?\UNC\SOURCE-FILER\testfolder\tr-3828_SnapDrive62WindowsBestPractices.pdf: Create: 1cd4ee636ba7040 LastA: 1d2db89956d3b74 LastWr: 1cd49f3b27efb4b Change: 1d09393305a999c
\\?\UNC\SOURCE-FILER\testfolder\tr-3965_ThinProvisioningDeploymentAndImplementationGuideDataOntap87-mode.pdf: Create: 1cd4ee636c37136 LastA: 1d2db899590efc4 LastWr: 1cd496422779fad Change: 1d09393305ae7b2
\\?\UNC\SOURCE-FILER\testfolder\Thumbs.db: Create: 1ce4bf5d00dcf28 LastA: 1d2db8994f3c65e LastWr: 1ce4bf5dbc19250 Change: 1d0b3fd4d24fd44
\\?\UNC\TARGET-FILER\testfolder\Thumbs.db: Create: 1ce4bf5d00dcf28 LastA: 1d2db89890de0ea LastWr: 1ce4bf5dbc19250 Change: 1d2db8994f02cce
\\?\UNC\TARGET-FILER\testfolder\tr-3197_TEchnicalOverviewofSnapDrive.pdf: Create: 1cd4ee6369beb3e LastA: 1d2db898929cd32 LastWr: 1cd49f3a4f22f93 Change: 1d2db8994fd4c7e
\\?\UNC\TARGET-FILER\testfolder\tr-3483_ThinProvisioningInANetAppSANorIPSANEnterpriseEnvironment.pdf: Create: 1cd4ee636a428c6 LastA: 1d2db898931bc72 LastWr: 1cd486bafe9d4dd Change: 1d2db8995022e74
\\?\UNC\TARGET-FILER\testfolder\tr-3563_ThinProvisioningIncreasesStorageUtilizationWithOnDemandUtilization.pdf: Create: 1cd4ee636a895aa LastA: 1d2db898967722c LastWr: 1cd49f39598b963 Change: 1d2db89953796ea
\\?\UNC\TARGET-FILER\testfolder\tr-3702_BestPractivesForMicrosoftVirtualizationAndSnapManagerHyperV.pdf: Create: 1cd4ee636b19682 LastA: 1d2db89898e5aea LastWr: 1cd4a3269d656a4 Change: 1d2db89955b9a18
\\?\UNC\TARGET-FILER\testfolder\tr-3828_SnapDrive62WindowsBestPractices.pdf: Create: 1cd4ee636ba7040 LastA: 1d2db8989a3907c LastWr: 1cd49f3b27efb4b Change: 1d2db8995689448
\\?\UNC\TARGET-FILER\testfolder\tr-3965_ThinProvisioningDeploymentAndImplementationGuideDataOntap87-mode.pdf: Create: 1cd4ee636c37136 LastA: 1d2db8989d0e1bc LastWr: 1cd496422779fad Change: 1d2db89958c6ea4
	                   7 -A--------D--	\\?\UNC\SOURCE-FILER\testfolder\
FindNextFile() Difference = - 16845h:22m:20.6824330s	\\SOURCE-FILER\testfolder\Thumbs.db
FindNextFile() Difference = - 17835h:22m:32.9412450s	\\SOURCE-FILER\testfolder\tr-3197_TEchnicalOverviewofSnapDrive.pdf
FindNextFile() Difference = - 17835h:22m:32.9712430s	\\SOURCE-FILER\testfolder\tr-3483_ThinProvisioningInANetAppSANorIPSANEnterpriseEnvironment.pdf
FindNextFile() Difference = - 17835h:22m:33.3192590s	\\SOURCE-FILER\testfolder\tr-3563_ThinProvisioningIncreasesStorageUtilizationWithOnDemandUtilization.pdf
FindNextFile() Difference = - 17835h:22m:33.5532690s	\\SOURCE-FILER\testfolder\tr-3702_BestPractivesForMicrosoftVirtualizationAndSnapManagerHyperV.pdf
FindNextFile() Difference = - 17835h:22m:33.6363180s	\\SOURCE-FILER\testfolder\tr-3828_SnapDrive62WindowsBestPractices.pdf
FindNextFile() Difference = - 17835h:22m:33.8692850s	\\SOURCE-FILER\testfolder\tr-3965_ThinProvisioningDeploymentAndImplementationGuideDataOntap87-mode.pdf
			SR GROUP OWNER DACL SACL - SECURITY_DESCRIPTOR_CONTROL
			     D     D   PIDP PIDP 
			SR   -     -   YI-- YI-- - Source
	      Ge„ndert		   25088 --SH---------	Thumbs.db
			SR GROUP OWNER DACL SACL - SECURITY_DESCRIPTOR_CONTROL
			     D     D   PIDP PIDP 
			SR   -     -   YI-- YI-- - Source
	      Ge„ndert		  754236 -A-----------	tr-3197_TEchnicalOverviewofSnapDrive.pdf
			SR GROUP OWNER DACL SACL - SECURITY_DESCRIPTOR_CONTROL
			     D     D   PIDP PIDP 
			SR   -     -   YI-- YI-- - Source
	      Ge„ndert		   98707 -A-----------	tr-3483_ThinProvisioningInANetAppSANorIPSANEnterpriseEnvironment.pdf
			SR GROUP OWNER DACL SACL - SECURITY_DESCRIPTOR_CONTROL
			     D     D   PIDP PIDP 
			SR   -     -   YI-- YI-- - Source
	      Ge„ndert		   3.4 m -A-----------	tr-3563_ThinProvisioningIncreasesStorageUtilizationWithOnDemandUtilization.pdf
			SR GROUP OWNER DACL SACL - SECURITY_DESCRIPTOR_CONTROL
			     D     D   PIDP PIDP 
			SR   -     -   YI-- YI-- - Source
	      Ge„ndert		   2.3 m -A-----------	tr-3702_BestPractivesForMicrosoftVirtualizationAndSnapManagerHyperV.pdf
			SR GROUP OWNER DACL SACL - SECURITY_DESCRIPTOR_CONTROL
			     D     D   PIDP PIDP 
			SR   -     -   YI-- YI-- - Source
	      Ge„ndert		  755698 -A-----------	tr-3828_SnapDrive62WindowsBestPractices.pdf
			SR GROUP OWNER DACL SACL - SECURITY_DESCRIPTOR_CONTROL
			     D     D   PIDP PIDP 
			SR   -     -   YI-- YI-- - Source
	      Ge„ndert		   2.3 m -A-----------	tr-3965_ThinProvisioningDeploymentAndImplementationGuideDataOntap87-mode.pdf
			SR GROUP OWNER DACL SACL - SECURITY_DESCRIPTOR_CONTROL
			     D     D   PIDP PIDP 
			SR   -     -   YI-- YI-- - Source
------------------------------------------------------------------------------
           Insgesamt   KopiertšbersprungenKeine šbereinstimmung    FEHLER    Extras
Verzeich.:         1         0         0         0         0         0
  Dateien:         7         7         0         0         0         0
    Bytes:    9.68 m    9.68 m         0         0         0         0
   Zeiten:   0:00:00   0:00:00                       0:00:00   0:00:00
Geschwindigkeit:           406352480 Bytes/Sek.
Geschwindigkeit:           23251.675 Megabytes/Min.
   Beendet: Freitag, 2. Juni 2017 12:18:40

OK. So the same command with Robocopy as it shipped with Windows XP:

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows     ::     Version XP010
-------------------------------------------------------------------------------

  Started : Fri Jun 02 12:28:25 2017

OSVersion : 6.2 (9200) 23F00206

   Source : \\?\UNC\SOURCE-FILER\testfolder\
     Root : "\\SOURCE-FILER\fpm\"
  VolName : "fpm"
   Serial : 80000415
  MaxName : 255
   FSFlag : 000400CF
   FSType : "NTFS"
   IsNTFS : 1

     Dest : \\?\UNC\TARGET-FILER\testfolder\
     Root : "\\TARGET-FILER\data\"
  VolName : ""
   Serial : 3B02D496
  MaxName : 255
   FSFlag : 0004004F
   FSType : "NTFS"
   IsNTFS : 1

    Fudge : 0

    Files : *.*
	    
  Options : *.* /S /E /COPYALL /PURGE /MIR /DEBUG /R:1000000 /W:30 

------------------------------------------------------------------------------

	                   7 -A--------D--	\\?\UNC\SOURCE-FILER\testfolder\

------------------------------------------------------------------------------

                Total    Copied   Skipped  Mismatch    FAILED    Extras
     Dirs :         1         0         1         0         0         0
    Files :         7         0         7         0         0         0
    Bytes :    9.68 m         0    9.68 m         0         0         0
    Times :   0:00:00   0:00:00                       0:00:00   0:00:00

    Ended : Fri Jun 02 12:28:25 2017

And finally, the "new" robocopy again but (re-) copying the folder to a Windows Fileshare (my 8.1 workstation in this case):

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robustes Dateikopieren fr Windows                              
-------------------------------------------------------------------------------
  Gestartet: Freitag, 2. Juni 2017 12:33:38
OSVersion : 6.3 (9600) 25800306
   Quelle : \\?\UNC\SOURCE-FILER\testfolder\
     Root : "\\o1903cifs\fpm\"
  VolName : "fpm"
   Serial : 80000415
  MaxName : 255
   FSFlag : 000400CF
   FSType : "NTFS"
   IsNTFS : 1
     Ziel : \\?\UNC\WINDOWS\testfolder\
     Root : "\\o0001\dmp\"
  VolName : "E_DATA"
   Serial : 6431E6C3
  MaxName : 255
   FSFlag : 00C700FF
   FSType : "NTFS"
   IsNTFS : 1
    Fudge : 0
    Dateien : *.*
	    
  Optionen: *.* /S /E /COPYALL /PURGE /MIR /DEBUG /R:1000000 /W:30 
------------------------------------------------------------------------------
\\?\UNC\SOURCE-FILER\testfolder\tr-3197_TEchnicalOverviewofSnapDrive.pdf: Create: 1cd4ee6369beb3e LastA: 1d2db8bad9a9546 LastWr: 1cd49f3a4f22f93 Change: 1d093933059611c
\\?\UNC\SOURCE-FILER\testfolder\tr-3483_ThinProvisioningInANetAppSANorIPSANEnterpriseEnvironment.pdf: Create: 1cd4ee636a428c6 LastA: 1d2db8bad9df0a6 LastWr: 1cd486bafe9d4dd Change: 1d093933059af46
\\?\UNC\SOURCE-FILER\testfolder\tr-3563_ThinProvisioningIncreasesStorageUtilizationWithOnDemandUtilization.pdf: Create: 1cd4ee636a895aa LastA: 1d2db8badceeb66 LastWr: 1cd49f39598b963 Change: 1d093933059fd5c
\\?\UNC\SOURCE-FILER\testfolder\tr-3702_BestPractivesForMicrosoftVirtualizationAndSnapManagerHyperV.pdf: Create: 1cd4ee636b19682 LastA: 1d2db8badf2edf4 LastWr: 1cd4a3269d656a4 Change: 1d09393305a4b86
\\?\UNC\SOURCE-FILER\testfolder\tr-3828_SnapDrive62WindowsBestPractices.pdf: Create: 1cd4ee636ba7040 LastA: 1d2db8badffbf2a LastWr: 1cd49f3b27efb4b Change: 1d09393305a999c
\\?\UNC\SOURCE-FILER\testfolder\tr-3965_ThinProvisioningDeploymentAndImplementationGuideDataOntap87-mode.pdf: Create: 1cd4ee636c37136 LastA: 1d2db8bae228938 LastWr: 1cd496422779fad Change: 1d09393305ae7b2
\\?\UNC\SOURCE-FILER\testfolder\Thumbs.db: Create: 1ce4bf5d00dcf28 LastA: 1d2db8bad8e6050 LastWr: 1ce4bf5dbc19250 Change: 1d0b3fd4d24fd44
\\?\UNC\WINDOWS\testfolder\Thumbs.db: Create: 1ce4bf5d00dcf28 LastA: 1d2db8994f3c65e LastWr: 1ce4bf5dbc19250 Change: 1d0b3fd4d24fd44
\\?\UNC\WINDOWS\testfolder\tr-3197_TEchnicalOverviewofSnapDrive.pdf: Create: 1cd4ee6369beb3e LastA: 1d2db899501cfd8 LastWr: 1cd49f3a4f22f93 Change: 1d093933059611c
\\?\UNC\WINDOWS\testfolder\tr-3483_ThinProvisioningInANetAppSANorIPSANEnterpriseEnvironment.pdf: Create: 1cd4ee636a428c6 LastA: 1d2db8995068ac8 LastWr: 1cd486bafe9d4dd Change: 1d093933059af46
\\?\UNC\WINDOWS\testfolder\tr-3563_ThinProvisioningIncreasesStorageUtilizationWithOnDemandUtilization.pdf: Create: 1cd4ee636a895aa LastA: 1d2db89953c1968 LastWr: 1cd49f39598b963 Change: 1d093933059fd5c
\\?\UNC\WINDOWS\testfolder\tr-3702_BestPractivesForMicrosoftVirtualizationAndSnapManagerHyperV.pdf: Create: 1cd4ee636b19682 LastA: 1d2db8995601bf6 LastWr: 1cd4a3269d656a4 Change: 1d09393305a4b86
\\?\UNC\WINDOWS\testfolder\tr-3828_SnapDrive62WindowsBestPractices.pdf: Create: 1cd4ee636ba7040 LastA: 1d2db89956d3b74 LastWr: 1cd49f3b27efb4b Change: 1d09393305a999c
\\?\UNC\WINDOWS\testfolder\tr-3965_ThinProvisioningDeploymentAndImplementationGuideDataOntap87-mode.pdf: Create: 1cd4ee636c37136 LastA: 1d2db899590efc4 LastWr: 1cd496422779fad Change: 1d09393305ae7b2
	                   7 -A--------D--	\\?\UNC\SOURCE-FILER\testfolder\
			SR GROUP OWNER DACL SACL - SECURITY_DESCRIPTOR_CONTROL
			     D     D   PIDP PIDP 
			SR   -     -   YI-- YI-- - Source
------------------------------------------------------------------------------
           Insgesamt   KopiertšbersprungenKeine šbereinstimmung    FEHLER    Extras
Verzeich.:         1         0         0         0         0         0
  Dateien:         7         0         7         0         0         0
    Bytes:    9.68 m         0    9.68 m         0         0         0
   Zeiten:   0:00:00   0:00:00                       0:00:00   0:00:00
   Beendet: Freitag, 2. Juni 2017 12:33:38

So here lies the problem then. It is a difference in the "change" timestamp. When the files are copied to a Windows Fileshare, Robocopy reads that value from the files correctly.

However when Robocopy get's the "last changed" timestamp from the NetApp, different values are showing!

Have a look at this comparison from the robocopy "to windows":

\\?\UNC\SOURCE-FILER\testfolder\Thumbs.db: Create: 1ce4bf5d00dcf28 LastA: 1d2db8bad8e6050 LastWr: 1ce4bf5dbc19250 Change: 1d0b3fd4d24fd44
\\?\UNC\WINDOWS\testfolder\Thumbs.db:      Create: 1ce4bf5d00dcf28 LastA: 1d2db8994f3c65e LastWr: 1ce4bf5dbc19250 Change: 1d0b3fd4d24fd44

and now what happens when we copy to NetApp:

\\?\UNC\SOURCE-FILER\testfolder\Thumbs.db: Create: 1ce4bf5d00dcf28 LastA: 1d2db8994f3c65e LastWr: 1ce4bf5dbc19250 Change: 1d0b3fd4d24fd44
\\?\UNC\TARGET-FILER\testfolder\Thumbs.db: Create: 1ce4bf5d00dcf28 LastA: 1d2db89890de0ea LastWr: 1ce4bf5dbc19250 Change: 1d2db8994f02cce

See the difference? On the NetApp the "change" value for the file differs from the value on the Windows fileshare. The value does NOT change when I re-run robocopy by the way, on neither Windows nor NetApp.

By the way, the "translation" for the timestamps:

1d0b3fd4d24fd44 => 01.07.2015 14:56:09

1d2db8994f02cce => 02.06.2017 12:18:30

So this is it them - when Robocopy copies a file to a Windows fileshare it actually successfully copies "changed" timestamp. But it is NOT able to do so on a NetApp fileshare where the value corresponds with the time when the file was actually copied. And since Robocopy is not able to properly fix this timestamp it "re-tries" to fix it every single time we run robocopy. So as I said earlier, this seems to me to be "on the NetApp side of things".

None of this helps us though. Because we all know that opening a case for this is futile, but NetApp's and Microsoft's L1 will blame the other company and send me away. Anyone got an "inside contact" who might be in a position to have a look at this?

Regards

Chris

Re: Fileserver Migration with robocopy

dryland404 — Fri, 09 Feb 2018 17:19:42 GMT

Stick to Server2008 for your update and final copy run .

I have found as well on 2016 server all subsequnt copy comparisons find the files have been modified even when using /fft flag to reduce time accuracy.

The same Robocopy command on 2008 server runs without issue and only copies the changed files.

Re: Fileserver Migration with robocopy

CHMOELLER — Fri, 23 Feb 2018 16:20:49 GMT

That's a workaround, not a solution ... especially since 2008 is in heavy phase-out, not planning to have many of them left. I think the situation is quite clear and I think it's on NetApps side but still, I know that calling Global support about this is futile and I have not "inside contact" ...

Re: Fileserver Migration with robocopy

PowderedToastMan — Tue, 08 Dec 2020 20:22:03 GMT

did you ever find a workable solution to this problem?

Re: Fileserver Migration with robocopy

GidonMarcus — Wed, 09 Dec 2020 00:05:37 GMT

Hi @PowderedToastMan

I think people on this thread raised 3 different scenarios that can cause this. With a mixture of windows Cdot and 7-mode mentioned and each scenario resolved differently. We can try and look on your scenario if you add a bit more details and try the different workarounds suggested here.

@ChristianSmetwho reported ODX was the cause in his case (In Win to Cdot). Referred to "/NOOFFLOAD" robocopy switch (not tested) and ODX disable on vserver "set adv; vserver cifs options modify -vserver [name] -copy-offload-enabled false"

@CHMOELLER and @BRECHTMON who reported that 2008/xp ropocopy.exe works for them and blame "last changed" not being copied hence being different. (on Cdot 7Mode mix with each other and with windows)

@CHMOELLERwho at some point confirmed /COPY:DATS takes longer from /COPY:DAT

for that one - I can confirm that in tests I did from Windows 2016 to Cdot 9.1 two years ago iv'e seen

/COPY:DAT only updates actual changed files, /COPY:DATO and /COPY:DATOS touches (but not copy) every file.

With DATOS I was also often losing access to dst files due to lack of permissions. Therefore, my approach is baseline with DATO, updates with DAT and cutoff with DATOS

Re: Fileserver Migration with robocopy

Greg_Wilson — Wed, 03 Nov 2021 12:00:20 GMT

Any update to this? I’m doing a large migration to 9.9.1P4 and every robocopy update is pretty much doing a full copy again. And saying all files have changed..

Re: Fileserver Migration with robocopy

mattgio — Thu, 04 Nov 2021 10:17:16 GMT

Hello Greg,

from experience I had success performing the copy using the following robocopy script:

#powershell
$source = "\\source\path"
$destination = "\\destination\path"
$log = "yourlogname$_(get-date -format 'yyyy-MM-dd-HH-mm').txt"
robocopy $source $destination /MIR /FFT /TEE /R:0 /W:0 /log+:C:\Logs\"$($log)"

The /FFT switch should anyway fix this and if you give it a try you might be able to actually turn any additional copies into incrementals.

Here's an explanation of what the /FFT does:

"/FFT is necessary to copy between file systems, such as Windows’ NTFS and Linux’s EXT4. If you don’t use this you can get wierdness like files looking like they’re newer than they really are, etc., since the two file systems keep time differently."

Hope it helps, let me know

Re: Fileserver Migration with robocopy

Joachim1 — Fri, 24 Jun 2022 09:14:50 GMT

Solved it for us:
https://serverfault.com/questions/828467/robocopy-all-data-marked-as-modified