2015-09-11 09:40 AM
Please, I need your help for a doubt about Widelinks.
My scenario is:
I have 1 volume/cifs share with 9TB
And I need to split this volume in 2 volumes.
So, I thinking to create another volume, move a big folder to this volume and create a kind of shutcut in the volume 1 to users does not see differences.
Future: \\netappserver\eng\bigfolder_widelink (target:\\netappserver\eng2\bigfolder)
If anyone can help-me.
Thanks a lot!
2015-09-11 11:28 AM
Hi Denis -
Widelinks are certainly one way to do this, and essentially the only way in 7-mode. If in cDot, creating a second volume with a mount point is much simpler and accomplishes the same result. You could still do widelinks in cDot, but it's way more trouble than it's worth.
Obvious first question is are you in 7-mode or cDot?
Second question is whether there is one (or maybe two) directories (folders) that form natural points to break up the current volume. If you have a wide flat structure, splitting with widelinks is not easy or fun. However if you have a narrow structure at the top of the directory hierarchy, it is simpler.
For example, let's say volume "vol1" is shared as "DATA" from CIFS name "NTAP". So you have stuff like "\\NTAP\DATA\Folder01", "\\NTAP\DATA\Folder02", etc. If you have 100 folders at this top level, not so easy to split. But if you only have say three folders at the top, and you want to split off one of them into a separate volume, that isn't hard to setup.
A CIFS widelink is a jump from a point in one CIFS presented file system to a point in another CIFS presented file system not in the same underlying volume - hence the "wide" part of the link. In fact, both can be on different servers, including onto a jump from a NetApp device onto a non-NetApp device. For all intents, it's a manual DFS like redirection from within one file system to another (just not integrated into AD like DFS).
The underlying mechanism can get confusing. Widelinks are triggered through a symbolic link in the underlying WAFL file system. Recall that under the covers WAFL is a filesystem with essentially POSIX semantics - in other words UNIX like. POSIX file systems support linked files. A hard link directory entry is literally a multiple pointer to the same physical media on the same physical volume. A symbolic link, which is what we care about, is a reference to another filesystem location. This is a natural device in an NFS world. Data Ontap does some special stuff to simulate the effects for CIFS.
When you enable widelinks on a CIFS share, anytime any traversal through that share hits a symbolic link in the WAFL filesystem, DoT will lookup the target of the symbolic link in a translation table. Remember that the "real" target of the symbloic link has POSIX (UNIX style) syntax. The translation table pairs a POSIX style target with a UNC name of a CIFS target. DoT sends a CIFS redirection to the requesting client that contains the UNC from the translation table. The client then connects to the UNC transparently and happily keeps processing along - exactly the way DFS works. The redirection isn't necessarily exposed to the client, depending on the tools they are using, and in general the client will cache the branch point and just know for subsequent accesses where the branch occurred to speed subsequent accesses (something to remember if you change the targets while users are in them - again just like DFS updates).
In a pure CIFS world the hard part is creating the symbolic link on the WAFL file system. NFS protocol users have semantics to do this directly - CIFS users do not. Fortunately there some utilities to call the ZAPI functions directly to create that link.
For a variety of reasons I have created this type of hierarchy before - typically around consolidating multiple instances of a large application data archive into a central data center from branch locations in 7-mode setups. It works well for most all situations, but you will want to test as every now and again an application pops up that cares about the redirection. These are usually easy to spot, as the applications would also generally have issues in a DFS environment.
Other downsides come into play when managing security and anything else that does full filesystem traversal - some applications by default don't follow redirected links. You want to keep really good docuemntation on what is where as you start any planning on this.
The general steps I would follow to split a directory are:
1. Choose a folder that represents is the top of a hierarchy I want to split out. You can split on any folder in the tree - there is no limitation to be at a qtree boundary
2. Create a volume to contain the split data and establish a CIFS share on that volume.
3. Copy the data from the current location to the new location - robocopy in the generic case - to seed the new location. users aren't using the new location yet.
4. Verify data accessibility and security on the new share location.
5. Choose a "target identifier" for the symbolic link you will create. A symbolic link in the POSIX sense doesn't have to point to something that exists - it is just a pointer to "something". Choose a naming convention carefully for the target if you expect to create more links.
6. Create/update the symlink translation table to add an entry from the target identifier in step 5 to the new UNC verified in step 4.
7. Turn on widelink capability on the original share if not already
All that is prep - users still see the original share. But, if all that is in place correctly, then these next steps finish the cutover. This is a hard cutover - it will be disruptive to active file users.
8. Rename the top level folder (step 1) to something like "FOLDER-OLD". You might have to force users sessions/files to close to do this.
9. Create a symbolic link (via multi-protocol NFS access to the original share or using a ZAPI utility) that has the top level folder name and points to the "target identifier" from step 5.
10. Check access through the original share to the folder in question via the original name. If all is working, the symbolic link is now redirecting you. You can verify that with a properties view of a file in the new location on a Windows client.
11. If needed, run a last file copy/update from what is now "FOLDER-OLD" to the original FOLDER name. This updates any changes made in the gap between the original copy and the access change. If you schedule things well, this step might not be needed.
12. Delete the FOLDER-OLD version to reclaim space.
OK - that's what you're in for with WideLinks. If you can answer the two questions, I can help you with more specific commands and references.
Widelinks were in my opinion a very under-utilized feature in 7-mode. They provided a lot of power to address data location independence situations. I use them even in cDot to address specific situations where legacy data layout is not conducive to data layout in Clustered DataOntap. I have absorbed legacy server setups where I create a template first level directory that is just widelinks to other volumes/shares even across SVMs. Makes it much easier to present independent administration points to different groups yet maintain the original logical layout, which is something even cDot mount points can't do.
I hope this helps you.
Lead Storage Engineer
Huron Consulting Group
NCDA Clustered | NCIE-SAN Clustererd
2015-09-11 11:57 AM
Hi Bob, Tks for answer.
Yes, I am in 7-Mode.
Well, after read all, I thinking for my "problem" It´s not the best options.
My main problem today is BACKUP. My backup is very slow. And after some research. I have 2 issue in my volume.
- First: Number of files, I have +/- 40 mi files on 1 volume;
- Second: Long path... many directories, have more than 255 caract.
I thinking to split my volume in 2 volumes, and create a widelink for my users do not using a new path to map a network drive for example.
But, now, I thinking it seems to me more easy publish a new folder for all users (~100users).
2015-09-11 02:39 PM
Yeah - if Backup is the issue, widelinks are necessarily the answer. You'd still have the same file count and be cycling through those.
Of course it begs the next question - what are you using for backup? Depending on what you are doing now and what you might have available, your issue might be attacked from a different angle...
2015-09-11 02:49 PM
My backup is:
1: 5TB - 86% used 13m files - 1,27TB Storage Efficiency saving
2: 8TB - 95% used 40m files - 2,52TB Storage Efficiency saving
2 Cifs shares
Many folders with long path problems
I using ArcServer Backup NDMP option - Using Ethenet of communication, In the past I use tape drive direct attched by FC on NetApp, but it´s the same results.
Library IBM = 1 Drive LTO 5 and new cambridge tape
My backup speed is 2,30 GB/min.
I really think my main problem is, number of files and long path problems.
2015-09-11 03:19 PM
Yup - NDMP drags badly as the file count increases. It certainly served a needed purpose at one time, but it was never designed for the massive scale we can see today.
2017-12-01 07:59 AM
Thanks for this excellent write up. Can you please share which version(s) of C-DoT you have used Widelinks on so far...?
We are trying to use it in the ONTAP-cloud implemented in AWS (which I believe is ONTAP-9.2) and running into some basic troubles.
Goal is to create a large Windows filespace by stitching together multiple CIFS volumes.
The vserver command to create CIFS widelink mapping succeeds, and displays a table correctly showing the translation we want.
But when we go to the fileshare itself... there is nothing.. i.e. no shortcut/directory corresponding to the widelink .. to be seen.
Wondering if the vserver command we are using only sets up the translation... and a separate step is needed for setting up the symbolic link itself as you have noted in your post:
"..... In a pure CIFS world the hard part is creating the symbolic link on the WAFL file system. NFS protocol users have semantics to do this directly - CIFS users do not. Fortunately there some utilities to call the ZAPI functions directly to create that link. ..."
If this is the case, can you please recommend some utilities that you have tried successfully...?
2017-12-01 10:24 AM - edited 2017-12-01 10:25 AM
For that use case, why not create multiple volumes, and mount them in the namespace as needed to create your file system? If that method works its going to be much cleaner than symlinks/widelinks.
If you do need to make a symlink from windows, try using the NetApp Powershell Toolkit. It has a cmdlet for creating symlinks.
get-help new-ncsymlink NAME New-NcSymLink SYNOPSIS Create a new symbolic link SYNTAX New-NcSymLink [-Symlink] <String> [-Path] <String> [-VserverContext <String>] [-Controller <NcController>] [-ZapiRetryCount <Int32>] [<CommonParameters>] DESCRIPTION Create a new symbolic link RELATED LINKS Get-NcSymlinkTarget Remove-NcFile New-NcFile REMARKS To see the examples, type: "get-help New-NcSymLink -examples". For more information, type: "get-help New-NcSymLink -detailed". For technical information, type: "get-help New-NcSymLink -full". For online help, type: "get-help New-NcSymLink -online"
2017-12-02 11:15 AM
2017-12-03 10:18 PM
Thank you both for your inputs.
To give you a bit more detail, we are trying to build a CIFS file store in AWS using ONTAP-Cloud (functionality is "nearly" identical to C-Dot 9.x)
Per current limitations, It is possible to build upto a 360TB fileshare here by stitching together multiple volumes using Junctions/Namespace.
All such volumes must come from the same SVM... and each volume is limited to 80-TB max size.
Since, per my understanding, Widelinks are translated to DFS re-directs that can reach into other SVMs... wondering if >360-TB CIFS volumes can be built using widelinks.
Much appreciate your thoughts/inputs on whether this is do-able and any experience with doing similar setups... and/or pointers on how to go about it.