Network and Storage Protocols

Using Widelinks for redirect a "folder" to other volume.

DENISOZORIO
10,365 Views

Hello Everybody!

 

Please, I need your help for a doubt about Widelinks.

 

 

My scenario is:

 

I have 1 volume/cifs share with 9TB

 

And I need to split this volume in 2 volumes.

 

So, I thinking to create another volume, move a big folder to this volume and create a kind of shutcut in the volume 1 to users does not see differences.

 

 

Today: \\netappserver\eng\bigfolder_name

 

Future: \\netappserver\eng\bigfolder_widelink (target:\\netappserver\eng2\bigfolder)

 

 

If anyone can help-me.

 

Thanks a lot!

Denis

10 REPLIES 10

bobshouseofcards
10,330 Views

Hi Denis -

 

Widelinks are certainly one way to do this, and essentially the only way in 7-mode.  If in cDot, creating a second volume with a mount point is much simpler and accomplishes the same result.  You could still do widelinks in cDot, but it's way more trouble than it's worth.

 

Obvious first question is are you in 7-mode or cDot?

 

Second question is whether there is one (or maybe two) directories (folders) that form natural points to break up the current volume.  If you have a wide flat structure, splitting with widelinks is not easy or fun.  However if you have a narrow structure at the top of the directory hierarchy, it is simpler.

 

For example, let's say volume "vol1" is shared as "DATA" from CIFS name "NTAP".  So you have stuff like "\\NTAP\DATA\Folder01", "\\NTAP\DATA\Folder02", etc.  If you have 100 folders at this top level, not so easy to split.  But if you only have say three folders at the top, and you want to split off one of them into a separate volume, that isn't hard to setup.

 

A CIFS widelink is a jump from a point in one CIFS presented file system to a point in another CIFS presented file system not in the same underlying volume - hence the "wide" part of the link.  In fact, both can be on different servers, including onto a jump from a NetApp device onto a non-NetApp device.  For all intents, it's a manual DFS like redirection from within one file system to another (just not integrated into AD like DFS).

 

The underlying mechanism can get confusing.  Widelinks are triggered through a symbolic link in the underlying WAFL file system.  Recall that under the covers WAFL is a filesystem with essentially POSIX semantics - in other words UNIX like.  POSIX file systems support linked files.  A hard link directory entry is literally a multiple pointer to the same physical media on the same physical volume.  A symbolic link, which is what we care about, is a reference to another filesystem location.  This is a natural device in an NFS world.  Data Ontap does some special stuff to simulate the effects for CIFS.

 

When you enable widelinks on a CIFS share, anytime any traversal through that share hits a symbolic link in the WAFL filesystem, DoT will lookup the target of the symbolic link in a translation table.  Remember that the "real" target of the symbloic link has POSIX (UNIX style) syntax.  The translation table pairs a POSIX style target with a UNC name of a CIFS target.  DoT sends a CIFS redirection to the requesting client that contains the UNC from the translation table.  The client then connects to the UNC transparently and happily keeps processing along - exactly the way DFS works.  The redirection isn't necessarily exposed to the client, depending on the tools they are using, and in general the client will cache the branch point and just know for subsequent accesses where the branch occurred to speed subsequent accesses (something to remember if you change the targets while users are in them - again just like DFS updates).


In a pure CIFS world the hard part is creating the symbolic link on the WAFL file system.  NFS protocol users have semantics to do this directly - CIFS users do not.  Fortunately there some utilities to call the ZAPI functions directly to create that link.

 

For a variety of reasons I have created this type of hierarchy before - typically around consolidating multiple instances of a large application data archive into a central data center from branch locations in 7-mode setups.  It works well for most all situations, but you will want to test as every now and again an application pops up that cares about the redirection.  These are usually easy to spot, as the applications would also generally have issues in a DFS environment.

 

Other downsides come into play when managing security and anything else that does full filesystem traversal - some applications by default don't follow redirected links.  You want to keep really good docuemntation on what is where as you start any planning on this.

 

The general steps I would follow to split a directory are:

1.  Choose a folder that represents is the top of a hierarchy I want to split out.  You can split on any folder in the tree - there is no limitation to be at a qtree boundary

2.  Create a volume to contain the split data and establish a CIFS share on that volume.

3.  Copy the data from the current location to the new location - robocopy in the generic case - to seed the new location.  users aren't using the new location yet.

4.  Verify data accessibility and security on the new share location.

5.  Choose a "target identifier" for the symbolic link you will create.  A symbolic link in the POSIX sense doesn't have to point to something that exists - it is just a pointer to "something".  Choose a naming convention carefully for the target if you expect to create more links.

6.  Create/update the symlink translation table to add an entry from the target identifier in step 5 to the new UNC verified in step 4.

7.  Turn on widelink capability on the original share if not already

 

All that is prep - users still see the original share.  But, if all that is in place correctly, then these next steps finish the cutover.  This is a hard cutover - it will be disruptive to active file users.

 

8.  Rename the top level folder (step 1) to something like "FOLDER-OLD".  You might have to force users sessions/files to close to do this.

9.  Create a symbolic link (via multi-protocol NFS access to the original share or using a ZAPI utility) that has the top level folder name and points to the "target identifier" from step 5.

10.  Check access through the original share to the folder in question via the original name.  If all is working, the symbolic link is now redirecting you.  You can verify that with a properties view of a file in the new location on a Windows client.

 

11.  If needed, run a last file copy/update from what is now "FOLDER-OLD" to the original FOLDER name.  This updates any changes made in the gap between the original copy and the access change.  If you schedule things well, this step might not be needed.

12.  Delete the FOLDER-OLD version to reclaim space.

 

OK - that's what you're in for with WideLinks.  If you can answer the two questions, I can help you with more specific commands and references.  

 

Widelinks were in my opinion a very under-utilized feature in 7-mode.  They provided a lot of power to address data location independence situations.  I use them even in cDot to address specific situations where legacy data layout is not conducive to data layout in Clustered DataOntap.  I have absorbed legacy server setups where I create a template first level directory that is just widelinks to other volumes/shares even across SVMs.  Makes it much easier to present independent administration points to different groups yet maintain the original logical layout, which is something even cDot mount points can't do.

 

 

I hope this helps you. 

 

Bob Greenwald

Lead Storage Engineer

Huron Consulting Group

NCDA Clustered | NCIE-SAN Clustererd

 

 

 

DENISOZORIO
10,325 Views

Hi Bob, Tks for answer.

 

Yes, I am in 7-Mode.

 

 

Well, after read all, I thinking for my "problem" It´s not the best options.

 

 

My main problem today is BACKUP. My backup is very slow. And after some research. I have 2 issue in my volume.

 

- First: Number of files, I have +/- 40 mi files on 1 volume;

- Second: Long path... many directories, have more than 255 caract.

 

I thinking to split my volume in 2 volumes, and create a widelink for my users do not using a new path to map a network drive for example.

 

 

But, now, I thinking it seems to me more easy publish a new folder for all users (~100users).

 

 

bobshouseofcards
10,310 Views

Yeah - if Backup is the issue, widelinks are necessarily the answer.  You'd still have the same file count and be cycling through those.

 

Of course it begs the next question - what are you using for backup?  Depending on what you are doing now and what you might have available, your issue might be attacked from a different angle...

 

 

 

DENISOZORIO
10,308 Views

My backup is:

 

2 volumes

 

1: 5TB - 86% used 13m files - 1,27TB Storage Efficiency saving

2: 8TB - 95% used 40m files - 2,52TB Storage Efficiency saving

 

2 Cifs shares

 

Many folders with long path problems

 

I using ArcServer Backup NDMP option - Using Ethenet of communication, In the past I use tape drive direct attched by FC on NetApp, but it´s the same results.

 

Library IBM = 1 Drive LTO 5 and new cambridge tape

 

My backup speed is  2,30 GB/min.

 

I really think my main problem is, number of files and long path problems.

bobshouseofcards
10,302 Views

Yup - NDMP drags badly as the file count increases.  It certainly served a needed purpose at one time, but it was never designed for the massive scale we can see today.

AnandManian
8,472 Views

Helo Bob...

 

Thanks for this excellent write up.  Can you please share which version(s) of C-DoT you have used Widelinks on so far...?

 

We are trying to use it in the  ONTAP-cloud implemented in AWS (which I believe is ONTAP-9.2) and running into some basic troubles.

Goal is to create a large Windows filespace by stitching together multiple CIFS volumes. 

The vserver command to create CIFS widelink mapping succeeds, and displays a table correctly showing the translation we want.

But when we go to the fileshare itself... there is nothing.. i.e. no shortcut/directory corresponding to the widelink .. to be seen.

 

Wondering if the vserver command we are using only sets up the translation... and a separate step is needed for setting up the symbolic link itself as you have noted in your post: 

 

".....  In a pure CIFS world the hard part is creating the symbolic link on the WAFL file system.  NFS protocol users have semantics to do this directly - CIFS users do not.  Fortunately there some utilities to call the ZAPI functions directly to create that link.  ..."

 

If this is the case, can you please recommend some utilities that you have tried successfully...?

 

Thanks.

 

 

 

-Anand Manian

General Electric 

SeanHatfield
8,447 Views

For that use case, why not create multiple volumes, and mount them in the namespace as needed to create your file system?  If that method works its going to be much cleaner than symlinks/widelinks.

 

If you do need to make a symlink from windows, try using the NetApp Powershell Toolkit.  It has a cmdlet for creating symlinks.

get-help new-ncsymlink

NAME
    New-NcSymLink

SYNOPSIS
    Create a new symbolic link


SYNTAX
    New-NcSymLink [-Symlink] <String> [-Path] <String> [-VserverContext <String>] [-Controller <NcController[]>]
    [-ZapiRetryCount <Int32>] [<CommonParameters>]


DESCRIPTION
    Create a new symbolic link


RELATED LINKS
    Get-NcSymlinkTarget
    Remove-NcFile
    New-NcFile

REMARKS
    To see the examples, type: "get-help New-NcSymLink -examples".
    For more information, type: "get-help New-NcSymLink -detailed".
    For technical information, type: "get-help New-NcSymLink -full".
    For online help, type: "get-help New-NcSymLink -online"

 

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

bobshouseofcards
8,268 Views
Sean -

Totally agree when that scenario can work. Widelinks are not for the feint of heart and have significant management overhead and possibility of confusion.

Building a good namespace using volumes mounted appropriately up front is best. A greenfield starting condition for that helps.

Consider cases of crossing SVM boundaries, crossing cluster boundaries, etc. Those situations sometimes require additional help, especially if path dependencies are already established on existing namespaces. Widelinks are a way to jump over those boundaries as well.

Bob

AnandManian
8,172 Views

Sean, Bob....

 

Thank you both for your inputs.

 

To give you a bit more detail, we are  trying to build a CIFS file store in AWS using ONTAP-Cloud (functionality is "nearly" identical to C-Dot 9.x)

 

Per current limitations, It is possible to build upto a 360TB fileshare here by stitching together multiple volumes using Junctions/Namespace.

All such volumes  must come from the same SVM... and each volume is limited to 80-TB max size.

 

Since, per my understanding, Widelinks are translated to DFS re-directs that can reach into other SVMs... wondering if  >360-TB CIFS volumes can be built using widelinks.

 

Much appreciate your thoughts/inputs on whether this is do-able and any experience with doing similar setups... and/or pointers on how to go about it.

 

Thanks.

 

-Anand Manian

General Electric

 

 

 

 

 

 

 

SeanHatfield
6,215 Views

I don’t see a reason that wouldn’t work. I have stitched together systems like that, and there are a couple of caveats. First, you can’t share a symlink/widelink. They can be top level directories within a share, but they can’t be the share itself. And second, you can’t share anything below the widelink, so nested share structures need to reside in the parent file system 

 

So it can work, but takes some planning and places some constraints on the overall file system layout. 

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.
Public