Tech ONTAP Blogs

Mountpoint for Amazon S3 alpha release with StorageGRID

aronk
NetApp
1,781 Views

If you are in the object storage or data lake space, you have probably heard about Amazon’s recent announcement of a new file system gateway for S3 object storage named Mountpoint for Amazon S3. Mountpoint provides high throughput file access to your S3 bucket of objects. Amazon is targeting the large data lake solutions used for research, analytics, and machine learning. In these cases, the data is ingested in object format, but many of the analysis tools rely on good old file access.  Though there are several Linux file access solutions for object today, most have struggled to meet the demands of these workloads.  Amazon is hoping to change that with Mountpoint by creating a solution that is tailor-made for these high-throughput, read-heavy applications. Unlike some of the other file gateways, Mountpoint is not POSIX compliant so it may not be suitable for all use cases. In the alpha release, the product is read-only with the plan to add support for sequential writes.

 

The question you probably have is, “Can Mountpoint be used with NetApp StorageGRID?”  The answer is YES! StorageGRID supports the necessary APIs and the configuration of the application allows for non-Amazon endpoints.

 

As a test, I installed mountpoint-S3 on CentOS 8 Stream. It was a painless experience, following the instructions in the Github repository on installing the dependencies, cloning the repo, and compiling the client. Once the client is compiled, create a credentials file in /home/username/.aws/ just as you would with the AWS CLI tool.

 

[aronk@mountpoint-s3 ~]# cat .aws/credentials

[default]

aws_access_key_id = ABCDEFGHIJKLMNOP1234

aws_secret_access_key = ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890

 

Create the directory where your bucket would mount on and run the mountpoint command to attach the bucket. In this example the bucket name is td02, the directory I am using for my mount point is ~/mnt/, and the StorageGRID endpoint is sgdemo.netapp.com.

 

[root@mountpoint-s3 ~]# mountpoint-s3/target/release/mount-s3 td02 ~/mnt --endpoint-url https://sgdemo.netapp.com

 

Now I can list and read the objects as if they are part of the filesystem.

 

[aronk@mountpoint-s3 ~]# ls mnt/

TestObject.0     TestObject.1284  TestObject.1570  TestObject.1857  TestObject.2142  TestObject.43   TestObject.716

TestObject.1     TestObject.1285  TestObject.1571  TestObject.1858  TestObject.2143  TestObject.430  TestObject.717

TestObject.10    TestObject.1286  TestObject.1572  TestObject.1859  TestObject.2144  TestObject.431  TestObject.718

TestObject.100   TestObject.1287  TestObject.1573  TestObject.186   TestObject.2145  TestObject.432  TestObject.719

TestObject.1000  TestObject.1288  TestObject.1574  TestObject.1860  TestObject.2146  TestObject.433  TestObject.72

TestObject.1001  TestObject.1289  TestObject.1575  TestObject.1861  TestObject.2147  TestObject.434  TestObject.720

 

[aronk@mountpoint-s3 ~]# cat mnt/ TestObject.43

 

bcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890!@#$%^&*()_+=-[]{};.abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890!@#$%^&*()_+=-[]{};.abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890!@#$%^&*()_+=-[]{};.abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890!@#$%^&*()_+=-[]{};.abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890

 

The object storage market is continuing to grow with its sights strongly fixed on the primary workloads dominated by filesystems. The need for file access protocols are not going quietly into the night. So, we are happy to see Amazon take on this challenge. Like many of you, I can’t wait to see how this product evolves and how we can help you integrate it into your workflows.

Public