If you are in the object storage or data lake space, you have probably heard about Amazon’s recent announcement of a new file system gateway for S3 object storage named Mountpoint for Amazon S3. Mountpoint provides high throughput file access to your S3 bucket of objects. Amazon is targeting the large data lake solutions used for research, analytics, and machine learning. In these cases, the data is ingested in object format, but many of the analysis tools rely on good old file access. Though there are several Linux file access solutions for object today, most have struggled to meet the demands of these workloads. Amazon is hoping to change that with Mountpoint by creating a solution that is tailor-made for these high-throughput, read-heavy applications. Unlike some of the other file gateways, Mountpoint is not POSIX compliant so it may not be suitable for all use cases. In the alpha release, the product is read-only with the plan to add support for sequential writes.
The question you probably have is, “Can Mountpoint be used with NetApp StorageGRID?” The answer is YES! StorageGRID supports the necessary APIs and the configuration of the application allows for non-Amazon endpoints.
As a test, I installed mountpoint-S3 on CentOS 8 Stream. It was a painless experience, following the instructions in the Github repository on installing the dependencies, cloning the repo, and compiling the client. Once the client is compiled, create a credentials file in /home/username/.aws/ just as you would with the AWS CLI tool.
[aronk@mountpoint-s3 ~]# cat .aws/credentials
[default]
aws_access_key_id = ABCDEFGHIJKLMNOP1234
aws_secret_access_key = ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890
Create the directory where your bucket would mount on and run the mountpoint command to attach the bucket. In this example the bucket name is td02, the directory I am using for my mount point is ~/mnt/, and the StorageGRID endpoint is sgdemo.netapp.com.
[root@mountpoint-s3 ~]# mountpoint-s3/target/release/mount-s3 td02 ~/mnt --endpoint-url https://sgdemo.netapp.com
Now I can list and read the objects as if they are part of the filesystem.
[aronk@mountpoint-s3 ~]# ls mnt/
TestObject.0 TestObject.1284 TestObject.1570 TestObject.1857 TestObject.2142 TestObject.43 TestObject.716
TestObject.1 TestObject.1285 TestObject.1571 TestObject.1858 TestObject.2143 TestObject.430 TestObject.717
TestObject.10 TestObject.1286 TestObject.1572 TestObject.1859 TestObject.2144 TestObject.431 TestObject.718
TestObject.100 TestObject.1287 TestObject.1573 TestObject.186 TestObject.2145 TestObject.432 TestObject.719
TestObject.1000 TestObject.1288 TestObject.1574 TestObject.1860 TestObject.2146 TestObject.433 TestObject.72
TestObject.1001 TestObject.1289 TestObject.1575 TestObject.1861 TestObject.2147 TestObject.434 TestObject.720
[aronk@mountpoint-s3 ~]# cat mnt/ TestObject.43
bcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890!@#$%^&*()_+=-[]{};.abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890!@#$%^&*()_+=-[]{};.abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890!@#$%^&*()_+=-[]{};.abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890!@#$%^&*()_+=-[]{};.abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890
The object storage market is continuing to grow with its sights strongly fixed on the primary workloads dominated by filesystems. The need for file access protocols are not going quietly into the night. So, we are happy to see Amazon take on this challenge. Like many of you, I can’t wait to see how this product evolves and how we can help you integrate it into your workflows.
