Amazon Web Services is making it possible to access data stored in its S3 cloud storage service as a traditional file system, bridging a divide between two types of storage that has frustrated developers and data scientists for nearly two decades.
The new capability, called Amazon S3 Files, lets applications running on AWS access an S3 storage bucket as if it were a local file system, reading and writing data using standard file operations rather than specialized cloud storage commands.
In practice, this means a machine learning team can run a training job directly against data in S3 without first copying it to a separate file system. Or, perhaps more importantly these days, an AI agent can read and write files in S3 using the same basic tools it would use on a local hard drive.
S3, launched 20 years ago, holds a huge amount of the world’s cloud data. S3 Files promises to open the door for a much broader range of apps and AI systems to work directly with that data.
The backstory: In an unusually candid essay coinciding with the news, Andy Warfield, a vice president and distinguished engineer who leads S3 engineering at AWS, described the technical and philosophical challenges of making the feature work, and why the first approach failed.
The core issue, Warfield wrote, is that files and objects are fundamentally different.
Files can be edited in place and shared across applications in real time, working the way most software has always expected. Objects in S3 work differently: they are designed to be stored and retrieved as complete units, and millions of applications are built around that assumption.
So they “did the only sensible thing you can do when you are faced with a really difficult technical design problem: we locked a bunch of our most senior engineers in a room and not let them out till they had a plan that they all liked,” Warfield wrote.
“Passionate and contentious discussions ensued,” he said. “And then finally we gave up.”
But ultimately, the team found its answer by no longer trying to hide the boundary between files and objects and instead making it a deliberate part of the design.
The approach: S3 Files uses a “stage and commit” model, borrowing the concept from version control systems like Git: changes accumulate on the file system side and are pushed back to S3 as whole objects, preserving the guarantees that existing S3 applications depend on.
Google and Microsoft offer their own tools for accessing cloud object storage through file system interfaces, but AWS is positioning S3 Files as a deeper integration, backed by a fully managed file system rather than a simple adapter.
S3 Files is available today in AWS regions worldwide, built on Amazon’s Elastic File System. The company says it has been in customer testing for about nine months.
Read the full article here
