During the recent re:Invent conference, AWS announced the general availability of S3 Express One Zone, a high-performance, single-AZ storage class that provides single-digit millisecond data access. Reducing request costs, the new storage class is designed for processing data in AI/ML training and financial modeling.
The S3 Express One Zone storage class can scale to process millions of requests per minute, relying on existing APIs, but requires a new bucket type, a new authentication model, and a new naming convention: it has a hierarchical namespace and store object key names in a directory-like manner, with the directory bucket name unique within the region. Jeff Barr, vice president and chief evangelist at AWS, explains why the storage class benefits mostly small objects:
This new storage class can handle objects of any size, but is especially awesome for smaller objects. This is because for smaller objects the time to first byte is very close to the time for last byte. In all storage systems, larger objects take longer to stream because there is more data to download during the transfer, and therefore the storage latency has less impact on the total time to read the object.
Among the suggested use cases for S3 Express One Zone, AWS includes machine learning and artificial intelligence training, interactive data analytics, high-performance computing (HPC), financial modeling, real-time advertising, and media content workloads. Ananth Packkildurai, principal engineer at Mural and editor of Data Engineering Weekly, writes about how it can impact data infrastructure:
S3 Express will open up the serverless data architecture, separating storage and computing from the mainstream data processing industry at all levels. We will see emerging patterns like stream ingestion directly into S3 Express, replicating S3 Express to S3 Standard for fault tolerance, and stream processing on top of S3 Express.
Objects are stored and replicated within a single availability zone that is chosen by the customer, allowing storage and compute resources colocation to further reduce latency. Even if storage costs are significantly higher, Packkildurai adds:
S3 Express significantly reduces TCO (...) I believe 8X the cost of S3 Express than S3 Standard is more tolerable than operating multiple distributed & stateful systems to achieve the same data processing capabilities.
Jack Vanlightly, staff technologist at Confluent, explains why the new class is not what he hoped for:
S3 Standard can be cheap and most definitely is highly durable. Its Achilles heel is the high, unpredictable latency (...) Express One Zone does not tick the "cheap" checkbox so the replicated, fault-tolerant cache implementations can continue to sleep easily that they are still relevant and highly valuable.
On separate S3-related announcements at re:Invent, Mountpoint for Amazon S3 and EMR now support the new storage class.
S3 Express One Zone is designed to deliver 99.95% availability with an availability SLA of 99.9%, starting at 0.16 USD/GB/month. The new class is currently only available in Northern Virginia, Oregon, Tokyo, and Stockholm regions.