Maciej Pocwierz, a senior software engineer, recently revealed a significant issue—an empty S3 bucket can unexpectedly result in a substantial AWS bill. In his case, nearly 100,000,000 S3 PUT requests were executed within a single day, leading to a bill that was far from negligible.
Pocwierz worked on a proof of concept for a client by creating a single S3 bucket in the eu-west-1 region and uploaded some files there for testing. When checking the AWS billing page to verify he stayed within the limits of the free tier, he found out the bill was $1300 USD caused by 100,000,000 S3 PUT requests executed.
When looking into the cause of the requests, Pocwierz writes:
By default, AWS doesn’t log requests executed against your S3 buckets. However, such logs can be enabled using AWS CloudTrail or S3 Server Access Logging. After enabling CloudTrail logs, I immediately observed thousands of write requests originating from multiple accounts or outside AWS.
Billed S3 usage per day per region (Source: Pocwierz Medium blog post)
The root cause of this incident was the default settings of a widely used open-source tool, which unintentionally targeted the same bucket name for its backups. This led to a relentless barrage of unauthorized requests with significant financial implications. Furthermore, half of the requests originated from a different region. S3 requests without a specified region are redirected to us-east-1, incurring an additional cost for the bucket owner.
The lessons learned from the experience were that S3 buckets are vulnerable to unauthorized requests, highlighting the need to add random suffixes to bucket names for enhanced security. Another key takeaway is that specifying the AWS region explicitly when executing requests can help avoid additional costs incurred through redirects. Additionally, the individual's decision to temporarily open their bucket for public writing revealed the alarming ease with which innocent oversights can lead to potential data leaks.
The findings of Pocwierz lead to a stir in the community. In a Reddit thread on his findings, seanamos-1 concluded:
Bucket names were never implied to need to be secret, and it's obvious they weren't designed to be that way. But if you don't keep them secret, you are vulnerable to a billing attack. This needs to be addressed.
Furthermore, in a Hacker News thread, a respondent, tedminston, writes:
What's crazier is that turning on Requester Pays does not actually mean the requester pays when the request is a 403. Essentially, every S3 bucket, public or private, whose name is discovered can be DDoSed, creating an insane bill. That's a platform level security issue for AWS to resolve. We do not need another Bucketgate.
Lastly, Jeff Barr, chief evangelist at AWS, stated in a tweet in response to Pocwierz's findings and community:
Thank you to everyone who brought this article to our attention. We agree that customers should not have to pay for unauthorized requests that they did not initiate. We’ll have more to share on exactly how we’ll help prevent these charges shortly.
With Corey Quinn, chief cloud economist at DuckbillGroup, responding in a subsequent tweet:
This feels like a potential turning point in AWS's approach to bill surprises, and I'm very much here for it.
Lastly, the company came up with a solution on the 13th of May, stating that Amazon S3 will make a change so unauthorized requests that customers did not initiate are free of charge.