An Empty, Private S3 Bucket Ran Up a $1,300 AWS Bill
- Platform
- AWS
- When
- April 2024
- Bill
- $1,300
- Outcome
- Bill canceled as an exception; AWS stopped billing for outside 403s
In April 2024, software engineer Maciej Pocwierz wrote that a private S3 bucket holding only a few test files cost him over $1,300 within two days. Nearly 100 million PUT requests from other people's systems hit the bucket in a single day, and S3 billed him for every one, even though each was rejected. AWS canceled the bill as an exception and, two weeks after his post, announced a change to S3 billing so that bucket owners no longer pay for most of these rejected requests.
What happened
- Setup: Pocwierz created one S3 bucket in
eu-west-1(Ireland) and uploaded a few test files for a document indexing proof of concept. - Two days later: He opened the billing page to confirm he was still inside the free tier. The bill was over $1,300, and the console showed nearly 100,000,000 S3 PUT requests in one day.
- Investigation: S3 does not log requests by default. After he turned on CloudTrail, he saw thousands of write requests coming from many AWS accounts and from outside AWS.
- Cause: A popular open-source tool used the same bucket name as a placeholder in its default backup configuration. Every deployment left on the defaults tried to write its backups to his bucket. He did not name the tool, because its users' data would have been exposed.
- Test: To confirm the theory, he briefly opened the bucket to public writes and received over 10 GB of other people's data in under 30 seconds.
- April 29, 2024: He published the story on Medium. Within days it was covered by Simon Willison's blog, The Register and InfoQ.
The tool's maintainers fixed the default after he reported it, but existing deployments kept the old setting.
Why the bill got so big
At the time, S3 billed the bucket owner for requests that failed with HTTP 4xx errors, including 403 Access Denied. AWS support confirmed this to Pocwierz in writing. Anyone who knew the bucket name could send unsigned PUT requests, get rejected, and still add to the owner's bill. No AWS account was needed to do it.
S3 Standard charges $0.005 per 1,000 PUT, COPY, POST or LIST requests, both in eu-west-1 and us-east-1, according to the AWS price list. That is $5 per million requests. At that rate:
- 100 million PUTs cost about $500.
- $1,300 corresponds to roughly 260 million PUT-class requests.
The post does not itemize the bill, so the gap between one day of traffic and the $1,300 total is not fully explained. Two details from the post likely account for part of it. The figure was the total after two days, not one. And over half of the bill came from us-east-1, a region where he had no buckets. S3 requests that do not specify a region go to us-east-1 first and are redirected, and the bucket owner paid for those redirected requests too.
A single machine can send thousands of requests per second, and a misconfigured tool deployed on many servers adds up faster still. Pocwierz noted that services like CloudFront or WAF cannot shield a bucket that is called directly through the S3 API.
How it ended
AWS canceled Pocwierz's S3 charges, but told him it was an exception. On April 30, AWS chief evangelist Jeff Barr wrote on X that AWS agreed that "customers should not have to pay for unauthorized requests that they did not initiate," and Pocwierz updated his post to say the S3 team was working on a fix.
On May 13, 2024, AWS announced that S3 would stop charging bucket owners for requests that return 403 Access Denied when the request comes from outside the owner's AWS account or AWS Organization. On August 19, 2024, AWS said the change was complete across all S3 APIs and all regions, including GovCloud and China. No application changes are needed.
The fix is narrower than some headlines suggested. The current AWS documentation still says owners are generally billed for 4xx responses. It lists specific error codes that are free, and the 403 exemption applies only to requests from outside your account or organization. Successful requests are still billed, so a bucket that allows public writes or reads is still exposed.
How to protect yourself on AWS
- Name buckets so they cannot collide. AWS recommends bucket names that are not predictable, such as a name with a GUID appended. You can also create buckets in your account regional namespace, which only your account can use. Avoid short, common names. If you delete a bucket, another account can later claim the name and receive traffic meant for you.
- Set an AWS Budget, but know its delay. AWS Budgets data updates up to three times a day, typically 8 to 12 hours apart. Alerts go to email or SNS. AWS warns that charges can pass your threshold before the alert arrives.
- Turn on Cost Anomaly Detection. Cost Anomaly Detection runs about three times a day on Cost Explorer data, which can lag by up to 24 hours. A new monitor takes 24 hours to start, and a newly used service needs 10 days of history before anomalies are detected.
- Alarm on S3 request metrics. S3 request metrics are opt-in, reported at 1-minute intervals, and billed like CloudWatch custom metrics. A CloudWatch alarm on
PutRequests,AllRequestsor4xxErrorsis the fastest AWS-native signal for this kind of traffic. AWS describes these metrics as best effort, so treat them as an early warning, not an exact count. - Cap your own Lambda functions. If your code is what calls S3 or other paid services, reserved concurrency sets a hard ceiling on how far a function can scale, at no extra charge. Setting it to 0 stops the function until you remove the limit. This limits runaway loops in your own code. It does nothing against outside requests to a bucket.
- Specify the region. Pocwierz's own advice: send S3 requests with an explicit region to avoid paying for redirects.
What would have caught it sooner
CostHex would not have caught this one. It reads usage every minute and alerts through Slack, Discord, Telegram or email with a link to the affected resource, using read-only access by default. Today it supports Cloudflare Workers only. AWS support is next, starting with Lambda, API Gateway and CloudFront, and is not available yet. S3 is not on the announced list. For an S3 bucket today, a CloudWatch alarm on request metrics is the closest thing to a minute-level warning, with Budgets and Cost Anomaly Detection as slower backstops.
Sources
- Developer's blog post: How an empty S3 bucket can make your AWS bill explode, Medium (Maciej Pocwierz)
- Link post with summary and Jeff Barr update, Simon Willison's Weblog
- News coverage of the incident and AWS response, The Register
- News coverage of the incident and billing change, InfoQ
- Announcement: Amazon S3 will no longer charge for several HTTP error codes (May 13, 2024), AWS
- Announcement: Amazon S3 no longer charges for several HTTP error codes (Aug 19, 2024), AWS
- Billing for Amazon S3 error responses, AWS documentation
- Amazon S3 pricing, AWS
- Managing your costs with AWS Budgets, AWS documentation
- Detecting unusual spend with AWS Cost Anomaly Detection, AWS documentation
- Monitoring metrics with Amazon CloudWatch (S3), AWS documentation
- S3 CloudWatch metrics and dimensions, AWS documentation
- Configuring reserved concurrency for a Lambda function, AWS documentation
- General purpose bucket naming rules, AWS documentation