Agoda engineers have developed an internal S3-compatible proxy called Storefront to improve load balancing, reliability, and operational control for large-scale object storage traffic in the company's data platform. The proxy sits between internal services and backend object storage systems, routing requests while addressing limitations the team observed with DNS-based load distribution used by S3-compatible endpoints.
Agoda relies on object storage for data processing and analytics workloads, including pipelines that read and write large volumes of files. According to the engineering team, the S3 endpoints exposed by their storage provider, VAST Data, use DNS round-robin to distribute traffic across multiple virtual IP addresses. Application clients often cache DNS responses, which can result in repeated requests to the same backend node. This behavior led to uneven load distribution, creating hotspots in which certain nodes received disproportionate traffic while others remained underutilized.
Uneven load distribution with DNS round-robin (Source: Agoda Blog Post)
To address this limitation, the team introduced Storefront as a reverse proxy that actively distributes S3 requests across backend nodes. The service is implemented in Rust and built on top of Pingora, an open-source proxy framework developed by Cloudflare. Instead of relying on DNS resolution to balance traffic, Storefront evaluates backend availability and request load in real time before routing requests. Early implementations used a least-in-flight requests algorithm, which the team later refined with latency-aware scoring to improve distribution under production workloads.
The proxy also introduced operational safeguards to improve reliability. Engineers added IO timeouts to handle cases where certain S3 clients failed to fully consume HTTP responses, which could otherwise exhaust backend connection pools. Storefront isolates cross-data-center traffic into dedicated backend pools and optimizes handling of the HTTP Expect: 100-continue header to reduce latency for object upload requests.
Storefront integrates credential-less authentication by automatically identifying the calling Kubernetes pods and applying access controls internally. This approach centralizes permission management, enabling services to securely access object storage without directly handling credentials, reducing operational complexity, minimizing the risk of credential leaks, and simplifying compliance across large-scale distributed workloads.
As emphasized by Desmond Xu, Technical Lead at Agoda
Storefront evolved from a simple reverse proxy into a core component of our data infrastructure.
In addition to request routing, Storefront incorporates operational capabilities such as credential-less authentication by identifying calling Kubernetes pods and applying access controls internally, reducing the need for services to manage storage credentials directly. The proxy also exposes telemetry via OpenTelemetry, including metrics on performance, resource utilization, traffic patterns, and S3 API usage.