Grab took advantage of the ability of Apache Kafka consumers to connect to the broker node in the same availability zone (AZ) introduced in Kafka 2.3 and reduced the traffic cost on AWS to zero for reconfigured consumers. The change has substantially reduced overall infrastructure costs for running Apache Kafka on AWS.
Grab has created a streaming data platform around Apache Kafka that supports all company’s products. Following Kafka best practices, their initial configuration used three replicas for each Kafka partition, spanning three different availability zones within the AWS region. The team responsible for this platform observed that the cross-AZ traffic accounted for half the cost of their Kafka platform as AWS charges for the cross-AZ data transfer.
Fabrice Harbulot and Quang Minh Tran remark on the cost considerations of the initial setup:
The problem with this design is that it generates staggering cross-AZ network traffic. This is because, by default, Kafka clients communicate only with the partition leader, which has a 67% probability of residing in a different AZ.
The overall cross-AZ traffic combines newly-published messages, data replication between brokers, and messages fetched by consumers.
The default consumer configuration where consumers fetch data from the partition leader (Source: Grab Engineering Blog)
Since Apache Kafka 2.3, it’s possible to configure consumers to fetch from partition replicas. That way, if consumers fetch messages only from the brokers in the same AZ, they can do so without incurring data transfer costs.
This feature requires that both the Kafka brokers and consumers are made aware of the AZ in which they reside. For Kafka brokers, the team configured broker.rack with the value of AZ ID (az1, az2, az3, etc.), which is different than the AZ name (1a, 1b, 1c, etc.) as the latter is inconsistent across AWS accounts. They also set the parameter replica.selector.class with value org.apache.kafka.common.replica.RackAwareReplicaSelector.
On the consumer side, the team updated the internal Kafka SDK to configure the client.rack parameter with the AZ ID based on EC2 host metadata so that application teams can enable the functionality by exporting an environment variable.
The custom consumer configuration where consumers fetch data from the closest replica (Source: Grab Engineering Blog)
After rolling out the new setup to some services, the team observed cross-AZ traffic costs go down and discovered some note-worthy side effects. Firstly, the end-to-end latency was increased by up to 500 milliseconds, which is to be expected considering most consumers fetch messages from replicas. Hence, the replication time is responsible for the extra latency. Any latency-sensitive data flows should ideally always fetch from the partition leader, even with the extra cost involved.
Secondly, in the case of broker maintenance, consumers fetching messages directly from replicas may experience brokers being unavailable during downtime, so they should wait/retry until the broker in the same AZ comes back online. Lastly, the team observed a skew in the brokers' load related to the number of consumers across AZs, which means that even distribution of consumers is essential to ensure a balanced load on brokers.