LinkedIn adopted Protocol Buffers for exchanging data between microservices more efficiently across its platform and integrated it with Rest.li, their open-source REST framework. After the company-wide rollout, they reduced the latency by up to 60% and improved resource utilization at the same time.
The LinkedIn platform employs a microservices architecture, and for years now, JSON has been used as the serialization format for over 50 thousand API endpoints exposed by microservices at LinkedIn. To help their teams build consistent interactions between services, the company created a Java framework called Rest.li, which became open-sourced.
The framework helps create servers and clients that use the REST style of communication and abstracts away many aspects of data exchange, including networking, serialization, or service discovery. It primarily supports Java and Python but can also work with Scala, Kotlin, JavaScript, Go, etc.
Data and Control Flow Between a Rest.li Server and Client (Source: Rest.li Documentation)
JSON is the default serialization format in Rest.li and has been selected due to its wide language support and being human-readable. The last property, however beneficial, introduces problems from the performance (and particularly latency) point of view.
Karthik Ramgopal and Aman Gupta, engineers at LinkedIn, share challenges with using JSON for inter-service communication:
The first challenge is that JSON is a textual format, which tends to be verbose. This results in increased network bandwidth usage and higher latencies, which is less than ideal. [...] The second challenge we faced was that due to the textual nature of JSON, serialization and deserialization latency and throughput were suboptimal.
The team has been considering alternatives to JSON, looking for a compact payload size and high serialization efficiency to reduce latency and increase throughput. They also didn’t want to limit the number of supported language stacks and enable gradual migration by integrating the new serialization mechanism into Rest.li. Finally, after a comprehensive review, they decided to go with Protocol Buffers (Protobuf), which scored the highest, based on the defined criteria.
The main difficulty around integrating Protocol Buffers into Rest.li was the dynamic schema generation based on the framework's custom schema definition system, PDL. The solution involved generating a symbol table that is used to generate Protobuf schema definition dynamically, but the method for delivering symbol tables varied depending on the type of client. Backend clients fetch and cache symbol tables on-demand, while for web/mobile apps, symbol tables are generated at build-time and included as versioned dependencies.
After changes to the framework were rolled out, the team gradually reconfigured the clients to enable Protobuf instead of JSON using HTTP headers. The result of Protocol Buffers adoption was an average increase in throughput by 6.25% for responses and 1.77% for requests. The team also observed up to 60% latency reduction for large payloads.
Latency comparison between JSON and Protobuf (Source: LinkedIn Integrates Protocol Buffers With Rest.li for Improved Microservices Performance)
Based on the learnings from the Protocol Buffers rollout, the team is planning to follow up with migration from Rest.li to gRPC, which also uses Protocol Buffers but additionally supports streaming and has a large community behind it.
See also the InfoQ Podcast: API Showdown: REST vs. GraphQL vs. gRPC – Which Should You Use?
Community comments
Symptom solved
by Mac Noodle,
Re: Symptom solved
by Rafal Gancarz,
Re: Symptom solved
by Mac Noodle,
Q
by Alexander Furer,
Re: Q
by Rafal Gancarz,
Was the performance benefit really worth it?
by Peter Thomas,
Re: Was the performance benefit really worth it?
by Mac Noodle,
Re: Was the performance benefit really worth it?
by Rafal Gancarz,
Symptom solved
by Mac Noodle,
Your message is awaiting moderation. Thank you for participating in the discussion.
If you're communicating between services with rest or grpc then that's called service oriented architecture those are not microservices. That typically means that they don't have the correct bounded context. Reduce the need to make those kinds of calls and use message Brokers to communicate. This is just a form of distributed processing which is been around for years. Microservices reinvisioning all of that not just deploying to kubernetes with smaller services.
Q
by Alexander Furer,
Your message is awaiting moderation. Thank you for participating in the discussion.
Kudos to LinkedIn engineers for adopting more than 10 years battle prooved technology !!!
I just wonder what were the alternatives when you say that
"Finally, after a comprehensive review
they decided to go with Protocol Buffers (Protobuf)"
Re: Q
by Rafal Gancarz,
Your message is awaiting moderation. Thank you for participating in the discussion.
Hi Alexander, I didn't include the list of other serialization technologies they evaluated, but these were: Flatbuffers, Cap’n’Proto, SMILE, MessagePack, CBOR, and Kryo (the source material, linked from the news, has more details, so please check it out). I grant you Protobuf (or gRPC, for that matter) is not a new tech, but the wheels of change turn slowly, and slower the bigger your company is.
I also believe they opted for a quick win by changing only the serialization format since Rest.li is used everywhere at LinkedIn, so replacing it with gRPC is going to be a much bigger endeavor.
Re: Symptom solved
by Rafal Gancarz,
Your message is awaiting moderation. Thank you for participating in the discussion.
Hi Mark, I agree with you that loosely coupled, asynchronous microservices architectures are preferable, but using synchronous interactions via APIs does not negate the architectural style, in my view. Many real-world systems will combine sync and async inter-service interactions, and I bet LinkedIn is similar in that regard. Your point re correct bounded contexts is an important one. I would also stress that in many cases, I observe teams focusing on the "micro" element too much, resulting in many more services than necessary and a system that has low cohesion, unnecessary coupling between services, challenges with end-to-end latencies, and system resiliency/availability.
Was the performance benefit really worth it?
by Peter Thomas,
Your message is awaiting moderation. Thank you for participating in the discussion.
The title says 60% but when you read the fine print - that applies to only large payloads and the average improvement does not seem like much.
Anyway, one of the reasons I dislike moving away from HTTP / JSON is because of how much harder it becomes to do quick (manual and exploratory) testing of APIs. Engineering leaders tend to forget how engineering teams operate and collaborate. For example: an engineer can quickly make an HTTP call using cURL, see what is happening and also share that cURL command on slack for inputs from a wider team. I see that agile way of working severely impacted when the architecture is changed wholesale like this - typically by the "platform team" who are in a "resume driven" or even "appraisal driven" mode of working and not really close to the end-users. Sorry for the rant, I've been on the receiving end of this too many times.
The other reason why I dislike these un-conventional approaches, which again most engineering leaders conveniently forget - is that these proprietary approaches don't scale well, on-boarding new engineers is hard, and engineers don't appreciate learning something that does not translate well to the job-market. And I've seen too many cases where the whiz-kids who created these frameworks unexpectedly leave the company and everything goes downhill after that. No matter how much the likes of LinkedIn can pride themselves on being able to re-invent a whole framework for app-development, all good things come to an end - and you simply cannot compete with popular approaches such as Spring-Boot, Quarkus, etc. Granted, Rest.li is open-source and has 2.3K stars - so that counters my argument somewhat. But is it really "mainstream"? I doubt it. I'm willing to bet that the engineers tasked with maintaining this framework are just going through the motions and their heart is not really in the job.
My question is - can you share more on how functional testing of these services is performed? Is it part of this "Rest.li" or is there some other in-house tooling? Is it at all possible to "explore" the service end-points as plain-text?
Re: Symptom solved
by Mac Noodle,
Your message is awaiting moderation. Thank you for participating in the discussion.
My point is that they're solving the symptoms are not solving the problem. Yes you can have different architectural Styles. It's been proven time and time again that having depend on another service which you will have to with this style is very problematic and you end up having SLA issues
Re: Was the performance benefit really worth it?
by Mac Noodle,
Your message is awaiting moderation. Thank you for participating in the discussion.
Yup. Most Engineers totally forget about total cost of ownership. They only focus on the pros and totally forget the cons. That's exactly what I'm facing
Re: Was the performance benefit really worth it?
by Rafal Gancarz,
Your message is awaiting moderation. Thank you for participating in the discussion.
Hi Peter, I can't comment on Rest.li's support for testing Protobuf serialization, but I can share a rather lengthy list of gRPC-specific tools, so please check these out (github.com/grpc-ecosystem/awesome-grpc#tools).