Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Guides Architectures You’ve Always Wondered About

Architectures You’ve Always Wondered About


As the flagship track at QCon, Architectures You've Always Wondered About showcases real-world examples of innovator companies pushing the limits with modern software systems. At recent conferences in San Francisco and London, the speakers clearly showed what "scalable" can really mean, from a trillion messages to exabytes of data. This eMag brings together several of these stories and hopefully provides advice and inspiration for your future projects.

At Dropbox, it is crucial to provide high performance and availability for the data storage solutions they provide. Facundo Agriel goes deep into the details about Dropbox’s exabyte-scale blob storage system for Magic Pocket. In addition to all the software that handles millions of queries per second, the physical hardware design must also be considered. Because each storage device contains over 100 drives, and hardware devices can fail, the system has to be self-healing. The architecture has to consider this and many other factors to achieve Dropbox's performance targets while remaining cost-effective.

Monzo serves 7 million banking customers daily, on an architecture of thousands of microservices that they deploy hundreds of times a day. Suhail Patel covers some of the specific technologies they use, including Cassandra and Kubernetes, and how a focus on platform engineering and developer experience has allowed them to scale up, even with a relatively lean engineering team. Because nothing ever goes entirely as planned, there were some mistakes and incidents along the way, but those led to some valuable lessons and a better system in the end.

Cloudflare provides DDoS protection for websites around the globe, and their services have relied on Kafka to send over 1 trillion messages. However, decoupling those microservices, and the teams that build them, required dedicated effort. Changing the message format from JSON to Protobuf, and the development of custom tools, led to more structured messages and better fault tolerance, which in turn reduced the cognitive load of teams and increased adoption. Andrea Medda and Matt Boyle explain how balancing technical and socio-technical needs is necessary to create a successful solution.

The eMag wraps up by looking at a new architecture pattern, which Bilgin Ibryam calls cloud-bound applications. This is the evolution of cloud-native architecture, and moves the abstraction of compute-centric concerns to an abstraction of application integration concerns. The CNCF project Dapr provides APIs that speak to these integration needs and allow architects to design a clean separation around core business logic.

Free download

The Architectures You’ve Always Wondered About include:

  • Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System - A horizontally scalable exabyte-scale blob storage system which operates out of multiple regions, Magic Pocket is used to store all of Dropbox’s data. Adopting SMR technology and erasure codes, the system has extremely high durability guarantees but is cheaper than operating in the cloud.
  • Banking on Thousands of Microservices - Lessons learned building a banking platform, starting from technological choices like using Cassandra and Kubernetes in the early days to maintain the speed of execution through platform engineering and developer experience. With some mistakes and incidents along the way.
  • Tales of Kafka at Cloudflare: Lessons Learnt on the Way to 1 Trillion Messages - Cloudflare uses Kafka clusters to decouple microservices and communicate the creation, change or deletion of various resources via protobuf, a common data format in a fault-tolerant manner. The authors suggest investing in metrics for problem detection, prioritizing clear SDK documentation, and balancing flexibility and simplicity for standardized pipelines.
  • What Are Cloud-Bound Applications? - The increasing adoption of application-first cloud services is causing applications to blend with the cloud services at levels much deeper than before. The runtime boundaries between the application and the cloud are shifting from virtual machines to containers and functions.

InfoQ eMags are professionally designed, downloadable collections of popular InfoQ content - articles, interviews, presentations, and research - covering the latest software development technologies, trends, and topics.