BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Podcasts Peter Bourgon on Gossip, Paxos, Microservices in Go, and CRDTs at SoundCloud

Peter Bourgon on Gossip, Paxos, Microservices in Go, and CRDTs at SoundCloud

Bookmarks

Peter Bourgon discusses his work at Weaveworks, discovering and imlemeting CRDTs for time-stamped events at Soundcloud, Microservices in Go with Go Kit and the state of package management in Go.

Key Takeaways

  • We’ve hit the limits of Moore’s law so when we want to scale we have to think about how we do communication across unreliable links between unreliable machines.
  • In an AP algorithm like Gossip you still make forward progress in case of a failure. In Paxos you stop and return failures.
  • CRDTs give us a rigours set of rules to accommodate failures for maps, sets etc. in communication that result in am eventually consistent system.
  • Go is optimised to readers/maintainers vs. making the programmers’ life easier. Go is closer to C than Java in that it allows you to layout memory very precisely, allowing you to, for example, optimise cache lines in your CPU. 
  • Bourgon started a project called Go Kit, which is designed for building microservices in Go. It takes inspiration from Tiwtter’s Scala-based Finagle which solved a lot of Micoservice concerns. 
  • Go has a number of community-maintained package managers but no good solution; work in ongoing to try and resolve this.

Work at Weaveworks

  • 0m:36s - Bourgon joined Weaveworks to work Weave Scope, a network mapping and monitoring APM solution.
  • 1m:37s - Later worked on Weave Flux, an early stage product for change management to automate as much of this as possible.
  • 2m:41s - Also worked on Weave Mesh, a Gossip and CRDT framework for building distributed applications. It was extracted from Weave Net, which is one of the oldest container SDNs.

Gossip vs. Paxos

  • 4m:00s - Gossip is a method for communicating between nodes on network. There are lots of messaging patterns - peer to peer, request/response, pub/sub and so on. Gossip is another class of pattern where communicating nodes send information to each other.
  • 4m:39s - There are various ways you can do this, but perhaps the most well known is rumour gossip, which is a class of dissemination protocols where information propagates outwards from a single node; you tell the fist node, that node tells two other nodes and so on. There are other forms of gossip, for example aggregate gossip.
  • 5m:40s - We’ve hit the limits of Moore’s law so when we want to scale we have to think about how we do communication across unreliable links between unreliable machines.
  • 6m:16s - Gossip is a way of communicating between nodes which is fundamentally unreliable. Paxos is a highly consistent consensus algorithm - a CP algorithm in the CAP theorem sense. Gossip is the other end of the spectrum - an AP algorithm.
  • 7m:56s - In an AP algorithm like Gossip you still make forward progress in case of a failure. In Paxos you stop and return failures. Paxos is desirable where data is sensitive, for example in banking or health insurance, but may be less suitable in some other cases like web systems.

CRDTs at SoundCloud

  • 9m:29s- CRDTs (Conflict-free Replicated Data Types) are data types like a list or a map which are a class of a solution to data-related problems.
  • 10m:31s - CRDTs give us a rigours set of rules to accommodate failures for maps, sets etc. in communication that result in am eventually consistent system.
  • 11m:52s - At SoundCloud designed a CRDT system for time-stamped events in 2012 called Roshi. For a long time it powered the SoundCloud stream and activity feeds.

Go

  • 14m:01s - Go is a statically typed, compiled language designed for programming by large teams.
  • 14m:40s - Go has a strong focus on maintainability, so the code is designed is to be read.
  • 15m:14s - Lots of languages aim at simplicity but I like this quote from user “pyon” on lobste: “For a Go user, “simplicity” means that the operational interpretation of a program fragment is unique, straightforward and never in question after you’ve read the code. From this no-nonsense point of view, higher-order functions are already suspicious, and fancy ones like Clojure’s transducers and Haskell’s lenses/traversals are complete abominations.”
  • 16m:11s - He goes on to say that “For a Clojure user, “simplicity” means ontological parsimony. From this point of view, data abstraction is undesirable, because it grows the number of kinds of things that exist in your program.” So in other words Clojure users are thinking more in the data domain.
  • 16m:48s - So Go values simplicity in the sense of knowing that the code does and not being fooled by magic. A good contrast to Go is Ruby where magic via higher order abstractions is the order of the day.

Go in large teams

  • 17m:31s - All about thinking about who is consuming the code you are writing. Go is optimised to readers/maintainers vs. making the programmers’ life easier.
  • 18m:59s - Programming in Go can be a little frustrating because of this but when you come back to the code you can see exactly what is happening.
  • 20m:07s - There is tooling to support this as well, such as Gofmt; there is one canonical way to format your Go code and it is mechanically enforced.
  • 21m:20s - Go is closer to C than Java in that it allows you to layout memory very precisely, allowing you to, for example, optimise cache lines in your CPU. It’s not as elegant as Rust for this, but you can do it.

Go and Java package management

  • 22m:11s - Go came from Google, which has a mono-repo where every bit of code they compile lives. In Go the idea of dependencies was quite simple. Go get allowed you to go and get a dependency but you couldn’t specify a version of the library.
  • 23m:48s - When you are building a large system like Docker or Kubernetes you need to be able to break an API so not being able specify a version is too limiting.
  • 24m:14s - It was left to the community to solve this, so lots of people tried to solve this problem independently so package management in Go was a mess.
  • 25:31 - This same problem exists in Java 9’s Jigsaw. You need a standard to specify which version you are using. Rust handles this better.
  • 26m:27s - Package management is a hard problem. There are sub-problems that don’t have correct solutions. Sam Boyer’s long blog post “So you want to write a package manager” describes some of this.
  • 28m:15s - About a year ago the Go core team realised they needed a solution to this. Bourgon took the project on with some prominent Go users to try and build a proper solution to this based on the various community efforts.
  • 30m:45s - An alpha pro type will be available soon, and a prototype will also hopefully be in included in Go 1.9. Once it is open-source feedback is welcomed.

Microservices in Go with Go Kit

  • 32m:38s - Go Kit is designed for building microservices in Go. It takes inspiration from Tiwtter’s Scala-based Finagle which solved a lot of Micoservice concerns.
  • 33m:55s - Go Kit includes things like circuit breakers and client-side load balancers. It’s also about ways to implement DDD principles, dependency injection, component models and so on that are very familiar to Java programmers.

Logging and tracing in a distributed environment

  • 35m:21s - We’ve had a blossoming of orchestration tools like Kubernetes. and monitoring tools like DataDog and Prometheus, and open-tracing, Zipkin and so on.
  • 35m:52s - For logging though this is less of a solved problem. You probably use Elasticsearch for logging. Syslog, it feels kind of like it’s still in a dark age.
  • 37m:31s - We have tools for this - Logstash and so on - but these are kind of point solutions. They are not end to end solutions. We need a different answer here that is more in tune with Micro-services and cloud-native logging concerns.

Links

Companies Mentioned

Products and libraries mentioned

More about our podcasts

You can keep up-to-date with the podcasts via our RSS Feed, and they are available via SoundCloud, Apple Podcasts, Spotify, Overcast and the Google Podcast. From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.

Previous podcasts

Rate this Article

Adoption
Style

BT