Transcript
Ross McFarlane: In 2021, the UK's banking regulator published this statement in which they expressed their concerns that with banks moving to the cloud and a limited number of cloud providers available, there were growing concentration risks that if any one cloud provider had an outage, they were going to take with them a big chunk of the UK's financial services. The key message from this document is that banks must be able to exit a cloud. One of the largest banks in the UK, one of our customers, heard this and said, we need a multi-cloud strategy. In turn we looked at this and thought, we need to be running on multiple clouds. Who are we? At Form3 we power account to account payments. We connect banks and financial institutions to payment schemes in the UK, Europe, and the U.S. If you've made a bank transfer recently, there's a pretty good chance that we handled it. My name is Ross.
I'm going to hand over to Kev, our VP of engineering, who's going to tell you how we got our platform running on three clouds at once.
v1 Architecture - Built on AWS to Move Fast
Kevin Holditch: Before we move on to the multi-cloud architecture, I just want to take you on a little bit of a journey through time. Back around 9 or 10 years ago in Form3 we were just starting out. The world was a very different place back then. We only had between 4 and 10 engineers for over the 1 to 2-year period. We built an architecture that enabled us to move quickly. We knowingly coupled ourselves to AWS as tightly as we could because we wanted to offload the responsibility of running technology to a large cloud provider so we wouldn't have to run it ourselves. A high-level architecture was because we're an API we have a load balancer where traffic comes in. We use Java to write microservices running in Docker on AWS ECS, which is obviously their proprietary Docker container runtime. Because we have asynchronous workflows, we needed some kind of message bus to coordinate those workflows, and for that we used AWS SQS.
To keep all of our data safe we stored our data in a Postgres database running on RDS. This was great for us. It enabled us to innovate, go to market quickly, and offload a lot of those responsibilities such as backing up our database, running it multi-AZ to AWS so we didn't have to worry about it. What changed? Ross obviously introduced this talk by telling you what changed where our customers started to need the multi-cloud strategy.
v2 Architecture
We were successful as we were growing, and those requirements were pushed down onto us. That caused us to take a step back and think about building a v2 platform that would be able to run across multiple cloud providers. What are some of the design goals we had for this platform? The first was we wanted it to be active-active-active. On the v1 architecture I just showed you, we got really good at running workloads where they could fail when a request was retried in another availability zone. We got really proficient at handling availability zone failure. We thought we could take that paradigm, and as crazy as it sounds, move it up to the cloud level. What if we could treat a cloud provider like it was an availability zone? Could we use some of those learnings from our v1 platform and apply them to multiple clouds, so if any cloud fails the world would carry on running like nothing had happened.
We also took a long hard look at the technologies that we'd use that had turbo boosted us for our first once, two years of Form3, and as I mentioned, they were completely locked into AWS. We needed to change that because we wanted to run across multiple cloud providers. We picked some technologies that were cloud agnostic and we really moved the paradigm from allowing the clouds to run our technologies for us to running them ourselves with the Form3 platform team. Then that touches into the third design goal in that we now were almost going the complete opposite where we wanted to avoid cloud-specific technologies. The reason for this is because we wanted the Form3 engineers who were building products to run payments and allow account to account payments. We wanted to run that same software on any cloud provider. We didn't want to be in a world where we needed one version of software for AWS, a different version for Google, and a different version for Azure because that would just be a maintenance nightmare.
This is the v2 architecture that we landed on. How it looks is we have a Kubernetes cluster in Google, a Kubernetes cluster in Azure, and a Kubernetes cluster in AWS. We have a load balancer which provides our API endpoint into any cloud so our customer is able to send their request to any cloud provider, and they're able to do client-side load balancing. If for example they try to send a request to GCP and they don't hear back they can send the request to Azure to make sure the request goes through. We moved away from Java and over to Go for our microservices. There are a few reasons for that is that we found Go to be much more lightweight to fit much better in the microservice architecture we were using, and provided a much smaller deployment footprint. The biggest thing I think Go has going for it is that you can't write magic code.
Especially in the microservices world, when you're moving from repository to repository, you can just read through all of the code including the dependencies and it's really easy to understand. Then the secret sauce for this architecture was that we offloaded the data storage challenges to two vendors, and that is NATS JetStream. NATS JetStream is a message broker written in Go, very high performance, that's designed to run across multiple nodes. We actually run a single logical NATS JetStream cluster across the three Kubernetes clusters. If you send a message in GCP you could receive that message back in Azure or vice versa. For data storage we chose CockroachDB. CockroachDB uses a Postgres compatible dialect, so it allowed for us to do an easy migration from Postgres we were using in v1 over to Cockroach in v2. The real key challenge that we're offloading to these vendors is keeping our data safe.
Writing a write consistent datastore that runs across multiple nodes is a real challenge, but luckily, it's not a challenge that at Form3 we had to solve because these two vendors could solve it for us. With all of this in place our applications could just run in a Kubernetes cluster locally to the cloud and just use NATS and Cockroach to send messages and save their data, and they almost didn't have to worry about how it was working under the hood. To connect all of this together we had to privately network our Kubernetes clusters together using highly available redundant links. Just to say for the challenges coming up, any pod in any Kubernetes cluster can talk to any pod in any other Kubernetes cluster via pod IP address. That was an underlying network that we put in place.
Three Peaks Challenge in Building the Architecture
I've almost done a Quentin Tarantino and shown you the finished solution. I just want to rewind a bit now and explain to you some of the challenges that we faced in building this architecture. There were many, but this is like our Three Peaks Challenge, if you like, are the biggest challenges that we had. The first one was bootstrapping this Cockroach cluster across the three Kubernetes clusters. As mentioned, each Kubernetes cluster is completely independent, but in order to run Cockroach across the three we wanted a nine-pod setup that is three Cockroach pods in each cloud. For this to work, you need to tell each Cockroach pod where all of the other pods are so they can connect together and then bootstrap the database and then become a single unified database. Within a single cloud this is actually quite straightforward because Kubernetes has a really good primitive for this called a headless service, where it basically gives you like a DNS name that tracks the pod IP address.
Within a cluster you can just talk to each other on like Cockroach-0.svc.cluster.local, as an example. The challenge really comes in how do we address a pod on another cloud. We can't hard code an IP address because the IP addresses come and go as the pods recycle. Our solution to this was that we decided to use DNS. Another challenge to this might be, how do you know what the DNS address is of another cluster? Our setup for this was within each Kubernetes cluster you have a built-in core DNS server that can resolve the local names, the .svc.cluster.local names that are built in out of the box. Then the secret sauce, the real engineering revelation we had was that, why not just introduce a new pretend suffix where we add the cloud into the address. To walk through an example, we can see that if Cockroach pod 2 on Google Cloud wants to talk to Cockroach pod 1 on Azure, it can address it using this special address where Azure is inserted into the DNS name.
To make this work, the core DNS within each cluster has some special rules in it where if it sees the cloud in the DNS name it forwards the request over to the other cloud. Why it can do that is because we've installed a network load balancer in each cloud on a static IP address, and that network load balancer just points to the core DNS server running in the respective Kubernetes cluster. With all of that in place, you can see that if I just walk through an example, the Cockroach pod 2 can address Cockroach pod 1 in Azure using the name you see on the screen. When it passes it across we have to do some funky rewrite rules within the cloud to basically strip off the Azure bit because we haven't really done anything special in Kubernetes itself, we still use the built-in DNS names.
We rip off the Azure part from the DNS name and then we just resolve using the normal headless service name, get the IP address, and then rewrite it back to the caller. With all of that in place that now enables us as if by magic just to refer to any pod in any other cluster using the normal DNS name just with the cloud inserted in it. I thought that was a really neat solution where we're building on top of the primitives that Kubernetes already provides.
The next peak or the next challenge we had is that each cluster thinks it's alone and that's because it is alone. We made a decision to run three separate Kubernetes clusters. The reason for that is we wanted to increase our fault tolerance. If you run a single logical Kubernetes cluster across the three clouds, you would have had a single point of failure, or if that control plane went down, you would have lost the whole cluster. We didn't want that so instead we knowingly took the decision to have three separate clusters. The challenge with that now is that some of the built-in primitives obviously only know about their local cluster. With this CockroachDB that we've just been speaking about that we want to run across the three clouds, that has nine Cockroach pods. Underneath the bonnet Cockroach uses an algorithm called Raft for data consensus. The Raft algorithm says that for a nine-node cluster you need five pods to be alive otherwise you have a database outage, which is obviously very bad.
Now, what you want to do with a workload such as this that you need to keep running, a high resiliency workload, is you want to nudge Kubernetes to tell it what it can and can't do when it's doing node rollouts and rescheduling pods. In order to do this Kubernetes actually provides a primitive out of the box called a pod disruption budget. What a pod disruption budget really is, is you can tell each cloud, ok, at maximum, you can only take down one of my pods if you're rolling a node or moving a pod from one node to another node. Remember this is actually a single database, so the best we can do with the built-in primitives is the setup you see on the screen to say per cloud you could have a max unavailable of one pod. Now that would mean that Kubernetes would be well within its rights to take one pod offline in every cloud, and that would be a valid configuration.
Now we're running very close to our resiliency tipping point where we need five nodes running, otherwise we're going to have an outage. In fact, if we got ourselves into this state, if we lost a cloud from here, we'd actually drop down to four nodes and we'd have a total database outage. That was obviously not going to work for us. Our solution to this problem was we built a new primitive called a cross-cluster pod disruption budget. We termed it X-PDB. With this in place it allows you to span a pod disruption budget across n clusters. In our situation we spanned it across three clusters, but with this in place we can now say only allow one pod to be taken down at any one time across the three clusters. If the Kubernetes in Azure is trying to move a pod, it won't be able to if there's one that's been taken down in Google as an example.
This gives you much higher resiliency and allows you to run with a lot more nodes online at any one time. Again, it was just building on the primitives that Kubernetes already provided and extending them across multiple clouds. If you're interested in this and want to dive a bit deeper, my colleague Moritz actually did a talk on this, you can go to that link to watch the YouTube video. We've also open sourced this operator on GitHub.
The last peak, this peak is actually a day 2 challenge that we had, which was after we built the platform and we were running it for a little while, we realized that some of the decisions we have made were coming back to bite us in terms of trying to maintain it and keep it online. You can imagine the world we're in, we are a payments provider. We have to keep everything updated all of the time, which basically means always rolling nodes every time they've got a CVE with the latest operating system. Our initial setup that you can see on the screen, per Kubernetes cluster we had multiple node pools where we grouped different technologies like NATS, Cockroach, Go, because we're an Infrastructure as Code shop, every one of those node pools was defined in code. You now imagine if you wanted to do an operating system update you'd have to do a PR per node pool per cloud, and that would be a lot of pull requests.
You could say you could change them all in one go but that would also be very disruptive because all the node pools would be rolling at the same time which you probably wouldn't want to do because you want to try and keep things more stable than that. This was a problem that was really killing us in terms of trying to maintain the system. I haven't even explained the problem, actually it was a lot worse than this because every environment doesn't live on its own. We have development. We have staging. We have production where we promote changes through. You can imagine all the PRs are just multiplied. It actually gets even worse because we run platforms in different geographies for different product sets for different customers. You can basically think keeping all these node pools up to date was basically a full-time job to create all these PRs and merge them and make sure everything's merged ok.
This is what I'd like you to consider is what is the right level of abstraction when you're modeling something as Infrastructure as Code. Because I think infrastructure as Code is great but the challenge we made is we were too granular with how we're modeling it, and it put ourselves into this bind where trying to keep things up to date was just a nightmare. The way we got out of this, or our solution to this, was we wrote our own operator to manage our node pools. The operator is called the cluster lifecycle operator or CLO. What it does is it manages all the node pools and then we can just define our own interface with the CLO to tell it how to operate. We can build in lots of rules like how many node pools you can run simultaneously, how quickly to roll them out and all of that good stuff.
Then at the top level we can just tell CLO what image we want it to be running across all of the node pools, and then it can just manage that for us. What was hundreds and hundreds of PRs becomes one PR per platform now, which is obviously much more manageable. That was a real game changer for us.
Does It Work?
So far, all this theory, that's all great and all, but you probably are asking yourself, does this multi-cloud thing really work? It looks quite far-fetched. The story I have for you was back in the summer of last year, I was on the train on the way home and the football team I support had actually just hired a new manager. The sucker I am really excited about this. I've got my phone out because I love watching the YouTube channel where they analyze the tactics and stuff. I'm sure you've seen it, but these YouTube channels today they're so high tech it's almost like watching a TV channel. There's like a bar across the bottom scrolling with the news. There's multiple people on the screen. There's music. There's a background. It's all very impressive. I got my phone out and then opened the channel and there was no scrolling bar across the bottom.
There was no multiple people on the screen. There was no background. There's just one guy glitching in and out, and I was like, what's going on here? It's like amateur hour. Then he was like, you have to bear with me, the streaming software I'm using is not working at the moment. I was like, ok, that explains it. Then I read in the comments what was going on and it was this, Google Cloud was having one of their largest outages which was around about three or four hours. Then it's that feeling that hits you, you're like, oh? I close my phone, work laptop out, you just have to go in and check the system and see what's happening. It was one of those really proud career moments where I could open my laptop, look at our alerts, and all that we had was a low-level alert where we had some pods crash looping in GCP. All of the payments were continuing to flow and they're picked up by the other clouds. That's where our decision to treat a cloud provider as an availability zone really got proved out.
Across the Pond - Resilience Story
Ross McFarlane: In 2022, having established ourselves in the UK and in Europe, Form3 decided it was time to make it in America. When I joined the team later that year, they were busily building out our triple active multi-clouds set up in the image of our UK environments, but our sales team in the States was having a hard time pitching this to our potential customers. What we came to realize is everything's bigger in America. If you look at it from a U.S. perspective, our state-of-the-art solution almost starts to look a little bit quaint. You can imagine how our customers in the States might look at this and see and think that we're putting all of our eggs in one geographical basket. Our initial reaction was to push back on this, and my predecessor as architect in the U.S., Sam Tavakoli, did some great work looking at all the different types of disasters that might befall us in the U.S., the natural, man-made, far-fetched, and he concluded that really there was no set of circumstances that would cause us to lose more than one cloud or data center at a time.
Really, we could just lift and shift our architecture from the UK to the U.S., and run it without a hitch. It was tempting to get a bit cynical about this, to imagine that our prospective customers were misguided in some way, or that the disasters they were preparing for were just asteroids of the mind. Really, we weren't telling the right resilience story for the U.S. market. I think any market or industry you look at has got its own set of resilience stories, its own mantras and metaphors that they live by. Our customers in the U.S. were used to talking about geographical resilience. They were used to running things on the East Coast with a backup on the West Coast. As a new entrant to this market with no proven experience in the U.S., we didn't want to give our sales team an extra hurdle to clear by trying to pitch this novel solution.
Now, we could conceivably take our triple active multi-cloud and try and spread it out across the U.S. This is not an option for us because everything's bigger in America including latency. Every time we process an inbound payment from scheme, we have to validate it, store it, orchestrate it across a bunch of services. Send it to the customer for approval. Then once it's approved, we'll send the message back to scheme, and then we await one more message so that we can set the final status on that transaction. Every time we write to the database and every time we put a message on that, we have to get quorum across the clouds. If we're spread out across the U.S., we could easily burn through our SLAs just waiting for electrons to cross the continent.
Thankfully, although our customers wanted to have geographical resilience, they also understood the associated tradeoffs. They were used to seeing a bit of downtime and accepting the possible loss of some data in the case of a disaster. Working with a non-zero RTO and RPO, we could go back to first principles and find a solution that fit the market and kept our cost sensible in the U.S. We evaluated a bunch of options, ruled most of them out for various reasons, and in the end, settled on an architecture that was probably more familiar to our customers in the U.S., just a good old active standby. We were going to run in AWS on the East Coast, and if the proverbial hits the fan, we'll flip over to GCP in the West Coast. In order to move a running payments platform between clouds, we've got to move three things, services, data and connectivity.
The services are the easy bit, so let's talk about data. We've got three datastores to worry about. We're using Cockroach for most things. We've got to keep that consistent because we're doing a lot of things like duplicate checks. OpenSearch powers our read model for payments. This is what you hit if you do a GET request to /payments. Then, Postgres is a little bit of a vestige from our v1 platform. This is used by our audit service, at least for now. Annoyingly, each of these has at least one replication mechanism, each of which requires its own setup and management. It's exacerbated by OpenSearch and Postgres because we're using hosted cloud services to run these, where we don't have exact control of which software version they're running, or we can't install plugins in some of them. This was going to be a bit of a pain to run.
We, with limited time, got ruthless. We decided, first of all, to do away with Postgres and the associated service on the disaster recovery site. Things happening over there would still be auditable, but only once you got back to AWS. For the other two, we decided just to lean into the RTO and RPO that we had available and do the most basic thing, which is to rely on backups. Cockroach and OpenSearch write their backups to S3. Kev's team provided us with a tool that pushes those over into GCS. Then if we have to fail over, we'll restore those from GCS into those databases. For the amount of data that we've got to shift and the frequency of backups that we can take, this gives us an acceptable RTO and RPO. Of course, after a disaster, although our customers are happy to accept some possible data loss, once we get them back up and running, they're understandably going to ask us what's happened to the payments they might have missed.
We didn't want to have teams trying to deal with this manually or poke around in the data. We've set up a process whereby we can ingest reports from the scheme to fill in some of the missing payment data. This gave us a new problem. Our payments are identified by UUIDs, but in the scheme, the ID space is much smaller. When we send the payment to scheme, we've got a process by which we map our IDs onto the scheme IDs. That process is deterministic, but it is lossy. We store that mapping in the database, send payment to scheme. Now, if after a disaster, we've forgotten about that mapping, then when we ingest the report from scheme, we're not going to know what scheme is talking about and we can't inform the customer of what's happened to their payment. What we decided to do here is to meet our customer halfway. They'll send us IDs of their recently submitted payments, which allows us to regenerate that mapping and inform the customer of the outcome.
That's data. Let's talk connectivity. The first payment scheme that we put live in the U.S. is TCH RTP. We connect into that using IBM MQ over VPN. The scheme has two physical locations, and we as participants have two logical sites, each with an MQ broker. We cross-connect them into scheme. The naive implementation of that would just be to say, let's put one on each cloud, except we're not going to be running in GCP very often, which leaves us with a single point of failure in AWS. In practice, we found that running MQ in Kubernetes needs a bit of effort. Losses of connectivity can be quite slow to recover. We have to be really careful about rolling nodes, software updates with MQ. What we decided to do instead was to run both of our MQs in AWS, and if the worst happens, we'll flip them over into GCP.
The process for flipping those over at scheme is that you fill out a Word document with the IP addresses of your MQ brokers and then email that to them. They'll pick that up in business hours and they'll process it four hours later. It's an admin process. It's not really intended to be used for emergencies. What we've done instead is have each of these pair of MQ brokers present the same IP addresses and then we flip between them using BGP. When we want to get from AWS to GCP, we stop publishing the routes to AWS, start publishing routes into GCP.
Now we move from the technical to the tactical. Under what circumstances are we actually going to initiate a failover from one site to the other? We settled on this question, which is now at the top of our failover runbook, which is, is it likely that AWS will take significantly longer to recover than our predicted failover time? Significantly longer because when we fail a customer over, they're going to want to get back to AWS. It's physically closer to them. As I mentioned, we're not running every service in the DR site. Our predicted failover time, this couldn't be a guess, this had to be empirical from measured rehearsals. Finally, likely, based on what we know about this incident and what we've seen in previous AWS incidents. My friend ChatGPT and I, went through the complete history of AWS incidents and looked at the outage duration and the affected services, and we concluded that based on our predicted failover time, there was one of those incidents where we might have benefited from failing over. A lot of the chat prior to the service going live was, we're never going to do this for real. We'll demonstrate it to the customer, but surely, we'll never actually use this thing.
The service went live in October of 2025 and enjoyed two silky smooth weeks of operation before this happened. AWS were having a bad day. We could see the list of affected services growing minute by minute. We could see impacts across the industry, but we were getting away with it. I think our choice to be cloud agnostic meant that things affecting AWS's control plane didn't really hit us until it did. Our VPN went down, we lost our connection to scheme, payments stopped flowing, so we had a decision to make. We could sit on our hands and wait for AWS to fix the issue, or we could initiate a failover and get our customer back up and running, yay, save the customer some time. As I mentioned, we will eventually have to get the customer back to AWS, so we've got to price that in, which reduces the benefit of the whole failover thing.
Of course, we don't know how long this underlying incident is going to last, so if we take too long to act or if it's shorter than we thought, we could actually be making the situation worse for our customer. This left us stuck between two bad options. Over the years, we've become pretty good at handling incidents. Our engineers take their on-call duties really seriously. We've got an excellent team of incident managers. They're all biased to action, want to fix stuff for the customer, but in this situation, they felt trapped between two bad options. In the end, AWS recovered. We didn't fail over. If we had failed over, the customer would have experienced the longer outage overall. We probably made the right decision, but it really didn't feel like it. If we'd been running our triple-active UK-style multi-clouds, we could have just laughed this off and spent our time writing blog posts about how resilient we are. Instead, we had to go back and look at our options and say, the only way that we're going to retain any control in this situation is if our whole failover process is faster.
We realized that data restore is a real bottleneck for us. We've set a whole bunch of things in motion. I'm going to give you a speed run through some work in progress that we're looking at now to make this better. The first thing is we're adding logical data replication in Cockroach between our clouds. Logical data replication is bidirectional, asynchronous, SQL-level. It doesn't guarantee consistency, but we'll be safe so long as we're only writing to one cloud at a time. It also has some good conflict detection so that we've got a bit of peace of mind there. The second thing, then, is rather than try to deal with all the other aggregating services and replicate each one individually, we're just going to replicate our events bus. We'll replicate our NATS streamed over into the other cloud, and then those services can consume and write to their datastores locally.
These two changes will vastly reduce our RTO and RPO, so we're really just limited then by how long it takes to do that BGP switch and reconnect to MQ on the other side. Our customers should be pretty satisfied by this, but in turn, they need to satisfy their regulators. One of the things that they'll need to do there is demonstrate their disaster recovery capacity in production annually. That's fine for them, but we run a multi-tenanted platform, so we need a way for them to do that without disrupting other customers on the platform. We're putting this change in place now to allow customers to shift their processing from one cloud to the other while maintaining their connection to scheme through AWS. We're doing that by splitting the NATS subjects between our MQ adapter component and the scheme gateways by customer, so that the gateways in each cloud can be configured for which customers they're interested in and just read those messages. Again, we're relying here on the fact that our data is segregated by customer, and so long as the customer is only active on one site at a time, we're safe in terms of consistency.
Behind the scenes, we've made some NATS changes. Services that act on individual transactions will only see events from the same cloud, whereas aggregating services will get events from everywhere. What this means is that individual transactions will always be processed to their conclusion on one cloud, but then after replication will be visible everywhere, so when the customer flips back from AWS to GCP, they can see the payments that they created in GCP. There are a couple of trip hazards with doing this sort of thing, particularly the asynchronous database replication, and one of those is outboxes. We use the outbox pattern in a load of services, where if you've got something like an API that writes a change to a payment database, say, it will write that change to the payment table. Then in the same transaction, it will write a row into an outbox table containing all of the events that need to be emitted relating to that change, and periodically we run an outbox processor that reads from the outbox table and emits an event over NATS.
Finally, then, it will delete that row from the outbox table. That's fine, but if you do it in duplicate with asynchronous replication, there's a danger that events that are written on one cloud get replicated and then get processed on the wrong cloud, meaning that event ends up in a service where that customer is configured off and nothing will happen. What we decided to do here is to keep the replication for the resources themselves, for the payments, but break the replication for the outboxes, meaning that the events will only be emitted on the clouds where the things have happened. We've got a similar issue here with timers. We do a number of things where we have long-running timers. We record the fact that something needs to happen maybe days hence, and so we want to make sure that those things get replicated so that they follow the customer around.
We then need to configure the processing part to make sure that they only act on timers in the cloud where that customer is configured on. Of course, this is all cool, but most of the time we're just going to have all of our customers running on AWS with just the peace of mind that we've got this really quick replication going on, we've got a really warm standby site, and a much faster failover time. One last thing I should mention here is that this last step I've mentioned, is the only time that we've had to make significant changes to our services. Everything else we talked about so far today has happened below the surface. It's been handled by our platform team.
How to Run on Three Clouds at Once
Kevin Holditch: How did we run on three clouds at once? The first one was that we picked cloud agnostic technologies such as Kubernetes, NATS, and Cockroach, so we could run the same codebases everywhere and not have to run separate codebases on each cloud provider. The second pillar was the fact that we used single datastores across the three clouds, that really made writing the software and thinking about the software, even though it is active-active-active across three clouds, much easier to reason about. The third one was that we treated each cloud provider like it was an availability zone.
When Not to Run Triple Active Multi-Cloud
Ross McFarlane: Reasons potentially to not go down this path are if what you're building doesn't fit the resilience stories for the market that you're selling into. If your state-of-the-art solution doesn't impress your customers, then it's maybe not worth your effort, or your money for that matter. Running triple active multi-cloud is an expensive business, and bankruptcy is incompatible with uptime. Then, finally, I wouldn't go down this road unless you've got a really robust platform engineering practice and a really capable and trusted platform team who can make this happen and let everybody else keep cracking on above the surface.
Questions and Answers
Participant 1: How did you get customers comfortable talking about non-zero RPO, RTO?
Ross McFarlane: That was actually the default in the market. In the U.S., the expectation is that you run on the East Coast and the West Coast. That's what most people seem to be doing. Multi-cloud seems to be quite a new concept there. We're only seeing some of our competitors starting to catch up to the idea of being multi-cloud now. There is an expectation of having to shift from one to the other and not run active-active across the two.
Participant 1: You come to customers and they understand East-West and they understand that means data loss?
Ross McFarlane: Yes, that was weird for us too, I have to say. It felt like an odd step backwards, but that's how we got to it.
Participant 2: How does the company generally feel about a potential outage in one of the cloud providers, and everybody having to go somewhere else. Then everybody is going somewhere else at the same time. What are your thoughts there?
Kevin Holditch: Do you mean on our multi-cloud side, on the active-active-active platform, or on the U.S.? Because I think the answer is probably different.
Participant 2: In multi-cloud, cloud-to-cloud.
Kevin Holditch: On the multi-cloud platform, we've gotten to the position, I wouldn't say we're blasé about it, but we don't really consider it. I'd say over the back half of 2025, we saw each of the three major cloud providers have decent-sized outages. The Google one that I spoke about, the one that Ross showed, and Azure had quite a large one as well, I think it was in October. None of those impacted our platform. We've actually gotten to the position now where the platform, it really is like treating a cloud provider like an availability zone. I think the key is we don't switch people across the cloud, they just use all of them at the same time.
Participant 3: How elastic are you with what you're doing in those Kubernetes clusters? Is stuff scaling up and scaling down, or are you looking at a more like fixed processing, and so that gives you more comfort in that scenario that we're talking about where, suddenly everybody wants to be on AWS because Google and the other one have gone away.
Kevin Holditch: The answer today is we are not very elastic at all. We are more pre-scaled for the volume that we know the customers can give us. Customers give us a six-month projected volume, and we'll test that on an early environment to make sure we can cope with it. We add some tolerance, so if one cloud's down, the other two clouds can handle the load. We don't do anything in the dynamic scaling space at the moment.
Ross McFarlane: Works nicely for that scenario, so yes, makes sense.
Participant 4: Obviously, you're getting a lot of resilience across the three clouds in the UK example. What are the tradeoffs in terms of cost? Because that's usually a big tradeoff, is we can have everything all singing, all dancing, but normally there's limits and constraints. How did you go about analyzing those tradeoffs, and especially the cost side of things, and the complexity as well?
Kevin Holditch: I think the cost question's an interesting one. We've made a number of strides forward to reduce the cost of running the platform. We're going to continue to do that. It's definitely more expensive to run across three clouds. There's no getting away from that. I do think there's things you can do to make it a bit cheaper than what it may seem. Yes, it's definitely an interesting tradeoff. The number one driving factor for us is that this is what the market demanded, what our customers wanted and needed. It's the sort of platform that we built to satisfy that market. I guess cost, yes, it's more expensive than a single cloud, but it was the product that fit the market, so that's why we went for that solution.
Ross McFarlane: There's an interesting tradeoff for me with cost recently that surprised me. When Kev was showing the number of different environments we're having to look after, particularly those pre-production environments that need to be scaled to at least production size or production plus six months. One of the things we've done is have multiple production environments, because if we had one enormous production environment, we would have to reflect that scale in the pre-production environment, and so you then end up carrying more cost overall. What we've ended up with is some multiple smaller production environments and then match that with the pre-production environment.
See more presentations with transcripts
