Key Takeaways
- A 99.99 percent SLA is a statistic, not a promise. Design for the tail of that distribution, because at sufficient scale you will eventually land in it.
- An alert that wakes a human only speeds up notification. Resilience comes from automation that acts on the signal and informs humans afterward.
- A passive region that has never carried production traffic is an untested hypothesis. Active-active keeps both regions continuously validated under real load.
- DNS TTL can nullify a perfectly executed failover. Choose it deliberately against your RTO, health-check interval, and false-positive tolerance, rather than accepting the module's default.
- Faster detection only sharpens the post-mortem timeline. Reliability requires automated recovery plus a confirmation loop that verifies the failover actually worked.
The Failure Nobody Expected From a Service Everyone Trusted
I almost titled this section The Failure I Should Have Expected, and honestly, that would have been the more accurate title. The conditions were all there. I just hadn't looked at them as a system that would eventually fail, only as infrastructure that had been working fine so far. My team owned a Terraform execution platform. The registry was backed by S3, which is standard stuff. Nothing else exotic was added. One morning the dashboard lit up, showing registry failures and artifacts not downloading. We declared an incident and started the usual process of elimination.
Two hours into our investigation, I was still convinced we had done something wrong. Perhaps we had a misconfigured bucket policy or added a permissions boundary incorrectly. Hopefully, this problem was embarrassingly fixable. I kept going back to our changes, our config, and our code. The problem wasn't in any of those places.
The actual cause took hours to surface. S3 signed URLs worked fine. You could manually call one and download an artifact without issue. But Terraform calls those URLs in sequence. When it resolves module dependencies, it fires a bunch of them back-to-back. Some of those requests were silently failing. Not all of them failed and not in any consistent pattern, just... some, which was enough, to break the pipeline. From every dashboard we viewed, it looked like a registry bug, rather than an S3 bug.
We found the real cause by building a controlled test environment on my laptop. The failures were intermittent and appeared only in production, where experimenting is discouraged, so I needed a safe way to reproduce them. I wrote a small Terraform configuration that downloads modules from our registry, the same path our pipelines use, and ran it under a load test. I also checked out the Terraform source and added logging for HTTP request and response headers. Logging exposed an easy-to-miss detail. Terraform hands module downloads to HashiCorp's go-getter library, which sends a HEAD request to each signed URL before the GET, to check whether the download can be resumed. The load test reproduced the failure within ten minutes; the logs traced it to that HEAD request rather than to our registry. I proposed that a recent change in how AWS handled the HEAD request was producing the downstream 404.
This hypothesis was initially not obvious even to their support personnel. I substantiated it with clear reproduction steps and diagnostic evidence. After they escalated internally to their architecture team, they confirmed that a recent change in request handling was the root cause and rolled back the change, restoring expected behavior. The lesson I took from this situation was when a managed dependency fails intermittently, reproduce your client's exact call pattern outside of production and instrument the client, not just the server.
Detection took a few hours. Recovery took nearly twenty-four hours, most of which was spent on support calls proving the issue was on their end waiting for AWS to acknowledge other customers were affected and waiting for the rollback to ship. Once AWS rolled back the S3 deployment, concurrent signed URL requests stabilized and the registry recovered without any changes on our end.
The mechanism here, signed URLs, concurrency, isn't the important part. The important part is the assumption I had been running on without examining it, that cloud infrastructure is reliable by default. That 99.99 percent is essentially "always works". I had treated the SLA like a guarantee rather than a statistical claim about a distribution with a tail I just hadn't seen yet.
Fifty-two minutes and thirty-six seconds of downtime per year. Considering the scale at which we were operating, hitting that window was not a matter of if, because I already had and more than once. I just hadn't treated it as evidence that my design needed to change.
Observability Should Drive Decisions, Not Just Display Them
Every organization I have worked for has made the same mistake and I bet that yours has too. Someone builds an observability system early on. They care about it. The dashboards are thoughtful, the alerts are calibrated, and the runbooks are linked. Then the system grows, engineers move on, haphazardly adding and wiring in new services without anyone updating what came before. The people who built the original dashboards aren't around to update them and the people who are around are nervous to touch something they didn't build and don't fully understand. Alerts drift out of calibration and dashboards start showing issues nobody on the current team knows how to interpret.
Then something breaks at a bad hour and the person on call opens Grafana and spends twenty minutes just trying to figure out what they are seeing. I have been in that position. The problem runs deeper than knowledge. It is a design failure, not a skills gap. The dashboards were built as a reporting layer, something that shows you what is happening. But they were never connected to anything that does something about what is happening. The human is still the bridge between signal and response.
The mistake has nothing to do with tooling choice, query language, or dashboard layout. The mistake is treating observability as a way to inform humans rather than a way to drive system behavior. The difference is significant in practice. In reporting, a human receives an alert, interprets it under stress at a bad hour, decides what to do, and acts. In a control plane, the system receives and evaluates the signal against a defined policy, acts, and tells the human what happened. One of those scales with your headcount and on-call rotation, while the other breaks the moment both are stretched thin.
None of the tooling required for this design is new. Prometheus has had alerting rules and webhook routing for years. Terraform can reconfigure routing weights. The components all exist; they just haven't been wired together with automation as the explicit goal. The pattern is the same at every scale, from pod autoscaling to regional failover: observe, decide, act, and confirm.
I have run Prometheus and Grafana stacks across hundreds of services, large builds and deployment pipelines, with thousands of executions per day. These tools are powerful but carry an operational cost teams consistently underestimate. Consider that expertise concentrates. When the person who really knows the stack leaves, calibration drifts silently until something breaks at the worst possible moment. Treat your observability stack with the same ownership model as your production services with regular reviews, real failure testing, and collective ownership that does not expire when the original author leaves.

Figure 1. Observability as a reporting layer vs. a control plane.The difference is whether a human or the system bridges signal to response. (Source: created by author.)
Designing Multi-Region Failover That Survives Real Failures
I have never worked with anyone who needed a whitepaper to understand why multi-region matters. Instead, they needed an outage. After a major regional failure hit one of our cloud providers, the conversation about moving the Terraform registry to active-active architecture happened. The registry is called during every pipeline execution: every build, every deployment, and every release. When it goes down, every build, every deployment, and every release across every team stops with it. The blast radius of that dependency is much larger than it looks on an architecture diagram; it took a real outage for that to land viscerally for everyone involved.
We chose active-active, not active-passive. The reason for this choice comes up less often in architecture discussions than it should. We wanted the secondary region under real production load before we needed it for failover.
Active-passive looks good in diagrams. The failure mode is subtle and tends to surface at the worst possible time. The passive region has never handled your actual traffic. Its configuration has drifted unnoticed. Caches are cold and downstream dependencies haven't been exercised under anything close to real load. You have tested it in staging, maybe, with synthetic load that doesn't match your real traffic profile. When you fail under pressure, during an active incident with people watching, you discover all of these issues simultaneously.
I have been on that incident bridge. It is a specific kind of unpleasantry. You are trying to recover from one problem while discovering a second set of problems you didn't know you had. Active-active costs more. You have to keep regions in sync. Data consistency under partition requires careful thought. But when a region degrades, you shift load in a system that has been running in two regions all along, not making a leap of faith into an untested environment. We ran 50/50 under normal operation, which gave us continuous validation. Both regions were exercised constantly under real load. But 50/50 is the steady state, not the failure state. When a region actually degrades, traffic shifts to 100/0.
That transition is what you are building toward and it has to be tested deliberately. We ran chaos testing specifically for this situation. Cut one region entirely, verify the other absorbs full load cleanly, and confirm recovery when it comes back. If you haven't tested the 100/0 state, you haven't tested your disaster recovery (DR) system. Instead, you have tested your routing config. Multi-region is expensive to build and maintain. For a critical build-time dependency like this registry, it was the right call. But it should be a deliberate decision, not a default assumption.
Two failure modes in this design are worth being honest about, because neither shows up in a clean chaos test. The first is a failure that is not regional at all. The S3 incident hit both regions at the same time, because both depended on the same S3 behavior. Regional failover cannot fix that problem. Failover shifts traffic to the other region, which is failing the same way, so failover triggers there too and traffic shifts back. The automation keeps moving traffic between two broken regions and adds noise to an incident it cannot resolve. Route 53 behaves the same way at the DNS layer. When every weighted record is unhealthy, it returns all of them rather than none. When both regions degrade together, the cause is a shared dependency. The correct automated response is to hold traffic where it is and page a human.
The second failure mode is a health check that reports the wrong thing. Our /health endpoint confirmed that the application process was running, not that it could do its job. If the backend database in one region goes down, the application keeps answering /health with 200 OK while real requests fail. This is exactly the case for which regional failover exists, but Route 53 sees a healthy endpoint and keeps sending traffic to the broken region. The confirmation Lambda checks the same endpoint, so it would also report a failover into that region as successful. A health check has to exercise the functionality the service delivers, such as a lightweight database read, not just confirm the process is up. Deep checks carry their own risk. If the dependency they test is shared by both regions, they bring back the first failure mode.
DNS TTL is the single most common DR misconfiguration I have seen in otherwise well-designed systems and it is worth addressing directly. TTL controls how long a resolver caches a DNS record before requerying. If the setting is too high (e.g., three hundred seconds, six hundred seconds, or whatever the module default happened to be) and your failover logic can fire correctly, then your routing weights can update correctly and clients will keep hitting the degraded region for minutes because their resolvers haven't expired the old record. The failover executed correctly, but the traffic never moved because resolvers were still serving the cached record.
If the setting is too low, then you have built an overreactive system. A transient latency spike triggers a DNS change. The issue resolves thirty seconds later. Now you have regions flip-flopping and amplifying instability that would have cleared on their own. The right TTL is a deliberate number, chosen against your RTO, your health check interval, and your false positive tolerance. Most teams are running whatever default Infrastructure as code (IaC) module was set years ago. They don't know what that number is until they are debugging a failover that didn't go the way they expected.

Figure 2. DNS TTL and failover timing. A weight change that fires correctly can still leave traffic stranded in the degraded region until resolver caches expire. (Source: created by author.)
Disaster Recovery as an Automated, Observable System
What separates DR systems that actually work from those existing only on paper is automation with confirmation, not just detecting failure, but acting on it and verifying the action succeeded. Most teams that have invested seriously in DR have detection, such as alerts that fire when error rates spike or health checks fail. Fewer have built the response layer, the automated action that runs when the signal fires. Fewer still have built the confirmation layer, the part checking whether or not the response actually worked.
This last piece matters more than people realize. Automated failover without confirmation is a system taking an action and then going quiet. Did the secondary region receive the traffic? Did DNS propagate before the TTL window? Are the downstream dependencies there responding correctly? A system capable of answering these questions by continuing to observe after acting is one you can trust. A system that only fires and then goes silent is an untested hypothesis.
The pattern looks like this in practice: A Prometheus alerting rule evaluates a composite signal, error rate above threshold AND p99 latency elevated AND sustained for a defined window. This observation is important. Single-metric alerts are noisy. A transient spike fires them constantly for things that resolve on their own in a minute or two. The composite condition with a duration requirement indicates you only trigger degradation that has actually persisted.
Most implementations stop at the alert. Here the alert is the entry point to an automated chain. How traffic moves depends on the topology. In an active-active setup, Route 53 does the switching. Each regional record carries a weight and a health check. When a region's health check fails, Route 53 stops returning that record, so traffic shifts to the healthy region with no custom code. In an active-passive setup, the alert drives the switch. Alertmanager routes it to a webhook, which invokes a Lambda function that flips the Route 53 failover record to the secondary region. In both cases, the alert also triggers a confirmation Lambda that polls the target region's health endpoint until it confirms the region is returning healthy responses. The human receives a notification only after the Lambda confirms the loop completed successfully. Detection, action, and confirmation form one connected loop. Observability drives the action and then verifies the outcome.

Figure 3. The detect–act–confirm loop. Prometheus fires, Route 53 shifts traffic, Lambda polls until the target region confirms healthy, then notifies the team. (Source: created by author.)
That confirmation loop, in practice, looks like this:
import urllib.request
import time
def lambda_handler(event, context):
# CONFIRM: Route 53 or the failover webhook has already moved traffic
# Lambda confirms the failover succeeded and notifies
confirmed = False
for attempt in range(10):
time.sleep(30)
try:
response = urllib.request.urlopen(
'https://registry-secondary.internal.example.com/health',
timeout=5
)
if response.status == 200:
confirmed = True
break
except Exception:
continue
return {
'failover_executed': True,
'confirmation_status': 'healthy' if confirmed else 'unconfirmed',
'attempts': attempt + 1
}
The loop does not close until the health check returns 200. If it never does, the function returns unconfirmed and that signal itself becomes an alert, not a silent failure.
This is a Terraform configuration for the Route 53 side. Both regions carry equal weight and Route 53 stops returning a region's record when its health check fails:
resource "aws_route53_health_check" "primary" {
fqdn = "registry-primary.internal.example.com"
port = 443
type = "HTTPS"
resource_path = "/health"
failure_threshold = 3
request_interval = 10 # or 30, fine tune based on the system
}
resource "aws_route53_health_check" "secondary" {
fqdn = "registry-secondary.internal.example.com"
port = 443
type = "HTTPS"
resource_path = "/health"
failure_threshold = 3
request_interval = 10
}
resource "aws_route53_record" "registry_primary" {
zone_id = var.zone_id
name = "registry.internal.example.com"
type = "A"
set_identifier = "primary"
weighted_routing_policy {
weight = 50
}
health_check_id = aws_route53_health_check.primary.id
ttl = 30 # or 60 can be fine tuned based on the system.
records = [var.primary_region_ip]
}
resource "aws_route53_record" "registry_secondary" {
zone_id = var.zone_id
name = "registry.internal.example.com"
type = "A"
set_identifier = "secondary"
weighted_routing_policy {
weight = 50
}
health_check_id = aws_route53_health_check.secondary.id
ttl = 30 # or 60 can be fine tuned based on the system.
records = [var.secondary_region_ip]
}
Prometheus rule:
groups:
- name: registry_availability
rules:
- alert: RegistryRegionalDegradation
expr: |
sum by (region) (rate(http_requests_total{job="registry", status=~"5.."}[2m]))
/
sum by (region) (rate(http_requests_total{job="registry"}[2m]))
> 0.03
and
histogram_quantile(0.99,
sum by (le, region) (rate(http_request_duration_seconds_bucket{job="registry"}[2m]))
) > 0.8
for: 90s
labels:
severity: critical
annotations:
summary: "Registry degradation in region {{ $labels.region }}"
description: "Error rate above 3% and p99 latency above 800ms sustained for 90 seconds in {{ $labels.region }}"
The by (region) clause is what makes this rule region-aware. It evaluates the composite signal independently per region, so when one region degrades while the other remains healthy, the alert fires with the specific region label that tells the automation exactly which region is degraded.
The for: 90s is doing real work there. Without it, this alert fires on blips. With it, the alert fires only when something has been consistently degraded for a minute and a half, which is a meaningfully different situation.
Finally, there is testing. The most effective practice I know for thorough testing is the game day, a scheduled exercise where the team deliberately injects real failures into production systems and works through detection, response, and recovery in real time, with the people who would handle a real incident in the room. It is not a runbook walk-through in staging.
Game days find the human gaps as reliably as they find technical ones. Who actually has decision authority when you cannot reach the incident commander? Where is the runbook for this failure mode, exactly? Does everyone know? The first time most teams run one, the answer to these questions is uncomfortable. That is the point. Finding it during a game day costs an afternoon.
The validation that teams most consistently skip is testing that the confirmation layer correctly recognizes a successful failover and stops taking action. People test that the failover fires. They rarely test what happens if it fires and the system thinks it didn't work. That is a real path to cascading failures you created yourself.
Conclusion
The S3 incident wasn't something I had modeled. Nobody had a runbook for "S3 silently breaks concurrent signed URL requests under load following an internal deployment". The assumption that underpins a lot of how we build systems, that the infrastructure below us is basically reliable, is fine most of the time, which is exactly why it is so rarely questioned.
Three steps will turn your failover plan into a working system. First, pick one alert that pages a human today and wire it to an automated response with a confirmation step. Second, put your secondary region under real production traffic before you need it for failover. Third, on your next game day, test the confirmation layer, not just the trigger. Verify that the system recognizes a successful failover and stops acting. If you haven't validated all three, you have a plan, not a system.
Infrastructure will fail, and the next failure will probably be one you haven't seen before. Whether a person has to piece together a response manually or the system handles it was decided long before the incident started.