Uber explained how it keeps its OpenSearch deployments running during a zone outage. It does this by using OpenSearch's built-in shard allocation and its own isolation-group system, which relies on the Odin container orchestration platform. This way, it maintains both query and ingestion capabilities. The design tolerates a full zone failure plus one additional node failure while preserving quorum and avoiding data loss.
Uber aimed to solve a problem: physical zones often lack balanced node counts. This issue messed up OpenSearch's allocation-awareness logic. It typically left clusters in a yellow state with unassigned shards. Mapping shard placement to physical zones led to disk skew and hot nodes when zone capacity was uneven. This happens because OpenSearch's rebalancing expects an even pool of nodes for each awareness attribute.
Uber's fix introduces isolation groups (IGs) as a logical layer between physical failure domains and OpenSearch's placement logic. Every IG is guaranteed an equal number of nodes, regardless of how many physical zones are underneath, and a node retains its IG membership through hardware replacements. Most Uber services, including OpenSearch, run 3 IGs, so a single zone maps to exactly one IG, and a zone failure removes at most about a third of capacity. Each index runs with a minimum of 2 replicas (3 total shard copies), one per IG, giving baseline resilience against a single-group loss.
The harder problem is what happens during the failure itself. By default, OpenSearch responds to missing shard copies by rebalancing quickly across the remaining nodes. This can overload disk I/O, CPU, and network, risking instability in healthy areas. Uber tackles this with forced shard allocation awareness. The cluster is set up with all expected IG attribute values from the start, not just the ones visible now. When an IG goes missing, OpenSearch spots the gap. It won’t over-allocate to the other groups, so affected shards stay unassigned. This causes the cluster to turn yellow instead of starting a rebalancing storm. Shards only get reassigned once the failed IG's nodes return or an operator explicitly updates the awareness configuration.
Uber uses 5 cluster manager nodes for quorum, instead of the usual 3. This works with OpenSearch's cluster.auto_shrink_voting_configuration setting. A zone failure can take out up to 2 manager nodes, leaving 3; the voting configuration automatically shrinks to those 3, electing a new primary with a 2-of-3 quorum. A subsequent single-node failure still leaves 2 nodes, enough to retain quorum and keep the cluster writable, which a standard 3-manager setup could not survive.
Uber reports that the IG abstraction fixed yellow-state and shard-assignment failures. These issues came from uneven physical zone sizes. Now, there's 100% shard assignment and consistently green cluster health. It also addresses disk skew and hot-node effects linked to zone asymmetry. This approach works for all Tier 3 and higher OpenSearch and Elasticsearch clusters at Uber. It builds on OpenSearch features such as shard allocation awareness and voting configuration shrinkage. It doesn't need a custom search engine or a forked version.
Engineers running multi-zone OpenSearch or Elasticsearch clusters can adopt the forced-awareness pattern without Uber's Odin-specific tooling. The key requirements are an even node distribution across a fixed, explicitly declared set of awareness attribute values, at least 3 shard copies mapped one-to-one to failure domains, and an odd number of cluster manager nodes (5 rather than 3) with auto-shrink voting enabled. This is to survive a zone-plus-node failure sequence.
Uber's method reflects a wider trend in the industry. It focuses on embedding failure-domain awareness in the data layer rather than relying solely on infrastructure-level failover. Uber used a similar isolation-group model for Apache Pinot. They mapped isolation-group IDs to replica-group pools. This setup ensures that segment replicas can survive when a zone completely fails.