Five Nines: Designing for Failure

Lessons in Resilience from Building Carrier-Grade Distributed Systems

In enterprise software, "high availability" can sometimes feel like an aspirational metric.

In telecommunications, it is an operating requirement.

Five-nines availability---99.999%---allows roughly five minutes of downtime in an entire year. When a platform sits in the transaction path of a mobile network, an outage isn't simply a degraded dashboard or an unavailable application. It can affect services being delivered to millions of subscribers.

I came to appreciate that distinction deeply while building Syntonic's Connected Services Platform.

We integrated the platform with the billing, policy, and data-gateway infrastructure of 16 mobile operators across North America, Latin America, Africa, and Southeast Asia, ultimately reaching a subscriber footprint of more than 375 million people across 11 time zones.

Working with Tier-1 operators such as AT&T and Verizon changed the way I thought about reliability.

Earlier in my career, I tended to think about high availability primarily as an architecture problem: eliminate single points of failure, add redundancy, replicate data, automate failover.

All of those things matter.

But carrier environments taught me a more fundamental lesson:

You don't achieve extreme reliability by trying to eliminate failure. You achieve it by controlling what happens when failure inevitably occurs.

In other words, achieving five nines requires designing for failure.

Distributed systems are always experiencing some form of partial failure. A network link becomes unreliable. A database slows down. A partner API stops responding. A region becomes unreachable. A software deployment behaves differently under production load than it did in testing.

The architectural challenge is to keep one of those events from becoming everyone's problem.

Several principles from that experience have stayed with me.

1. Contain the Blast Radius Before You Add Redundancy

The conventional answer to high availability is redundancy.

Two servers are better than one. Two availability zones are better than one. Two regions must therefore be better than one.

Not necessarily.

Active-active architectures can provide extraordinary resilience, but they also introduce another category of distributed-systems problems: replication lag, conflicting state, split-brain behavior, complex routing decisions, and cascading failovers.

Redundancy without isolation can actually increase the number of ways a system can fail.

A transient network partition, for example, should be a localized infrastructure event. Poorly designed failover logic can turn it into a global outage by shifting traffic toward infrastructure that is already degraded.

At Syntonic, we approached the problem by making carrier deployments as self-contained as practical.

Regional cells had dedicated ingress paths and locally available policy state. Problems associated with one carrier integration or geography could therefore be isolated rather than automatically propagating through the global platform.

If an integration in Brazil experienced a routing anomaly or local database contention, our goal wasn't to make Brazil's problem disappear instantaneously.

Our first goal was to ensure it did not become Seattle's---or Southeast Asia's---problem.

That distinction changed how I think about resilient architecture.

Containment often matters more than redundancy.

Before asking, "How quickly can the entire system fail over?" ask a more important question:

How much of the system needs to know that this component failed at all?

The smaller that answer is, the more resilient the architecture tends to become.

2. Standardize the Core. Embrace Variance at the Edge.

Large enterprise platforms face another reliability challenge that doesn't appear on most architecture diagrams:

Every customer is different.

Telecommunications made this particularly obvious. Carriers had different authentication systems, billing infrastructure, policy controls, operational procedures, and network interfaces.

The tempting solution is to customize the platform itself for every major customer.

That works---for a while.

Eventually the core becomes a collection of customer-specific conditionals. Every release carries the risk of breaking a bespoke integration somewhere else in the world.

We learned to push that variability outward.

Our experiences with AT&T and Verizon illustrate the two extremes.

AT&T already had sponsored-data infrastructure, but partner onboarding could be cumbersome. CSP could operate as an API layer over that infrastructure, simplifying how partners interacted with the underlying carrier capabilities.

Verizon represented a different integration model. For its FreeBee sponsored-data offering, CSP could operate as a more complete white-label platform, including infrastructure used to support zero-rated experiences such as NFL streaming.

Very different deployments.

The architectural principle underneath them was the same.

Keep the core platform standardized and isolate customer-specific complexity behind well-defined boundary adapters.

Carrier-specific authentication, billing protocols, policy interfaces, and network behavior belonged at the integration boundary---not scattered throughout the transaction engine.

That separation had benefits beyond maintainability.

It protected reliability.

A change required for one carrier could remain localized to that carrier's adapter instead of modifying code paths shared by every deployment.

The lesson applies far beyond telecom:

A scalable enterprise platform shouldn't eliminate customer variance. It should give that variance a safe place to live.

3. Degrade Gracefully Instead of Waiting Perfectly

One of the easiest ways to take down a distributed system is to wait politely.

A service receives a request and calls a downstream dependency.

The dependency becomes slow.

Threads begin waiting.

Queues grow.

Memory pressure increases.

Retries create additional traffic.

Soon a relatively minor problem in one external service has consumed resources across the system.

In carrier infrastructure, that failure mode is unacceptable.

A subscriber shouldn't lose connectivity simply because an ancillary billing or telemetry endpoint is responding slowly.

We therefore treated external dependencies with suspicion.

For critical transaction paths, we used aggressive timeouts and circuit-breaker patterns. In one implementation, if a real-time carrier billing validation endpoint exceeded a 150-millisecond response threshold, the platform could stop waiting synchronously and fall back to an asynchronous optimistic-grant model with subsequent reconciliation.

From the subscriber's perspective, the transaction continued.

The dependency could recover independently.

And, most importantly, its failure remained contained.

This led to another principle that has stayed with me:

Never allow a non-critical dependency to determine the availability of a critical transaction.

That doesn't mean ignoring errors or compromising correctness.

It means deciding explicitly which guarantees must be synchronous and which can be reconciled later.

It means establishing latency budgets rather than waiting indefinitely.

It means bounding retries rather than creating retry storms.

And it means designing degraded modes while the system is healthy rather than improvising them during an outage.

The best fallback is rarely invented at 3:00 AM.

4. Reliability Is an Engineering Culture, Not an Operations Function

Some of the most important lessons about five-nines reliability had nothing to do with architecture.

They had to do with ownership.

At Syntonic, our engineering organization spanned Seattle, India, and Brazil. We shipped frequently and maintained on-call responsibility close to the engineers and technical leads who understood the systems being operated.

That matters.

When the people making architectural and implementation decisions also experience the operational consequences of those decisions, behavior changes.

Error handling stops being cleanup work.

Telemetry stops being something Operations asks for later.

Health checks, deployment safety, rollback behavior, observability, resource limits, and failure modes become part of the design conversation from the beginning.

There is a useful feedback loop:

You build differently when you know you may be the person diagnosing the failure.

This does not mean every developer should spend every night watching dashboards.

It means reliability cannot be delegated organizationally to a team that receives software only after engineering has declared it "done."

SRE and Operations teams can provide enormous leverage, expertise, tooling, and discipline.

But the people designing and building a system must retain ownership of how that system behaves in production.

Five-nines is ultimately as much a cultural property as a technical one.

Designing for Failure

The most important thing carrier infrastructure taught me about reliability is that the goal is not perfection.

Perfection is the wrong mental model for a distributed system.

Networks partition. Dependencies slow down. Hardware fails. Software contains bugs. Deployments introduce unexpected behavior. People make mistakes.

At sufficient scale, unlikely events stop being unlikely.

The job of a resilient architecture is therefore not to create a world in which components never fail. It is to prevent individual failures from determining the behavior of the entire system.

Contain the blast radius.

Keep the core independent from integration complexity.

Put strict boundaries around downstream dependencies.

Design graceful degradation before you need it.

And make the engineers who build systems accountable for how those systems behave in production.

Five-nines availability is measured as a percentage. But the systems that achieve it are built around a different idea:

Assume failure. Contain it. Recover from it. Keep the system operating.

That is what designing for failure means.

And it may be the most important lesson I took away from building carrier-grade systems at global scale.