Scalability & Reliability

Scalability Overview

Vertical vs Horizontal Scaling

Every system eventually hits a traffic ceiling. The question is: do you make the box bigger, or add more boxes?

Stateless vs Stateful Services

Horizontal scaling only works well when your services are stateless. Here's why.

Stateless Services

The server doesn't remember anything between requests. Every request contains all the information needed to process it. Think of it like a cashier who doesn't recognize you — you show your receipt every time.

• Session data is stored externally (Redis, database, JWT token)
• Any server can handle any request — the load balancer picks freely
• Easy to scale: add more servers, remove failed ones, no data loss
• Example: REST API servers, serverless functions (AWS Lambda)

Stateful Services

The server remembers something about the client between requests. Like a bartender who knows your "usual."

• Sticky sessions required — the load balancer must route the same client to the same server
• Harder to scale: if that specific server goes down, the user's state is lost
• Example: WebSocket connections, in-memory session stores, game servers

Replication

Replication means keeping copies of your data on multiple machines. It's your first line of defense against data loss and your main tool for scaling reads.

Master-Slave (Primary-Replica)

One master handles all writes. Multiple replicas handle reads. The master pushes changes to replicas.

• Reads scale: add more replicas to handle more read traffic
• Writes don't scale: all writes still go to one master
• Failover: if the master dies, promote a replica to master (manual or automatic)
• Replication lag: replicas may be slightly behind the master (eventual consistency)

Master-Master (Multi-Primary)

Multiple nodes accept both reads and writes. Changes are replicated between them.

• Writes scale better: multiple nodes accept writes
• Conflict risk: two nodes may update the same row simultaneously — needs conflict resolution
• More complex: harder to set up and debug than master-slave
• Use case: multi-region deployments where you need low-latency writes in each region

Sync vs Async Replication

Sharding (Partitioning)

Sharding splits your data across multiple databases. Each shard holds a subset of the total data. This is how you scale writes — each shard handles its own writes independently.

Sharding Strategies

Hash-Based Sharding

Hash the shard key (e.g., user_id) and use modulo to pick a shard. Distributes data evenly.

• Pro: Even distribution of data
• Con: Adding/removing shards requires rehashing (use consistent hashing to minimize this)

Range-Based Sharding

Assign ranges to shards (e.g., users A-M on shard 1, N-Z on shard 2).

• Pro: Range queries are efficient (all data in one shard)
• Con: Hot spots if distribution is uneven (e.g., more users with names starting with "S")

Consistent Hashing

Nodes are placed on a virtual ring. Data maps to the next node clockwise. When you add/remove a node, only a fraction of data moves. This is how DynamoDB, Cassandra, and many CDNs work.

Cross-Shard Queries

The biggest pain point. If you need data from multiple shards (e.g., "find all orders across all users"), you must query every shard and merge results. This is slow and complex. Design your shard key so that most queries hit a single shard.

CAP Theorem

The CAP theorem says a distributed system can guarantee at most two of three properties:

CP vs AP in Practice

Reliability Patterns

A reliable system keeps working (maybe in a degraded state) even when things go wrong. Here are the patterns that make that happen.

Redundancy

Eliminate single points of failure. Every critical component should have at least one backup. Two load balancers (active-passive), multiple app servers, replicated databases, multi-AZ deployment.

Health Checks

The load balancer pings each server periodically. If a server stops responding, traffic is automatically routed away from it. Types: TCP checks (is the port open?), HTTP checks (does /health return 200?), deep checks (can the app reach the database?).

Circuit Breakers

Think of an electrical circuit breaker. If a downstream service starts failing, the circuit "opens" and requests fail fast instead of waiting and timing out. This prevents cascading failures.

• Closed: normal operation, requests pass through
• Open: service is down, fail immediately (return cached data or error)
• Half-open: after a cooldown, let a few requests through to test if the service recovered

Retry with Exponential Backoff

When a request fails, retry — but wait longer between each attempt: 1s, 2s, 4s, 8s... Add jitter (random variation) so thousands of clients don't all retry at the exact same moment (thundering herd).

Rate Limiting

Protect your service by limiting how many requests a client can make in a given time window. Common algorithms:

• Token bucket: a bucket fills with tokens at a fixed rate. Each request costs one token. When empty, requests are rejected. Allows bursts up to bucket size.
• Sliding window: count requests in a rolling time window. More accurate than fixed windows. Used by most API gateways.
• Fixed window: count requests per calendar interval (e.g., per minute). Simple but has edge cases at window boundaries.

Graceful Degradation

When part of the system is overloaded, serve a reduced experience instead of failing completely. Examples: show cached data instead of real-time, disable recommendations but keep search working, serve static pages when the API is down.

SLAs, SLOs, SLIs

How do you measure and promise reliability? Three related but different concepts.

The "Nines" of Availability