On this page
Description
One server is easy: clients know its address, and every request goes there. The moment you need two - because traffic doubled, or because one server dying shouldn't take the service down - you need something in front of them that knows which servers are healthy right now and spreads requests among them. That's a load balancer. In front of a public API it usually grows into an API gateway, which also checks who's calling and how often.
Why not just list both servers' addresses in DNS? Because DNS answers are cached for minutes by clients and resolvers: when one server crashes, a slice of users keeps being sent to the dead machine, and nothing retries them elsewhere. In this lesson:
- What a load balancer does - one address, spreading load, health checks, draining.
- How it picks a server.
- Layer 4 versus layer 7 balancing.
- Stateless servers, and why sticky sessions hurt.
- API gateways and rate limiting with a token bucket.
What a load balancer does
- One stable address. Clients call
api.example.com; the balancer owns it. - Spreading load over a pool of backend servers.
- Health checks. Every few seconds it probes each server (
GET /health); after a few failures the server is taken out of rotation, and put back when it recovers. - Connection draining. When a server is being removed - a deploy, a scale-in - the balancer stops sending it new requests but lets in-flight ones finish.
Choosing a server
| Algorithm | How | Good for |
|---|---|---|
| Round robin | Each server in turn | Requests of similar cost |
| Weighted round robin | In turn, bigger servers more often | Mixed server sizes |
| Least connections | The server with the fewest open requests | Requests of very different cost (a 10 ms read next to a 5 s report) |
| Hashing (by client or key) | The same key always to the same server | Caches local to a server; rarely needed with stateless servers |
Round robin can pile slow requests onto one server while others idle; least-connections adapts to it.
L4 versus L7
- Layer 4 (transport) balancers forward TCP (or UDP) connections by address and port. They're fast and protocol-agnostic, but can't see the HTTP request - no routing by path, no retries of individual requests. Once a long-lived connection is placed, every request on it goes to the same server.
- Layer 7 (application) balancers terminate the connection, read the HTTP request and can route
/api/*and/images/*to different pools, terminate TLS, add headers, retry a failed request on another server and balance individual requests over long-lived HTTP/2 connections.
Most web APIs sit behind an L7 balancer; L4 shows up for raw TCP protocols or as the very first tier in front of L7 balancers. A typical public API stacks them:
Stateless servers, and why sticky sessions hurt
If a user's session lives in one server's memory, every request must go back to that server - "sticky sessions", usually by a cookie the balancer reads. It works until that server dies (the users on it are logged out) or you scale in (same). It also unbalances load, because heavy users stay where they are. The fix is to keep servers stateless: sessions in a shared store (Redis) or in a signed token the client carries, so any server can handle any request and the balancer is free to route.
The balancer is a single point of failure
A single load balancer in front of twenty servers means one machine can take the whole service down. Production setups run balancers as an active-passive or active-active pair sharing a virtual IP, or use a managed cloud balancer that's redundant across zones. On a design diagram, give the balancer 2+ instances or say it's a managed, multi-zone service.
API gateways
An API gateway is an L7 entry point with API concerns built in:
- Authentication - validating tokens once at the edge.
- Routing -
/orders/*to the order service,/users/*to the user service. - Rate limiting and quotas - per API key or user.
- Request shaping - timeouts, retries, response caching, request size limits.
Load balancers still sit behind it (or it balances itself) for each service's pool.
Rate limiting
Without limits, one buggy client in a retry loop - or a scraper - can take the capacity everyone else needs. Limit per client identity (API key, user, or IP for anonymous traffic) at the gateway, before the request costs anything.
The token bucket is the usual algorithm: a bucket holds up to B tokens and refills at r per second; each request takes a token, and a request that finds the bucket empty is rejected. It allows a burst of B requests at once while holding the long-run rate to r. (A fixed window - "100 per calendar minute" - is simpler but lets a client send 200 in two seconds across a window boundary.)
Rejected requests get 429 Too Many Requests with a Retry-After header, so well-behaved clients back off instead of hammering. With several gateway nodes, either keep the counters in a shared store such as Redis (an atomic increment per request), or give each node a fraction of the limit and accept some slack.
Which tool at which layer?
| Need | Where it belongs |
|---|---|
| One address, health checks, spreading load | Load balancer (L4 or L7) |
| Routing by path or header, TLS termination, per-request retries | L7 balancer or gateway |
| Authentication, API keys, quotas, rate limits | API gateway |
| Spreading load across regions | DNS-based or anycast global load balancing (deep dive) |
| Service-to-service calls inside the platform | Client-side balancing or a service mesh (deep dive) |
Going further. The deep dive (next tab) covers how balancers pick servers well at scale (the power of two random choices), retry storms and retry budgets, health checks that take down healthy fleets, why long-lived connections defeat L4 balancing, global load balancing, and rate limiting across many gateway nodes.
Check yourself
Try to answer before opening each one - recalling beats re-reading.
1. Half your requests take 5 ms and a few take 3 seconds. Round robin or least-connections?
Show answer
Least-connections. Round robin keeps handing new requests to a server that's stuck on several 3-second ones; least-connections sends them to whichever server has the fewest in flight.
2. Why does DNS round robin alone make a poor load balancer?
Show answer
It has no health checks, and DNS answers are cached by clients and resolvers - so a dead server keeps receiving traffic until caches expire, with nothing to retry the request on a healthy server.
3. A token bucket has a capacity of 20 and refills at 5 per second. A client has been idle for a minute and sends 30 requests at once. What happens?
Show answer
The bucket is full (20 tokens), so the first 20 go through immediately and the other 10 get 429s. After that the client can sustain 5 requests per second.
Key takeaways
- A load balancer is one address, spread load, health checks and connection draining.
- Round robin for uniform requests; least-connections when request costs vary.
- L4 forwards connections; L7 reads HTTP - path routing, TLS termination, per-request retries.
- Keep servers stateless; sticky sessions tie users to a server that can die.
- Run the balancer redundantly - one balancer is a single point of failure.
- Rate-limit per client at the gateway with a token bucket; answer 429 with Retry-After.