Load Shedding: Failing Fast to Protect a Service Under Overload

When traffic exceeds what a service can handle, queues grow, latency explodes, clients time out and retry, and the extra retries make everything worse. Load shedding breaks that cycle by deliberately rejecting some work early so the rest succeeds.

Shedding vs rate limiting vs backpressure

• Rate limiting enforces a per-client quota regardless of server health.
• Backpressure slows producers by signalling upstream (bounded queues, flow control).
• Load shedding reacts to the server's own health: when it's saturated, it rejects requests quickly, often with HTTP 503 or 429 and a Retry-After header.

Deciding when to shed

1. Concurrency limits: cap in-flight requests; adaptive limits (inspired by TCP congestion control) adjust the cap from observed latency.
2. Queue time: if a request has waited longer than its client's deadline, drop it: the client has already given up.
3. Resource signals: CPU, memory or event-loop lag thresholds.

Deciding what to shed

Not all traffic is equal. Attach a priority to each request (user-facing checkout above background sync above analytics) and shed from the lowest tier first. Some systems also prefer to serve requests from users already mid-session, so partial work isn't wasted.

Make rejection cheap

Shedding only helps if a rejection costs far less than serving. Reject before authentication-heavy or database work, ideally at the load balancer or the first middleware.

Don't forget the clients

• Retries with exponential backoff and jitter.
• Retry budgets so a fleet doesn't multiply load.
• Propagate deadlines so downstream services can shed work nobody is waiting for.

Interview summary

"Bound concurrency, measure queueing delay, and when saturated reject the lowest-priority requests cheaply and early, while clients back off with jitter." Pair it with autoscaling: shedding buys time while new capacity comes up.

Disclosure: from the builders of an AI interview assistant (Linkjob AI).