Skip to content
atom®technologiesBook a call
Performance and reliability

Shedding order for an overloaded SaaS platform: the pre-incident test that settles it

The Google SRE Book supports per-customer limits and degraded responses as deliberate shedding mechanisms; this article proposes a controlled overload test that compares them with arrival-order errors against per-class SLOs before you commit.

A packed auditorium during a technology keynote

The short answer

Treat the shedding order as a decision your own test settles, not a rule to look up. The Google SRE Book's Handling Overload chapter supports two deliberate mechanisms: per-customer limits that confine errors to misbehaving customers, and degraded responses that are cheaper to compute. The evidence to collect first is an SLI and SLO per request class, then a controlled overload test comparing arrival-order errors, per-customer limits, and a documented business-operation priority against those same targets.

What the SRE Book supports

The Handling Overload chapter states that no matter how efficient a load balancing policy is, eventually some part of the system will become overloaded, and that gracefully handling overload conditions is fundamental to running a reliable serving system. It names two mechanisms that make shedding deliberate rather than accidental. One is serving degraded responses: responses that are not as accurate as or contain less data than normal responses but are easier to compute, such as searching only a small percentage of the candidate set or relying on a local copy of results that may not be fully up to date. The other is per-customer limits: when global overload occurs, the chapter states it is vital that the service only delivers error responses to misbehaving customers while other customers remain unaffected. It also caveats that under extreme overload a service might not be able to compute and serve degraded responses at all, and may have no immediate option but to serve errors.

The case against arrival-order errors is a reasoned inference, not a source rule. When a platform sheds in arrival order, the requests that die are whichever arrived after capacity ran out, with no relation to what the business needs to keep completing; a payment confirmation and a background poll have the same chance of being dropped. Whether that matters for your platform depends on your request mix, which is a hypothesis to check.

Check the capacity model before comparing orders

The Handling Overload chapter reports that modeling capacity as queries per second, or using static request features as a proxy for resource consumption, often makes for a poor metric, because different queries have vastly different resource requirements and the ratios can change, sometimes drastically, with new software versions. The chapter's stated preference is to measure capacity directly in available resources, with a concrete example of a service reserving 500 CPU cores and 1 TB of memory in a datacenter and using those numbers to model capacity. It reports that in a majority of cases, though not all, CPU consumption alone has worked well as the provisioning signal, partly because garbage-collected platforms translate memory pressure into CPU consumption. Its load-balancing example shows the arithmetic the chapter itself calls insufficient at scale: 100 backend tasks each processing up to 500 requests per second cap a datacenter at 50,000 queries per second.

If your shedding thresholds are expressed in requests per second, a hypothesis to check on your own system, then any comparison of shedding orders runs against a moving target. Verify the model first; the chapter's evidence says the QPS proxy often performs poorly, not that it always does.

Give each request class a measurable target

The comparison only works if you can tell whether a business operation completed, not just whether the service looked healthy. The SRE Book's Service Level Objectives chapter defines an SLI as a carefully defined quantitative measure of some aspect of the level of service provided, and an SLO as a target value or range of values for that indicator. The chapter states that most services consider request latency a key SLI, with error rate, often expressed as a fraction of all requests received, and throughput, measured in requests per second, as other common indicators; availability is often defined as the fraction of well-formed requests that succeed. Its worked example sets an SLO that average search request latency should be under 100 milliseconds, and it notes the industry convention of expressing high availability in nines: 99 percent is 2 nines, 99.999 percent is 5 nines, and the published target for Google Compute Engine availability is 99.95 percent, described as three and a half nines. The chapter also warns that choosing an appropriate SLO is complex, and that for incoming HTTP requests the queries-per-second metric is essentially determined by users, so an SLO cannot really be set for it.

The proposed diligence step, which goes beyond what the chapters prescribe: inventory your request classes by the business operation each one completes, and assign each class an SLI and an SLO before any test. Without that mapping, an aggregate error rate cannot distinguish a dropped payment from a dropped poll, and every shedding order looks equally fine.

The test that compares the three orders

This is a proposed test design, not a claim that one order is universally correct. Run a controlled overload exercise in a staging or shadow environment where you can drive load past capacity on purpose. Compare three policies against the same SLOs: errors in arrival order, errors above a per-customer limit, and shedding by a documented business-operation priority. Record, per request class, whether the named operation completed and at what latency, and record which customers saw errors under each policy. Business-priority shedding enters the test as a hypothesis with the same standing as the other two; if arrival-order errors preserve your critical operations just as well, the simpler policy is the reasonable choice, and the test exists to make that visible either way.

Treat the result as a starting decision, not a settled one. A single controlled exercise gives you a defensible default that should be re-measured as your request mix, capacity and client behavior change.

Check retry behavior before trusting the result

One condition deserves a check: what clients do when you reject them. The Amazon Builders' Library article Making retries safe with idempotent APIs states that retrying a service call as mitigation for a transient fault rests on a simplifying assumption that the operation can be retried without side effects, and that AWS uses idempotent API operations to mitigate the undesirable side effects of retries. Its example is the EC2 RunInstances operation for a singleton workload: a network timeout leaves the provisioning process unsure whether the workload is running, and a naive retry could produce multiple workloads. The article notes that AWS SDK implementations automatically retry requests that fail from network IO issues, server-side faults, or service rate limiting.

The inference to check on your platform, stated as an unverified hypothesis: if clients or SDKs retry the requests your policy rejects, the rejection could change the intended load reduction, and retries could duplicate side effects where operations are not idempotent. The AWS article supports the side-effect premise but does not quantify retry amplification, and it is undated vendor documentation about AWS services, so measure the actual effect in your own test rather than assuming a multiplier.

Who owns each step

This is a suggested assignment, not a claim about how your organization must work. The product owner is a natural candidate to name the business-operation priority, since that role carries the context on which operations must keep completing and what degradation customers will accept. The SRE or platform owner can define the SLIs, the SLOs and the per-customer limits, and own the capacity model expressed in resources rather than requests per second. The service owner implements the degraded responses and the error behavior, and instruments each request class so the test can record completions per class. The inference behind the split: if nobody owns the priority list, the default silently becomes arrival order, which is the outcome this work exists to prevent.

Conditions that change the recommendation

Four conditions should reopen the decision. First, if the controlled test shows degraded responses cost more to compute than they save under your actual request mix, the Handling Overload chapter's extreme-overload caveat applies and errors arrive sooner than modeled. Second, if your request mix shifts so that a class that was cheap becomes expensive, the chapter's warning that static proxies drift means your per-customer limits and capacity model need re-measurement, not just re-tuning. Third, if a customer contract or regulator defines which operations must complete, that external constraint overrides the tested ordering wherever they conflict. Fourth, if measured retry behavior turns rejections into additional load, the shedding policy needs to account for it. Absent those, the tested order is your starting default, revisited as the mix, the contracts and the clients change.