Skip to content
atom®technologiesBook a call
Engineering notes

Load-test validity: when does a result support production capacity?

A request rate is meaningful only with its workload and environment. Use a catalog-cache example to examine data distribution, arrival patterns, shared resources, and overload behavior before relying on a load test.

A large data center hall

State the production decision the test must support

A team wants to use a catalog API load test to approve a product launch. The generator repeatedly requests a small set of popular products, while the launch is expected to bring a broader selection of searches and product views. A successful run would establish behavior for the submitted workload. The launch decision also needs a reason to believe that workload represents the demand the service will receive.

Bound the decision before configuring the generator. Identify the proposed deployment, expected request mix, and response objectives. Include the overload outcome the application must preserve when demand exceeds its supported range. This catalog example is a proposed test design, with illustrative arithmetic and no observed benchmark or production-capacity claim.

Maintain an assumption record beside the result. For each material workload or environment property, state its evidence and the uncertainty that remains. A missing production trace can justify a range of scenarios; it cannot establish that an arbitrary synthetic distribution is representative.

Make the data exercise the intended work

Describe product cardinality and request popularity separately. A large database can still receive requests for the same few keys. A small fixture may fit entirely in cache even when the production working set does not. Include relevant payload sizes, filter selectivity, and the largest supported responses. Uniform random keys and one repeatedly requested key are different assumptions about customers.

The consequence is visible in a small calculation. Assume 1,000 product requests per second, one datastore lookup per cache miss, and none per hit. At a 95 percent hit rate, the lookup rate is 50 per second. At 70 percent, it is 300 per second: six times the lookup rate at the same incoming request rate. Request coalescing or additional lookups would change this model.

Preserve the workflow mix too. If production performs catalog updates while serving reads, a read-only test leaves that contention unexamined. Identify whether the run begins with a cold or warm cache and how the cache reaches that state. Use synthetic or appropriately sanitized records that preserve the relevant sizes and distributions without exposing customer data.

Specify arrivals and concurrency as different inputs

A fixed number of users waiting for each response represents a population that slows its next action when the service slows. Independently arriving requests represent a different source of demand. Grafana k6's documentation distinguishes these closed and open workload models. Choose the model that matches the production behavior being evaluated; neither a virtual-user count nor an arrival-rate setting describes the complete workload.

Record burst shape and the overlap of operations on shared resources. Requests spread evenly across accounts can miss contention that appears when many callers use one account or product. Keep session sequencing and connection reuse consistent with the relevant clients. Specify what client retries do during overload so they do not become an unrecorded source of traffic.

Check the generator's achieved schedule and its own resource use. A run that cannot start the intended work leaves the target rate untested. Record dropped or delayed starts separately from server rejection. Otherwise a generator limit can look like a stable service limit.

List the environment differences that can change the answer

Microsoft's performance-testing guidance calls for realistic data and an environment that mirrors production as closely as practical. For this catalog path, capture the service release, resource limits, and replica count. Record database size and indexing, cache capacity, network placement, and any dependency limits used by the request. Keep the observed deployed configuration with the run.

A local dependency stub with immediate responses removes its latency and contention from the experiment. A test database with a different working set may exercise different storage behavior. A smaller environment can still compare a bounded change, but multiplying its result by the planned replica count requires evidence that shared dependencies will scale with it.

Record autoscaling policy and starting capacity when scaling behavior belongs to the launch decision. A run that starts fully expanded cannot demonstrate how the service handles demand before extra instances become ready. State which differences restrict the conclusion and which additional observation could resolve them.

From launch assumptions to an applicable resultThe test supports only the workload, environment, and operating behavior it actually exercises. Preserve these conditions with the result.
  1. Decision
    Name the launch requirement

    Set request mix, response objectives, and the required overload outcome.

  2. Model
    Bind workload and environment

    Record data distribution, arrivals, shared-resource overlap, deployment, and unresolved differences.

  3. Exercise
    Run a bounded comparison

    Control one material assumption and observe both the generator and the service.

  4. Conclude
    State what the evidence permits

    Approve only the exercised range; retain recovery observations and explicit limits on extrapolation.

Use a bounded comparison to challenge one assumption

For the first catalog experiment, hold deployment, request mix, and arrival schedule fixed. Compare a concentrated key distribution with a broader distribution over the same dataset. Restore the same initial cache state before each run and retain the random seed or request sequence. Measure the resulting hit rate and datastore activity instead of assuming the generator produced either illustrative percentage.

Use two runs with a fixed duration and request ceiling agreed before execution. Select those bounds to cover the workload condition being investigated; a short cache experiment cannot establish endurance. Record response correctness as well as latency and rejected work. A fast response for the wrong product must fail the experiment.

In a separate, explicitly bounded overload phase, establish whether excess demand produces the agreed rejection or degraded behavior and whether the service recovers when demand falls. Google's SRE testing guidance uses stress tests to investigate service limits because components may fail abruptly beyond them. Set stop conditions before the run, especially for shared dependencies.

The decision record should retain the workload and environment manifests, actual arrival evidence, observed results, and unresolved assumptions. If broader keys break the objective, investigate that boundary before approving the launch. If both runs pass, the result supports those exercised conditions; an untested launch burst or different shared database remains an open part of the decision.