Tail latency: separate queueing delay from service time

Locate the delay before choosing the change

A slow request can spend much of its time waiting for a worker to become available. Optimizing the handler requires different evidence from changing the queue or worker allocation. Google's SRE guidance describes how requests waiting ahead of a thread pool add latency, and how resource exhaustion can make that queue grow.

Start with a named boundary: a request enters a queue in one process, a worker starts it, and that worker finishes. Record all three timestamps for the same request attempt. The interval before the worker starts is queueing delay. The interval from start to finish is service time for this worker boundary.

That definition of service time includes any waits inside the handler, such as a database call or another queue. It is not a CPU-time measurement. If that interval dominates, inspect its internal work and dependency waits before deciding which code or resource to change.

Work through one request on one clock

Suppose a request enters the queue at zero milliseconds, starts on a worker at 80, and finishes at 100. Its queueing delay is 80 milliseconds and its worker service time is 20. The observed segment lasts 100 milliseconds. These are invented timestamps for a calculation, with no measured service result.

Use an elapsed-time clock appropriate to the runtime and keep the three observations in the same clock domain. Node.js, for example, documents high-resolution process timing relative to an arbitrary origin, with duration obtained by subtracting readings. Wall-clock timestamps from different hosts need a separate treatment of clock error before their differences can establish a short wait.

If the worker interval fell from 20 to 10 milliseconds while this request's queueing delay stayed fixed, the segment would fall from 100 to 90. That arithmetic is not a prediction under load: faster service can also change how long later requests wait. A comparison must measure both intervals again.

Keep the full request and its attempts visible

Retain the caller's total latency alongside the measured segment. A missing portion could belong to connection acquisition, transit, another queue, or work after this worker finishes. Instrument the suspected boundary before assigning that remainder to a cause. Pair observations by request and attempt identity so a retry cannot borrow another attempt's timing.

Collect outcomes with the durations. Google SRE distinguishes successful and failed request latency because fast errors can distort a combined latency figure. Record rejected, cancelled, and timed-out attempts as well as successful ones. A request that leaves the queue without reaching a worker needs an outcome and observed wait, even though it has no completed service interval.

Use the same workload class and observation window for the comparisons. Inspect slow requests with their queue intervals and downstream activity intact. An overall resource average or an isolated slow span can suggest where to investigate; neither alone identifies why a particular request missed its deadline.

Calculate the percentile of the measured whole

Per-request intervals can add when they partition the same path. Their percentiles generally cannot. Consider 100 invented observations: 98 have zero queueing and 10 milliseconds of service; one has 90 milliseconds of queueing and 10 of service; one has zero queueing and 100 of service. Their totals are 98 observations of 10 milliseconds and two of 100.

Define p99 here by sorting 100 values and taking the 99th. Queue p99 is zero, service p99 is 10 milliseconds, and total p99 is 100 milliseconds. Adding the first two would report 10. The slow queue and slow service belong to different requests.

Calculate the total distribution from total observations. State the percentile method, population, and window. Prometheus also cautions against combining precomputed summary quantiles across replicas; compatible distributions must be aggregated before calculating a fleet percentile. Its warning concerns replica aggregation, while the worked example establishes the separate error in adding stage percentiles.

Check whether the observer waited too

A workload tool may start an operation later than scheduled. PostgreSQL 18's pgbench documents this as schedule lag: under rate control, its reported latency starts at the scheduled time and includes that lag. It reports the lag separately. This is a useful example of preserving delay before actual execution begins.

Keep generator scheduling delay separate from the service's queueing measurement. If the observer cannot maintain its intended arrival schedule, the offered workload has changed. Retain scheduled and actual starts so the reviewer can tell whether a result describes server behavior, delayed dispatch, or both. No workload was generated for the arithmetic examples here.

Choose one intervention and remeasure the path

If queueing dominates while worker intervals remain similar, investigate admission, worker availability, and the resources those workers share. If worker intervals grow, inspect their internal waits and resource use. More workers may shift contention to a dependency. Treat either diagnosis as a hypothesis until a controlled comparison explains the change.

Rehearse one bounded intervention on an environment your team controls, keeping the workload and measurement definitions comparable. Retain queue and service distributions, caller latency, outcomes, and the relevant resource observations. Set acceptable behavior before interpreting the result, including failures and requests that never reach a worker.

A useful assessment identifies the owner of the waiting boundary and the evidence that supports the next change. Its deliverable should let another engineer distinguish a verified reduction in user-visible delay from a shorter interval inside an otherwise unchanged slow request.