Signals considered

The client names a time window and a failure ratio rather than saying "it is bad"
Failures carry readable HTTP status codes or error codes that allow layering
Provider status, your own routing configuration and client retry records are not yet aligned

“We keep failing at peak hours. Is something wrong with your route?”

That message shows up in the client group of an AI API reseller. Two reactions are common. One is to apologise and spend the night retuning routing weights. The other is to ask questions first, until the client concludes you are dodging.

Both reactions share an assumption: that you already know what “unstable” refers to. In the composite scenario below, that assumption is the problem.

The short version: an “unstable” complaint has to be split into three things and evidenced separately - what status the provider returned, how your routing and keys are configured, and whether the client is retrying without backoff. The order matters, because the category most often mistaken for a provider fault originates on the client side.

NOTICE: the clients, groups, messages and usage figures in this piece are composite illustrations used to demonstrate an order of judgement. They do not represent real clients, incidents, contracts or closed deals.

“Unstable” does not say which layer failed

Read the message again. The only extractable facts are a time window, peak hours, and a symptom, failures. Which layer failed, at what ratio, and what the error looked like are all absent.

That is not the client being vague. “Stable” is a word grown from the experience of using something. It describes an outcome, not a cause. Beneath it there are at least three possibilities.

The provider really is degrading or overloaded. Your own routing, weights or key reuse has a configuration problem. Or the client’s retry logic has turned one ordinary rate limit into a stampede. All three can coexist, but in any single incident one of them is the main cause. Finding it is what the industry usually calls attribution, and it determines what you change next - and more importantly, whether you should be the one changing it.

Turn it into fields you can check against

The first step in attribution is not investigation. It is replacing a claim you cannot verify with fields you can. These four are usually available within an hour, and they do not require much cooperation from the client:

The time window in which failures occurred, down to the minute
The distribution of HTTP status codes on failed requests, for example how much is 429 versus 503
The error code and error type string in the response body
The total request volume curve your side recorded in the same window

The third one gets skipped most often. Many teams look only at the status code: 429 means rate limiting, 503 means the provider is down. That reading is right most of the time, but it misses one important case, and that case is routine in this business.

429 and 503 point in opposite directions

In the provider’s own documentation the two errors are handled separately.

429 with the rate_limit_error type means a rate limit was touched. 503 with the service_unavailable_error type means the requested model is temporarily overloaded. The first is a quota problem, the second a capacity problem. For the first you adjust your own request rhythm; for the second there is very little you can do beyond honouring Retry-After and retrying later, or switching routes.

The consequence for attribution is this: 503 plus model overload is one of the few signals that places responsibility on the provider directly. If the failure window is dominated by 503 rather than 429, then “the route is unstable” is broadly pointed in the right direction, and the conversation moves to backup routes and degradation policy.

If the window is almost entirely 429, the direction flips. Because 429 has a subdivision that almost nobody reads carefully.

slow_down: your client’s ramp rate, not the provider’s outage

The same documentation contains a paragraph worth reading word for word:

A slow_down error can occur even when your traffic is within its requests-per-minute and tokens-per-minute limits. It reflects how quickly traffic increased, not whether you exhausted those limits.

The published rule of thumb: once traffic reaches one million input tokens per minute, it should not grow faster than 50 percent every 15 minutes. Above that slope, slow_down starts appearing.

This sentence rewrites the default reading of 429. The 429s you see at peak hours are quite possibly not because anyone exhausted a quota, but because the rate of increase exceeded the ramp slope the provider allows. And that growth rate is set by the party sending requests - your client, or the retry code your client wrote.

At this point the attribution target has shifted from “is the provider stable” to “who is sending requests at what slope”.

The retry amplifier: why “send it again” makes it worse

The next point essentially locks responsibility to the client side.

The provider’s quota documentation carries a warning: Unsuccessful requests still count toward your per-minute rate limit. Continuously resending a request without backing off makes throttling worse.

Behind that sentence sits a common implementation defect. A client receives a 429, retries immediately, fails again, retries immediately again. If it also runs concurrency, every failure multiplies by the concurrency factor and becomes a self-amplifying request surge. What you see is a sudden jump in failure rate. What the provider sees is a key whose request volume multiplied within seconds. The actual cause is in the client’s retry loop with no backoff.

You do not need to guess. Look at your own volume curve: if failed requests and total requests rise steeply in the same window, and total volume climbs well beyond normal business variation, retry amplification is almost certainly present. Ordinary quota exhaustion is a gently rising line. Retry amplification is a needle standing straight up.

One change worth recommending to the client right away: most official SDKs already retry with exponential backoff and jitter, typically around two attempts by default. If the client wrapped its own “retry immediately on failure” logic around that, it overrides the SDK’s protection.

The layer people skip: routing config and shared keys

Once provider overload and retry amplification are ruled out, what remains usually sits on your side, concentrated in one place: several clients or business lines sharing the same upstream key.

Rate limits attach to the key, not to the client. The moment two clients’ traffic runs through one key, you have tied their quotas, their failure retries and their traffic peaks together. Client A runs a batch job at peak hours while Client B runs live conversations, and what Client B feels is “an unstable route”.

Evidence for this layer is also direct: group the requests in the failure window by upstream key and look at the curves. If one key’s volume equals the sum of several business lines, this layer is the main cause. What you change then is not routing weight but the granularity of key isolation.

Change one variable at a time

By this point you have enough evidence to form a hypothesis. The next step has one hard constraint: change one variable at a time.

Adjusting routing weights, scaling up and asking the client to fix its retry logic simultaneously is the most expensive bad habit in this business. If the metrics improve, you cannot tell which change did it. If they worsen, you cannot tell what to roll back, and the client’s patience is being consumed by the hour.

A steadier order: first have the client stop retrying without backoff and watch one window; then handle key isolation; only then touch upstream routes. The cost of this order is speed. The benefit is that each action can be verified, and only verified actions are reusable.

When this triage does not hold

Everything above rests on one premise: layered evidence is available to you. In the following cases the method fails and you need a different approach.

First, the client will only offer “it does not work”, with neither a time window nor a failure sample. There are no fields to split, so run a minimal reproduction first and do not promise anything on instinct.

Second, the provider collapses error codes into a generic 500 or a bespoke error body, so status codes no longer carry clear meaning. The “layer by status code” step is void, and you fall back on request curves cross-checked against the provider’s public status page.

Third, the failure window is too short. A few minutes of jitter often falls below the aggregation granularity, so no curve exists and only scattered log samples remain. Drawing conclusions here is risky; recording it as “to be observed” is usually more honest than forcing an attribution.

One boundary worth stating: this piece only handles the question of which layer is responsible. It does not cover how to design a high-availability architecture, nor whether to switch providers. The first is discussed separately on this site, and the second is a commercial decision that should not be derived from a single incident’s attribution.

How a Signal worth attention is found

See how Top Prospect finds and organizes Signals worth checking, keeps the original Telegram context, removes duplicates, and helps you decide what to review first. You decide whether to follow up and what to do next.

Open the methodology and core definitions

"Your route is unstable": separate a provider fault from your own config and your client's retries