How Kunavo fails over: the four layers between an upstream error and your client

Every gateway post you read quotes an availability percentage. This one does not, because we do not publish an SLA and we would rather describe a mechanism you can check than a number you cannot. What follows is the whole path between an upstream returning an error and your client seeing one: four layers, four constants, and the reasons each constant has the value it has.

Why a single upstream is not enough

A model provider having a bad twenty minutes is ordinary. Regional incidents, rate-limiter tightening, a capacity crunch on a newly launched model — none of this is exotic, and all of it is invisible to you until your own error rate moves. If your application talks to one endpoint, your availability is that endpoint's availability and there is nothing you can do about it from the client except retry, by which point the user has already waited.

The alternative is to have somewhere else to go. Kunavo resolves a routing chain for each model — up to three independent channels that can serve it, ordered by cost — and the four layers below are how a request moves through that chain without you writing any retry logic.

Layer 1: the circuit breaker (before the request is sent)

The cheapest failover is the one that never costs a round trip. Each gateway process keeps a rolling five-minute failure count per channel. Three failures inside that window and the channel is considered open: for the next sixty seconds it is removed from the chain before a request is dispatched at all.

// Layer 1, simplified from lib/providers/provider-health.ts.
// A channel that has been failing is skipped BEFORE we send anything,
// so the cost of avoiding it is zero — no request, no timeout, no wait.
const WINDOW_MS   = 300_000;  // failures older than 5 minutes are forgotten
const FAIL_THRESHOLD = 3;     // 3 failures inside that window opens the circuit
const COOLDOWN_MS = 60_000;   // and the channel is bypassed for 60 seconds

function isOpen(channel: string, now = Date.now()): boolean {
  const s = state.get(channel);
  return s != null && s.openUntil > now;
}

function recordFailure(channel: string, now = Date.now()) {
  const s = state.get(channel) ?? { failures: [], openUntil: 0 };
  s.failures = s.failures.filter((t) => now - t < WINDOW_MS);
  s.failures.push(now);
  if (s.failures.length >= FAIL_THRESHOLD) s.openUntil = now + COOLDOWN_MS;
}

This is the only layer that is genuinely instantaneous, and it is worth being precise about why: nothing is being measured quickly. The decision was made up to five minutes ago by an earlier request that took the failure on your behalf. The state is per process, so each machine learns independently — under real traffic a bad channel trips on all of them within seconds of each other, and a channel that was only briefly unwell recovers on its own when the cooldown expires.

Layer 2: in-request failover (up to three hops)

When a channel that looked healthy returns an error anyway, the request falls through to the next one in the chain. The cap is three hops, ROUTING_CHAIN_MAX, and the chain is ordered by cost, so the first attempt is always the cheapest channel that can serve the model.

// Layer 2. The routing chain is resolved from the catalog, cost-ordered,
// and capped at ROUTING_CHAIN_MAX hops (3). It is static configuration, not
// a learned weighting: you can read the exact chain for any model in the
// admin UI, and it is the same chain for every customer.
async function dispatch(req: ChatRequest) {
  const chain = routingChain(req.model)          // up to 3 channels, cheapest first
    .filter((c) => !isOpen(c.id));               // layer 1 removes the known-bad

  let lastError: UpstreamError | null = null;
  for (const channel of chain) {
    try {
      return await callUpstream(channel, req);
    } catch (err) {
      if (!isRetryable(err)) throw err;          // a 400 is yours, not ours
      recordFailure(channel.id, Date.now());
      lastError = err;
    }
  }
  throw lastError ?? new Error("all channels exhausted");
}

Client errors do not fail over. A 400 for a malformed request or a 404 for a model that does not exist is the same answer from every channel, so retrying it three times would only make you wait three times as long for the same message. Only retryable upstream failures walk the chain.

The honest cost of this layer is one failed round trip per hop. If a channel accepts your connection and then 503s after two seconds, you waited those two seconds before the second attempt started. That is the real reason layer 1 matters: it keeps a channel that is reliably failing from charging every single request that two-second toll.

Layer 3: the stream gate (30 seconds)

Streaming makes failover harder in a way that is easy to miss. An upstream can return 200 OK, open a stream, and only then emit an error event. By the time that arrives, a naive proxy has already sent your client a header and some bytes, and there is no longer any way to fall through — the request is committed to a channel that is not going to answer it.

// Layer 3, from lib/providers/stream-gate.ts. An upstream can answer 200 OK
// and then put the error inside the stream. Committing that to your client
// means you get a truncated answer and we cannot fail over any more, so the
// gateway holds the first frames back until it knows which it is.
export const STREAM_GATE_TIMEOUT_MS = 30_000;

// Outcomes:
//   a decisive content frame  -> commit, stream the rest through untouched
//   a decisive error frame    -> discard, fall through to the next channel
//   nothing decisive in 30s   -> commit anyway (a slow model is not an error)

So the gateway does not commit on the status code. It holds the opening frames until it sees something decisive: real content, in which case the stream is committed and passed through untouched from then on; or an error, in which case nothing has reached you and the request continues down the chain. Thirty seconds is the ceiling on that decision, and it exists for reasoning models that legitimately think for a long time before emitting a first token. Past that we commit, because a slow model is not a broken one.

Layer 4: the header timeout (240 seconds)

The last layer is the one nobody enjoys. If an upstream accepts the connection and then returns no response headers at all, the request hangs, and the only thing that ends it is a timeout — UPSTREAM_HEADERS_TIMEOUT_MS, four minutes.

Four minutes is a long time to wait, and it is deliberate. Video and image generation legitimately take minutes; a timeout short enough to feel responsive on chat would cancel video jobs that were going to succeed. A hung upstream is also rare compared to one that fails loudly, and the loud cases are already handled by layers 1 to 3. We would rather be slow in the rare case than wrong in the common one.

If you want a tighter bound than this for your own application, set a client-side timeout. Ours is a backstop, not a latency target.

What we do not claim

• No millisecond failover figure. Layer 1 costs no round trip because the decision predates your request. Layers 2 to 4 cost exactly as long as the failing upstream took to fail. Neither of those is a number we can put on a marketing page honestly.
• No availability percentage. There is no SLA and no credit scheme, so a number would be decoration. The status page shows the success rate we actually observe, including the windows where it was not good.
• No silent model substitution. Failover moves a request between channels serving the same model. We never answer with a cheaper model than you asked for — a practice worth asking any gateway about directly, because it is invisible from the response.

What this means when you are building on it

Mostly it means you can delete a retry loop. Retryable upstream errors are already retried across independent channels before you ever see a failure, so a client-side retry on a 5xx from us is retrying something that has been tried three times. Keep your timeout, drop the backoff.

The one thing worth handling yourself is a 402 insufficient balance, which is not a failover case at all — no amount of retrying refills a wallet. See the error reference for which statuses are worth a second attempt and which are final.