Reiki

Security proxy for agent and LLM traffic: isolates keys, filters what leaves, keeps access reliable

View the Project on GitHub ddproxy/reiki

Failover

Reiki keeps a stream working when one key fails. It tries the next key in the stream’s chain. When no key can serve, it can hand the request to a fallback upstream. This page covers how each failure changes a key, how long the key stays out, and what the client sees.

Chains

[streams.client-a]
token_sha256 = "<sha256 of the stream token>"
chain = ["zen-a", "zen-b", "openrouter-a"]
strategy = "sticky"          # or "round_robin"

Key states

Each key is in one of three states:

stateDiagram-v2
    direction LR
    state "cooling (whole key)" as cooling_key
    state "cooling (one route and model)" as cooling_scope
    available --> cooling_key: quota 429, or a second transient in a row
    available --> cooling_scope: rate-limit 429, or 402
    cooling_key --> available: deadline passes
    cooling_scope --> available: deadline passes
    available --> disabled: 401, or an invalid key in a 400 or 403
    cooling_key --> disabled: 401 or invalid key, cooldown kept
    disabled --> available: enable with reiki keys, or change the key file and reload

A combination cooldown is a fourth, narrower state. The key stays available for every other model and route. Only the (route, model) pair that failed is held back.

Status to class to key effect

Reiki reads the upstream status first. Then it reads the first 64 KiB of the body, lowercased. Body checks are substring matches unless noted.

Upstream status Body check Class Key effect Next step for the request
200-399 none success Transient count reset Response is delivered
401 none unauthorized Disabled Next key
400 JSON pair "reason": "API_KEY_INVALID" unauthorized Disabled Next key
400 credit balance is too low quota_exhausted Cooling, whole key Next key
400 anything else pass-through None Response is relayed
402 body contains fewer max_tokens or can only afford (case-insensitive) pass-through None Response is relayed
402 anything else quota_exhausted Cooling for (route, model) Next key
429 usage exceeded, quota, subscribe to go, free usage limit or insufficient quota_exhausted Cooling, whole key Next key
429 none of the above rate_limited Cooling for (route, model) Next key
403 error code: 10 (edge block page) pass-through None Response is relayed
403 access is disabled, free tier or not enabled access_disabled None Skip key for this request
403 names an invalid, expired or revoked key or token unauthorized Disabled Next key
403 anything else access_disabled None Skip key for this request
5xx access is disabled access_disabled None Skip key for this request
5xx rate limit rate_limited Cooling for (route, model) Next key
5xx anything else transient Cooling after two in a row Skip key for this request
other (404, 422, …) none pass-through None Response is relayed
No response (connect or timeout) none transient Counted as a transient failure See transport rules below

(route, model) means that combination. For a request with no model, the same failure cools the whole key.

With pass_errors = true on the upstream, every status is a pass-through. No key changes state. See providers.

With hold_unauthorized = true, a 401 does not disable the key until a later key serves the same request. See providers.

A second key that refuses the same request with a 401 is not disabled. One request disables at most one key.

Cooldowns

These values are compiled in. The config file has no setting for them.

Class Cooldown Scope
unauthorized Until enabled again. No timer. Key
rate_limited Retry-After when present (zero means no cooldown), else 60 seconds (route, model)
quota_exhausted (402) Same as the quota row below (route, model)
quota_exhausted (429) Same as the quota row below Key
access_disabled None. The key is skipped for the request only. None
transient 30 seconds, after two failures in a row with no success between Key

Quota cooldown takes the first of these that applies:

Order Source Cooldown
1 quota_resets on the key: daily HH:MM zone or monthly D HH:MM zone Until the next reset
2 Retry-After on the response, non-zero That long
3 Neither 5 hours

Retry-After is capped at 24 hours, whether it is seconds or an HTTP date. An HTTP date in the past gives zero. For a quota, zero counts as absent, so the next rule applies.

Since 0.10.3, a quota 429 with Retry-After cools the key for that long, not for the 5-hour default. A configured quota_resets still wins over the header. With neither, the 5-hour window applies. The test a_quota_429_with_retry_after_cools_the_key_for_that_long in tests/exhausted.rs covers this.

A cooldown only moves later. A shorter failure does not shorten a longer cooldown. A later, longer failure extends it.

Scoped cooldowns

A scope is the pair of route and model:

Only two failures are scoped: a rate_limited failure and a 402 that is treated as quota_exhausted (not a fewer max_tokens/can only afford request-size pass-through). A quota 429 is always key-wide, as are 401s, transients and any failure on a request with no model.

A key with many cooling models is treated as broken. scope_escalate_after sets the count per upstream. The default is 3, and the allowed range is 2 to 16. When three different models are cooling on a key, or the combinations reach twice that count, the next scoped failure cools the whole key for its duration. A whole-key cooldown clears the scoped entries.

Each key remembers at most 64 combinations. When the map is full, the soonest-ending entry is dropped. Scoped cooldowns are held in memory only. They are not saved to the state file.

The cooling_combinations count in a peer’s view is the number of combinations cooling on that upstream’s keys. See peers.

Client responses when keys run out

Situation What the client gets
No key can serve, and no refusal was recorded during the walk 503, JSON code reiki_exhausted, Retry-After
No key can serve, and every usable key is disabled 503, reiki_exhausted, no Retry-After
A refusal was recorded during the walk (access_disabled, a transient 5xx, or a second 401) The upstream’s own status and body from the last refusal
The chain is out and a fallback is configured The fallback’s answer, under the same rules
The stream’s rate_limit_per_minute is exceeded 429, rate_limited, Retry-After
Every key is shaped out and no refusal was recorded 429, reiki_shaped, Retry-After, x-reiki-origin: local

Retry-After on reiki_exhausted is the shortest wait until any non-disabled key recovers. It is rounded up to whole seconds and is at least 1. When the request has a model, the wait accounts for that model’s cooldowns too.

A refusal does not cool the key. The client sees the upstream’s error, which says more than reiki_exhausted.

Confirming a quota 429

confirm_quota = true is set per upstream. The default is false. With it, the first quota 429 from a key in a request is not recorded. Reiki waits 3 seconds and sends the request to the same key again. Only a second quota 429 from that key in the same request cools it. Use this for an upstream that sometimes returns a quota 429 for a good key. It does not apply to 402.

Fallback upstream

[upstream.zen.fallback]
upstream = "openrouter"
model_aliases = { "*" = "openrouter/free" }

What happens after bytes reach the client

Reiki chooses a response at its status line. Once it has a 2xx or 3xx, it streams that response. There is no failover after that point. If the body breaks mid-stream, the client sees the error. The key is not cooled.

Error bodies are read in full before the decision is made, up to 64 KiB. Only the final answer reaches the client.

Transport failures

A transport failure counts as transient against the key.

Failure Method Behaviour
Connection refused before any bytes were sent Any Next key
Timeout or reset, bytes may have been sent GET, HEAD, PUT, DELETE, OPTIONS, TRACE Next key
Timeout or reset, bytes may have been sent Any other method, such as POST or PATCH Not replayed. The client gets 504 gateway_timeout on a timeout, or 502 bad_gateway otherwise.

Replaying a non-idempotent request could duplicate a side effect, so Reiki does not replay it.

Re-enabling keys

Watching failover

Where What it shows
reiki keys Per key: ok, cooling or disabled, cooldown seconds, last status, streams. See cli.
reiki keys --json Adds a scopes array for keys with cooling combinations.
GET /status (admin) keys[].state is available, cooling or out_of_rotation. Also retry_after_seconds, cooldown_class, consecutive_failures, last_status_code and scopes. See monitoring.
reiki top Live view of cooldowns and failures. See top.
/metrics reiki_cooldowns_total{key,class,scope} and reiki_failovers_total{stream,key,class}. Needs the metrics plugin.
Log upstream key failed (with class, status, whether it cooled and seconds remaining), falling back, key refused this request, trying the next key, and not replaying a non-idempotent request.

For a 503 that does not clear, check the log for upstream key failed and whether the key shows as disabled. The troubleshooting page covers the status codes Reiki sends.