Faults, latency, credentials, inspection

These request and response headers make the mock LLM useful behind an AI gateway: they trigger the failures a gateway must handle, slow streams down, prove which credential arrived and expose what the upstream received.

Response headers

Every provider response (the /ai/* routes other than /ai/requests) carries:

Faults

X-Rustybin-Fail: <kind>[:<percent>] answers with the provider's native error (status, body shape and headers), always or with the given probability. The query string works too, for clients that cannot set headers: ?fail=<kind> and ?fail_rate= (a probability 0.3 or a percent 30; fail_rate alone means 503).

Faulted responses carry X-Rustybin-Fault: ai. The X-Rustybin-Delay header (see Fault injection) also works on /ai/*. Every fault that fires (content_filter included) is counted in rustybin_llm_faults_total{provider,kind} and every served request in rustybin_llm_requests_total{provider,model_family,streaming}, next to the token counter (see Metrics).

POST {{base_url}}/ai/openai/v1/chat/completions
X-Rustybin-Fail: 429
Content-Type: application/json
{"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]}
HTTP 429
[Asserts]
header "X-Rustybin-Fault" == "ai"
header "Retry-After" == "2"
header "x-ratelimit-remaining-requests" == "0"
jsonpath "$.error.code" == "rate_limit_exceeded"

POST {{base_url}}/ai/anthropic/v1/messages
X-Rustybin-Fail: overloaded
Content-Type: application/json
{"model": "claude-sonnet-4-5", "max_tokens": 10, "messages": [{"role": "user", "content": "hi"}]}
HTTP 529
[Asserts]
jsonpath "$.type" == "error"
jsonpath "$.error.type" == "overloaded_error"

POST {{base_url}}/ai/gemini/v1beta/models/gemini-2.5-flash:generateContent?fail=rate_limit
Content-Type: application/json
{"contents": [{"role": "user", "parts": [{"text": "hi"}]}]}
HTTP 429
[Asserts]
jsonpath "$.error.status" == "RESOURCE_EXHAUSTED"

POST {{base_url}}/ai/openai/v1/chat/completions
X-Rustybin-Fail: context_length
Content-Type: application/json
{"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]}
HTTP 400
[Asserts]
jsonpath "$.error.code" == "context_length_exceeded"

# content_filter answers 200 with the provider's filtered finish reason.
POST {{base_url}}/ai/openai/v1/chat/completions
X-Rustybin-Fail: content_filter
Content-Type: application/json
{"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]}
HTTP 200
[Asserts]
jsonpath "$.choices[0].finish_reason" == "content_filter"
jsonpath "$.choices[0].message.content" == null

# Percent suffix or ?fail_rate=: 0 never fails.
POST {{base_url}}/ai/openai/v1/chat/completions?fail=503&fail_rate=0
Content-Type: application/json
{"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]}
HTTP 200

Latency and streaming pace

Latency values are capped like X-Rustybin-Delay (30 s, 10 s in public mode) and the pacing of one stream at 60 s (20 s in public mode). Use them for timeout, time-to-first-token and streaming-throughput policies.

POST {{base_url}}/ai/openai/v1/chat/completions
X-Rustybin-Latency-Ms: 200
X-Rustybin-TTFT-Ms: 100
Content-Type: application/json
{"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]}
HTTP 200
[Asserts]
duration >= 300

# 20 tokens per second: a 10 token stream takes about half a second.
POST {{base_url}}/ai/openai/v1/chat/completions
X-Rustybin-Tokens-Per-Second: 20
Content-Type: application/json
{"model": "gpt-4o", "stream": true, "max_tokens": 10, "messages": [{"role": "user", "content": "hi"}]}
HTTP 200
[Asserts]
duration >= 400

Credential checks

Each provider looks for its own credential:

The credential that was seen is always reported in X-Rustybin-Credential (redacted). Checks are enforced only when asked for: per request with X-Rustybin-Require-Auth: true, or for the whole instance with RUSTYBIN_AI_REQUIRE_AUTH=true or RUSTYBIN_AI_API_KEY=<key> (then only that exact key is valid; for SigV4 it is compared with the access key id). Failures use the provider's native status and body (401; 403 for Gemini and Bedrock when the credential is missing).

The classic demo: the client sends no key, the gateway injects the real one from its vault, and X-Rustybin-Credential proves it arrived.

POST {{base_url}}/ai/openai/v1/chat/completions
X-Rustybin-Require-Auth: true
Content-Type: application/json
{"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]}
HTTP 401
[Asserts]
header "X-Rustybin-Credential" == "none"
jsonpath "$.error.type" == "invalid_request_error"

# A gateway injecting the key makes the same call succeed; the redacted
# credential proves which key arrived.
POST {{base_url}}/ai/openai/v1/chat/completions
X-Rustybin-Require-Auth: true
Authorization: Bearer sk-gateway-injected-1234
Content-Type: application/json
{"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]}
HTTP 200
[Asserts]
header "X-Rustybin-Credential" == "bearer ****1234"

POST {{base_url}}/ai/anthropic/v1/messages
X-Rustybin-Require-Auth: true
x-api-key: sk-ant-demo-5678
anthropic-version: 2023-06-01
Content-Type: application/json
{"model": "claude-sonnet-4-5", "max_tokens": 5, "messages": [{"role": "user", "content": "hi"}]}
HTTP 200
[Asserts]
header "X-Rustybin-Credential" == "x-api-key ****5678"

POST {{base_url}}/ai/gemini/v1beta/models/gemini-2.5-flash:generateContent?key=AIzaDemo9999
X-Rustybin-Require-Auth: true
Content-Type: application/json
{"contents": [{"role": "user", "parts": [{"text": "hi"}]}]}
HTTP 200
[Asserts]
header "X-Rustybin-Credential" == "query-key ****9999"

POST {{base_url}}/ai/bedrock/model/amazon.nova-pro-v1:0/converse
X-Rustybin-Require-Auth: true
X-Amz-Date: 20260101T000000Z
Authorization: AWS4-HMAC-SHA256 Credential=AKIADEMOKEY0001/20260101/us-east-1/bedrock/aws4_request, SignedHeaders=host;x-amz-date, Signature=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
Content-Type: application/json
{"messages": [{"role": "user", "content": [{"text": "hi"}]}]}
HTTP 200
[Asserts]
header "X-Rustybin-Credential" == "sigv4 ****0001"

Request inspection

GET /ai/requests/{id} (the id from X-Rustybin-Request-Id) shows what the upstream received: provider, endpoint, model, mode, stream flag, headers with credentials redacted, the raw body, the normalised prompt (the same rendering as echo mode), token usage, finish reason, a preview of the reply and, under captured, the matching request inspector entry (status, latency, client IP). GET /ai/requests lists recent exchanges, newest first (?limit=, default 50, at most 200; ?session=).

Records are kept for an hour, at most 1000 (200 in public mode). In public mode callers only see their own session (X-Rustybin-Session or the client IP).

POST {{base_url}}/ai/openai/v1/chat/completions
X-Request-Id: docs-ai-0001
Authorization: Bearer sk-secret-abcd
Content-Type: application/json
{"model": "gpt-4o", "messages": [{"role": "system", "content": "Injected by the gateway"}, {"role": "user", "content": "hello"}]}
HTTP 200
[Asserts]
header "X-Rustybin-Request-Id" == "docs-ai-0001"

GET {{base_url}}/ai/requests/docs-ai-0001
HTTP 200
[Asserts]
jsonpath "$.provider" == "openai"
jsonpath "$.model" == "gpt-4o"
jsonpath "$.headers.authorization" == "Bearer ****abcd"
jsonpath "$.normalized_prompt" == "system: Injected by the gateway\nuser: hello"
jsonpath "$.usage.total_tokens" > 0
jsonpath "$.captured.inspector_id" exists

GET {{base_url}}/ai/requests?limit=5
HTTP 200
[Asserts]
jsonpath "$.requests[0].request_id" == "docs-ai-0001"