Control-plane security, metrics and operations

By default the control plane (/_rustybin/*) and the web console are open, which is what you want on a laptop or in a lab. When an instance is reachable by others (a team server, a shared cluster, a hosted offering that runs one instance per customer), lock the control plane down while the data plane stays public:

RUSTYBIN_CONTROL_AUTH=token RUSTYBIN_ADMIN_TOKEN=change-me rustybin

Data-plane routes (everything outside /_rustybin/) are never affected: they are the product your gateway calls.

Modes

Always reachable without credentials, in every mode:

• GET /_rustybin/ready, the readiness probe (see below);
• the data plane, including GET / (liveness) and /health (the demo toggle);
• the console's static files under /ui/. They are the same open-source files for every instance and carry no instance data: a browser must be able to load them to pick up a token from the URL fragment, which never reaches the server. Everything the console shows comes from /_rustybin/* with the token.

Without a token the control plane answers 401 with a WWW-Authenticate: Bearer challenge and a JSON body naming the reason (missing_token, expired, invalid_audience, invalid_signature, invalid_algorithm, not_yet_valid, missing_claim, malformed, invalid_token):

GET {{secured_url}}/_rustybin/status
HTTP 401
[Asserts]
header "WWW-Authenticate" == "Bearer realm=\"rustybin-control\""
jsonpath "$.reason" == "missing_token"
jsonpath "$.control_auth" == "jwt"

GET {{secured_url}}/_rustybin/ready
HTTP 200
[Asserts]
jsonpath "$.status" == "ready"

GET {{secured_url}}/echo
HTTP 200

GET {{secured_url}}/ui/
HTTP 200
[Asserts]
body contains "<meta name=\"rustybin-control-auth\" content=\"jwt\">"
body contains "<meta name=\"rustybin-console-title\" content=\"Docs demo\">"

Unauthenticated control-plane requests are rejected before plan limits count them; authenticated ones are counted but never rejected by the limiter. token mode without RUSTYBIN_ADMIN_TOKEN (or jwt mode with neither a key nor an admin token) refuses every control-plane request and logs a warning at startup.

Signed access tokens (jwt mode)

A portal, a CI job or a script that holds the private key issues short-lived tokens; the instance only needs the public key. A token must have:

A mutation (any method other than GET, HEAD or OPTIONS) needs the admin token or the admin scope; other scopes get 403 with reason: insufficient_scope. Two POST routes change nothing and are open to the console scope: previewing a mock spec (/_rustybin/mocks/import) and validating a body against a mock operation (/_rustybin/mocks/{name}/operations/{op}/validate).

GET {{secured_url}}/_rustybin/status
Authorization: Bearer {{console_token}}
HTTP 200
[Asserts]
jsonpath "$.control_auth" == "jwt"
jsonpath "$.console.title" == "Docs demo"
jsonpath "$.console.backlink" == "https://portal.example.com/instances/docs-secured"

DELETE {{secured_url}}/_rustybin/requests
Authorization: Bearer {{console_token}}
HTTP 403
[Asserts]
jsonpath "$.reason" == "insufficient_scope"
jsonpath "$.scopes" contains "console"

GET {{secured_url}}/_rustybin/requests?limit=5
Authorization: Bearer {{inspector_token}}
HTTP 200
[Asserts]
jsonpath "$.requests" isCollection

GET {{secured_url}}/_rustybin/config
Authorization: Bearer {{inspector_token}}
HTTP 403
[Asserts]
jsonpath "$.reason" == "insufficient_scope"

GET {{secured_url}}/_rustybin/status
Authorization: Bearer {{expired_token}}
HTTP 401
[Asserts]
jsonpath "$.reason" == "expired"

GET {{secured_url}}/_rustybin/status
Authorization: Bearer {{other_audience_token}}
HTTP 401
[Asserts]
jsonpath "$.reason" == "invalid_audience"

The admin token keeps working in jwt mode (for automation such as a usage collector):

DELETE {{secured_url}}/_rustybin/requests
Authorization: Bearer {{admin_token}}
HTTP 200

Keys and tokens with openssl

RUSTYBIN_CONTROL_JWT_PUBLIC_KEY takes a PEM public key (newlines may be written as \n), the base64 of its DER, or the base64 / base64url of the raw 32 bytes.

openssl genpkey -algorithm ed25519 -out control.key
openssl pkey -in control.key -pubout -out control.pub
RUSTYBIN_CONTROL_AUTH=jwt RUSTYBIN_CONTROL_JWT_PUBLIC_KEY="$(cat control.pub)" \
  RUSTYBIN_CONTROL_JWT_AUDIENCE=team-demo rustybin

Minting a token needs nothing but openssl 3 (_mint-control-jwt.sh <key> <audience> [scope] [ttl], used by the documentation runner):

# Optional claim binding the token to one API key (shared mode).
extra=""
if [[ -n "$rb_key" ]]; then
  extra=$(printf ',"rb_key":"%s"' "$rb_key")
fi
now=$(date +%s)
header=$(printf '{"alg":"EdDSA","typ":"JWT"}' | b64url)
payload=$(printf '{"aud":"%s","sub":"docs","scope":"%s","iat":%d,"exp":%d%s}' \
  "$aud" "$scope" "$now" $((now + ttl)) "$extra" | b64url)
input="$(mktemp)"
trap 'rm -f "$input"' EXIT
printf '%s.%s' "$header" "$payload" >"$input"
signature=$(openssl pkeyutl -sign -inkey "$key" -rawin -in "$input" | b64url)
echo "$header.$payload.$signature"

Any JWT library with EdDSA support works too (PyJWT with cryptography, jsonwebtoken in Rust, jose in JavaScript).

The console with a token

Open the console with the token in the URL fragment: https://rustybin.example.com/ui/#token=<jwt> (or #/traffic&token=<jwt> to land on a view). The console keeps the token in memory and in sessionStorage (this browser tab only), removes it from the address bar at once, and sends it as Authorization: Bearer on every control-plane call, including the live traffic feed, which it reads with fetch rather than EventSource so the header can be sent (no token in a query string). When the token is missing, refused or expired (the console also watches the exp claim), it shows a sign-in screen where a token can be pasted, with a link back to RUSTYBIN_CONSOLE_BACKLINK when one is set. Console settings, Sign out forgets the token.

Hosted mode

RUSTYBIN_HOSTED_MODE=true is a preset for platforms that run one instance per customer: control auth jwt and JSON logs. Explicit RUSTYBIN_CONTROL_AUTH and RUSTYBIN_LOG_FORMAT values win over the preset. A typical setup:

RUSTYBIN_HOSTED_MODE=true
RUSTYBIN_INSTANCE_ID=acme                       # also the default JWT audience
RUSTYBIN_CONTROL_JWT_PUBLIC_KEY="<platform public key>"
RUSTYBIN_ADMIN_TOKEN="<random, for the platform's own collector>"
RUSTYBIN_GRPC_ON_HTTP=true                      # gRPC through the HTTPS edge
RUSTYBIN_TRUST_FORWARD=true
RUSTYBIN_CONSOLE_TITLE="Acme demo"
RUSTYBIN_CONSOLE_BACKLINK=https://portal.example.com/instances/acme
RUSTYBIN_USAGE_FILE=/app/certs/usage.json       # on the writable volume

Readiness

GET /_rustybin/ready answers 200 {"status": "ready"} whenever the process serves HTTP, whatever the /health toggle says. It needs no credentials, is never limited by a plan and is never captured by the inspector, so it is the right target for an orchestrator's readiness or health check (a presenter flipping /health to 503 must not get the instance restarted).

Metrics

GET /_rustybin/metrics serves Prometheus text (version 0.0.4), protected like the rest of the control plane (console scope or the admin token):

Every label is bounded: raw paths, query strings and model names never become label values, so a scanner hitting random URLs cannot blow up the series count. Model names map to a fixed list of families, providers and fault kinds come from fixed lists, and every other value becomes other.

GET {{secured_url}}/_rustybin/metrics
Authorization: Bearer {{console_token}}
HTTP 200
[Asserts]
header "Content-Type" startsWith "text/plain; version=0.0.4"
body contains "rustybin_requests_total{route=\"/echo\",method=\"GET\",status_class=\"2xx\"}"
body contains "rustybin_build_info{version="

The mock LLM series after one completion and one injected rate_limit error:

# One mock LLM call and one injected native error, then the LLM series.
POST {{base_url}}/ai/openai/v1/chat/completions
{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hi"}]}
HTTP 200

POST {{base_url}}/ai/openai/v1/chat/completions
X-Rustybin-Fail: rate_limit
{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hi"}]}
HTTP 429

GET {{base_url}}/_rustybin/metrics
HTTP 200
[Asserts]
body matches /rustybin_llm_requests_total\{provider="openai",model_family="gpt-4o",streaming="false"\} [1-9]/
body matches /rustybin_llm_tokens_total\{provider="openai",model_family="gpt-4o",direction="input"\} [1-9]/
body matches /rustybin_llm_faults_total\{provider="openai",kind="rate_limit"\} [1-9]/

Latency percentiles

rustybin_request_duration_seconds has the buckets 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30 and 60 seconds (plus +Inf), so histogram_quantile gives usable values from a fast stub (about 1 ms) up to a slow mock LLM stream or a long injected delay. No two neighbouring bounds are more than 3x apart, so an estimated percentile is within one bucket of the true value.

# p50 and p99 per route over the last 5 minutes
histogram_quantile(0.50, sum by (le, route) (rate(rustybin_request_duration_seconds_bucket[5m])))
histogram_quantile(0.99, sum by (le, route) (rate(rustybin_request_duration_seconds_bucket[5m])))

# p99 over every route
histogram_quantile(0.99, sum by (le) (rate(rustybin_request_duration_seconds_bucket[5m])))

# Plan limit rejections per second, by dimension
sum by (dimension) (rate(rustybin_limit_rejections_total[5m]))

# Share of streaming mock LLM requests
sum(rate(rustybin_llm_requests_total{streaming="true"}[5m]))
  / sum(rate(rustybin_llm_requests_total[5m]))

The histogram measures the time until response headers: for an SSE or streaming LLM response that is the time to the first byte, not the length of the stream.

A Prometheus scrape job:

scrape_configs:
  - job_name: rustybin
    metrics_path: /_rustybin/metrics
    authorization:
      credentials: change-me        # RUSTYBIN_ADMIN_TOKEN
    static_configs:
      - targets: ["rustybin.example.com:80"]

GET /_rustybin/usage also carries the build (version, git_sha, profile), the instance id and the control auth mode, so one call tells a collector which version runs with which plan:

GET {{secured_url}}/_rustybin/usage
Authorization: Bearer {{console_token}}
HTTP 200
[Asserts]
jsonpath "$.instance_id" == "docs-secured"
jsonpath "$.build.version" exists
jsonpath "$.control_auth" == "jwt"

Logs

RUSTYBIN_LOG_FORMAT=json writes one JSON object per line: timestamp, level, target, message and the event fields at the top level, plus a span object with the request's method, path and request_id (the X-Request-Id the response carries), so log lines join with gateway logs on the request id:

{"timestamp":"2026-10-05T12:00:00.000000Z","level":"INFO","message":"finished processing request","latency":"0 ms","status":200,"target":"tower_http::trace::on_response","span":{"method":"GET","path":"/uuid","request_id":"7d0c...","name":"request"}}

Containers

The Docker image runs as an unprivileged user (uid and gid 10001) and works with a read-only root filesystem. Rustybin writes only two things: the demo PKI next to RUSTYBIN_TLS_CERT (default /app/certs) and the optional RUSTYBIN_USAGE_FILE. Put both on a writable volume:

docker run -d --read-only --cap-drop ALL --security-opt no-new-privileges \
  -v rustybin-certs:/app/certs \
  -e RUSTYBIN_USAGE_FILE=/app/certs/usage.json \
  -p 8080:80 -p 8443:443 -p 50051:50051 rustybin

Without a writable /app/certs the demo certificates live in memory and change at every restart (client certificates issued earlier stop verifying). Docker lets an unprivileged process bind ports below 1024 inside the container (net.ipv4.ip_unprivileged_port_start=0). On runtimes that do not (Kubernetes without that sysctl, some micro-VM platforms), move the listeners up: RUSTYBIN_HTTP_PORT=8080 RUSTYBIN_HTTPS_PORT=8443.