Mock LLM
Rustybin's mock LLM is a deterministic, multi-provider model server. It speaks the
native wire format of seven providers, so an AI gateway can be pointed at it
exactly as it would be pointed at the real API, and the official client SDKs
accept its responses (the conformance/ai/ scripts drive it with the OpenAI,
Anthropic and Google Gen AI Python SDKs in CI).
Nothing is generated by a real model: replies come from keyword tables, scripted rules, the prompt itself or a seeded generator, so the same request gives the same answer and demos are reproducible.
What you can demo with it
| Gateway capability | What Rustybin provides |
|---|---|
| AI proxy, provider abstraction, model routing | Seven providers' APIs at stable paths; any model name accepted; X-Rustybin-Provider and X-Rustybin-Model response headers |
| Token based rate limiting, cost accounting | Deterministic token counts in native usage fields and rate-limit headers |
| Prompt decorators and templates | Echo mode returns the exact prompt the upstream received |
| Prompt guards, response sanitisers | Scripted mode returns fake PII, secrets, unsafe text, URLs on demand; plus mock guardrail services |
| Semantic caching | Similarity-preserving embeddings |
| Retries, fallback, load balancing | Native errors (429 with rate-limit headers, Anthropic 529, timeouts, context length) on demand or at a rate; X-Rustybin-Instance |
| Streaming | Native SSE, NDJSON and AWS event stream framing with configurable pace and time to first token |
| Credential injection, key vaults | Native credential checks that report the redacted key they received |
| Tool calling, structured output, agent flows | Tool calls with arguments generated from your JSON schema; structured output valid against your schema |
| Observability | Request inspection: what the upstream received, normalised |
Pages
- Providers and paths: base URLs for SDKs and gateways, endpoints per provider.
- Behaviour: modes, scripted rules, tool calls, structured output, tokens and limits.
- Faults, latency, credentials, inspection: the request and response headers that make the mock useful behind a gateway.
- Embeddings and rerank.
- Guardrails and content safety.
- MCP server and A2A agents for agentic traffic.