Crescendo Lab

Agent Lab AI Gateway · Enterprise AI control plane

Global AI access.
Local data control.

Claude, GPT, Gemini and 80-plus models behind one key. Decide which data may reach which model and who may spend how much before a call leaves; every call is on the record.

Rather read the docs first? See how to connect.

The basics

What is an AI gateway?

An AI gateway (or LLM gateway) is the control layer between a company and its AI models: every tool calls models through it, and it decides where data may go, which model answers and how much may be spent, and records each call.

Models
Claude, GPT, Gemini and 80-plus more, via Azure AI Foundry and Google Vertex AI
APIs
OpenAI and Anthropic compatible; change the base URL
Data
Personal data and secrets blocked or masked in requests and replies; each key held to set models
Model choice
Name one, or let auto pick by task difficulty
Limits
Budgets and rates per workspace, team, key and end user
Records
Who called, which model, what it cost; no prompts stored

Four questions IT asks first

Reach every model. Keep data in bounds. Know the cost. Stay in control.

01 Access

One key to 80-plus models

Diagram. The workspace decides which models each team may use; auto currently picks between Claude and GPT.

Change one URL

Point Claude Code or Codex at the gateway with its key; no code changes.

No single vendor

80-plus models from many makers, switchable any time, or held to a few.

Picked by capability

Images, audio or tool calls go only to models that can handle them.

02 Control

Decide which data may reach which model, before it is sent

The gateway enforces the rules, so nobody has to remember them.

Diagram. Guardrails match the text of requests and replies; images and files are not inspected. They are off until an owner or admin turns them on.

Guardrails

Emails, phone numbers, card numbers, national IDs and API keys blocked or masked in requests and replies; add your own terms.

Allowlists

Hold a workspace, each key or a directory group to set models; keys also to source IPs.

Routing rules

By tag, end user, spend or rate, narrow the models and providers a request may use, or refuse it.

No content stored

Prompts and replies never reach a database, log or trace; only metadata is kept.

03 Cost

What every call cost, and simple work on cheap models

Illustration: the fields are the usage page's and CSV export's real fields; the values are examples. The Gateway stores no prompt or reply text.
Task assessed asauto:costautoauto:quality
Extraction, rewriting, chatCheapestCheapestStandard
Analysis, coding, planningCheapestStandardStronger
Multi-stepStandardStrongerStronger
Could not tellStandardStandardStandard
When unsure it uses the standard model, never quietly a cheaper one. With under 20% of the monthly budget left, work that would get the stronger model gets the standard one. Claude Code and Codex turn it on with one header, X-AgentLab-Routing: auto.

Usage reports

Spend split by key, model, team and provider; export a month as CSV.

What auto saved

Daily savings against always using the strongest model, net of the assessment's own cost.

Cheaper-model suggestions

A sample shows whether each key's model fits its work; switch to a cheaper one in one click.

Quality sampling

2% of requests answered by a cheaper model are rerun on the standard one and compared.

Real usage

We run on it every day

Our own workspace, last 30 days as of 2026-10-29, numbers from the console's usage page.

requests
31.3K
99.8% routed by auto
tokens
5.5B
92.8% of input from the prompt cache
success rate
97.2%
887 failed in 30 days
P95 latency
22 s
P50 5.8 s, end to end, model time included
Illustration redrawn from the console's usage page. The figures and shares are real; the daily columns only show the layout, not each day's data.
Used for a public benchmark

Seven models tested through one key

  • CAAC Go's November 2026 agent benchmark ran Claude Opus 5.5, Claude Sonnet 5.5 and GPT-6.1 Sol through Agent Lab AI Gateway, in the same table as the models it already had on Azure OpenAI and Vertex AI.
  • No new vendor accounts: one key served the Anthropic Messages API and the OpenAI Responses API, and every call came back with the token counts the cost table is built from.
  • Read the benchmark on caacgo.cresclab.com

04 Govern

Who may spend what, and who changed which rule, on record

Workspace

  • Monthly spend
  • Tokens and requests
  • Rate limits
  • Allowed models
  • Guardrails

Team

  • Monthly ceiling
  • 80% and 100% alerts

Key

  • Model allowlist
  • Source IPs
  • Expiry and rotation
  • RPM / TPM
  • Concurrency

End user

  • Monthly cap
Each level holds the most a request could cost before it is sent; if it does not fit, it is not sent and nothing is charged. Where several limits apply, the tightest wins.
Illustration. The policy register lists each kind of setting with its current value and every change, before and after.

Hold, then send

Cost is held up front, so concurrent calls cannot overspend together.

Key controls

Expiry, allowed models, source IPs, rotation without downtime; stored as hashes.

Directory groups

Groups from your enterprise SSO (OIDC) carry allowed models and a monthly budget.

Alerts and spikes

Alerts at 80% and 100%, and when spend jumps to three times a usual day.

Multi-region companies

Offices in several regions, one set of rules

When a company has offices in Taiwan, Hong Kong or Singapore, IT and compliance ask first: where is data kept, who may use which models, and can we trace what happened.

Where data is kept

Settings, usage and the audit trail stay in Google Cloud Taiwan; models run in each provider's cloud.

Sensitive data stopped before it leaves

Guardrails check both what is sent and what the model replies, and can mask or refuse; off by default, turned on by IT.

Each office, its own access

Set the models and budget each office may use, per key or per enterprise SSO group.

More than one model family

The same key reaches open models such as DeepSeek, Llama, Mistral and gpt-oss, and you can self-host a model in your own project.

Each model provider has its own supported regions and usage policies; book a demo and we will confirm with you which models apply where your company operates.

Compare

How it compares with other AI gateways

HostingBudget enforcementAuto model choiceBest for
Agent Lab AI GatewayRun by us, or self-hosted per projectHeld before sendingAuto by difficultyCompanies that do not want to run it
LiteLLMOpen source, self-hostedReserved before sendingAuto Router (beta)Engineering teams that self-host
Portkey (now Prisma AIRS AI Gateway)SaaS, hybrid, self-hostedPer-key limitsRule-basedRequest logs and observability first
Cloudflare AI GatewayRun by Cloudflare; core freeCharged after completionAuto RouterTeams already on Cloudflare
Kong / Azure APIMExtends existing API managementKong: next request; Azure: tokens onlyLoad balancingTeams already on them
TrueFoundrySaaS; enterprise self-hostPer user or modelWeight, latencyTeams that also host models

From each vendor's docs, checked 2026-10-21: LiteLLM, Portkey, Cloudflare, Kong, Azure, TrueFoundry.

FAQ

AI gateway questions

What is an AI gateway?
An AI gateway (LLM gateway) is the control layer between a company and its AI models. Every tool calls models through it, and it decides where data may go, which model answers and how much may be spent, and records each call.
How is an AI gateway different from an API proxy?
An API proxy only forwards requests. An AI gateway applies data rules, budgets and limits before sending, picks the model for the task, and records what every call cost.
What are the security risks of an AI gateway, and how do you reduce them?
Every prompt passes through the gateway, so ask whether it stores content, how keys are kept and whether sensitive data can be stopped before sending. Agent Lab stores no prompts or replies; keys are kept as hashes and can be held to models, source IPs and an expiry; guardrails block or mask personal data in requests and replies. Requests go to Azure AI Foundry or Google Vertex AI, under their data terms.
Can we control which data goes to which model?
Yes, in three layers: guardrails block or mask emails, phone numbers, card numbers, national IDs and API keys; allowlists hold a workspace, a key or a directory group to set models; routing rules narrow the models a request may use by tag or user, or refuse it. Guardrails match text; images and files are not inspected.
Which models does it support?
Claude, GPT, Gemini and 80-plus models including DeepSeek, Llama and Mistral, served through Azure AI Foundry and Google Vertex AI; chat, embeddings, images and speech use the same key.
Does the gateway store my prompts?
No. Prompts and replies pass through memory only and never reach a database, log or trace; what is kept is metadata such as key, team, model, tokens, cost and duration.
Can it connect on-prem models?
The hosted gateway serves cloud models. When data cannot leave your network, the gateway can be installed in your environment per project and connected to on-prem models with an OpenAI-compatible API, such as vLLM or Ollama.
How do I control Claude Code spend?
Point ANTHROPIC_BASE_URL at the gateway and set ANTHROPIC_AUTH_TOKEN to a gateway key. Cost is held before each call and a call past the workspace, team or key limit is not sent; add X-AgentLab-Routing: auto and simple steps go to cheaper models.
Can offices in different countries share one AI gateway?
Yes. Offices share one workspace, and keys or enterprise SSO groups set the models and budget each may use. Settings and the audit trail are stored in Google Cloud Taiwan; which models are available where follows each provider's policy, which we confirm with you before turning it on.
How do I choose an AI gateway?
Ask five things: does it store prompts, can it control where data goes, does it stop spend before sending or count it after, can it pick models by task, and do you run it or does someone run it for you. Agent Lab stores no prompts, holds cost before sending and is run for you; if you want open source you run yourself, LiteLLM fits better.
How do I monitor and trace LLM activity through the AI gateway?
The usage page records every request: time, key, team, end user, tag, model, the reason it was chosen, tokens, cost and duration, and splits the same records by key, model, team, end user, tag and prompt. Export it as CSV, send traces to your own OpenTelemetry collector, and read the append-only audit log of who changed which rule. Prompt and reply text is never stored.
How do I run a proof of concept for an AI gateway?
Start with one tool and one team: turn it on free, create a key for that team with a limit, and point Claude Code or Codex at the gateway. After a week or two, read the usage page: where spend went, what auto saved and how quality sampling came out.

Start free,
or book a demo

  • Which models and tools you use
  • Rough monthly usage
  • Whether you need self-hosting or on-prem models

Already have a workspace? Open the AI Gateway page in the console. The docs cover keys, limits and errors.

Book a demo

Optional, helps us schedule faster