# Global AI access. _Local data control._

> An AI gateway (LLM gateway) is the control layer between tools and AI models. Agent Lab's: Claude, GPT and Gemini on one key, with data rules before sending.

Web page: https://agentlab.cresclab.com/en/gateway  
繁體中文 (Markdown): https://agentlab.cresclab.com/gateway.md

Agent Lab AI Gateway · Enterprise AI control plane

Claude, GPT, Gemini and 80-plus models behind one key. Decide which data may reach which model and who may spend how much before a call leaves; every call is on the record.

[Start free](https://app.agentlab.cresclab.com/?lang=en&intent=gateway) [Book a demo](https://agentlab.cresclab.com/en/gateway#demo)

Rather read the docs first? [See how to connect](https://agentlab.cresclab.com/docs/en/gateway/quickstart).

The basics

## What is an AI gateway?

An AI gateway (or LLM gateway) is the control layer between a company and its AI models: every tool calls models through it, and it decides where data may go, which model answers and how much may be spent, and records each call.

**Models**

Claude, GPT, Gemini and 80-plus more, via Azure AI Foundry and Google Vertex AI

**APIs**

OpenAI and Anthropic compatible; change the base URL

**Data**

Personal data and secrets blocked or masked in requests and replies; each key held to set models

**Model choice**

Name one, or let auto pick by task difficulty; the fallback order is yours to set

**Limits**

Budgets and rates per workspace, team, key and end user

**Records**

Who called, which model, what it cost; no prompts stored

**Pricing**

[Models at list price with no markup; prepaid credit, with the fee charged on the top-up](https://agentlab.cresclab.com/en/gateway#pricing)

Four questions IT asks first

## Reach every model. Keep data in bounds. Know the cost. Stay in control.

1. [01 **Access** _Which models can we use?_ 80-plus models, one key, one address.](https://agentlab.cresclab.com/en/gateway#models)
2. [02 **Control** _Will sensitive data leave?_ Blocked, masked or held to set models by rule, before sending.](https://agentlab.cresclab.com/en/gateway#control)
3. [03 **Cost** _Where does the money go?_ Every call priced; auto sends simple work to cheap models.](https://agentlab.cresclab.com/en/gateway#cost)
4. [04 **Govern** _Who may spend what? Who changed a rule?_ Limits hold before sending; rule changes stay on record.](https://agentlab.cresclab.com/en/gateway#govern)

What the records show feeds the next round of rules

01 Access

## One key to 80-plus models

Diagram. The workspace decides which models each team may use; auto currently picks between Claude and GPT.

### Change one URL

Point Claude Code or Codex at the gateway with its key; no code changes.

### No single vendor

80-plus models from many makers, switchable any time, or held to a few.

### Picked by capability

Images, audio or tool calls go only to models that can handle them.

### Retries and fallback

Auto tries models in the order you set. Retries and fallback calls count as one request, and the usage page shows how many requests used a fallback.

02 Control

## Decide which data may reach which model, before it is sent

The gateway enforces the rules, so nobody has to remember them.

Diagram. Guardrails match the text of requests and replies; images and files are not inspected. They are off until an owner or admin turns them on.

### Guardrails

Emails, phone numbers, card numbers, national IDs and API keys blocked or masked in requests and replies; add your own terms.

### Allowlists

Hold a workspace, each key or a directory group to set models; keys also to source IPs.

### Routing rules

By tag, end user, spend or rate, narrow the models and providers a request may use, or refuse it.

### No content stored

Prompts and replies never reach a database, log or trace; only metadata is kept.

Roadmap Next: data classes decide where requests go

- Label each model with the data it may handle (public, internal, restricted) and send each request only to models cleared for its class.
- Hold requests to regions or clouds, set in the console.

03 Cost

## What every call cost, and simple work on cheap models

Illustration: the fields are the usage page's and CSV export's real fields; the values are examples. The Gateway stores no prompt or reply text.

| Task assessed as | `auto:cost` | `auto` | `auto:quality` |
| --- | --- | --- | --- |
| Extraction, rewriting, chat | Cheapest | Cheapest | Standard |
| Analysis, coding, planning | Cheapest | Standard | Stronger |
| Multi-step | Standard | Stronger | Stronger |
| Could not tell | Standard | Standard | Standard |

When unsure it uses the standard model, never quietly a cheaper one. With under 20% of the monthly budget left, work that would get the stronger model gets the standard one. Claude Code and Codex turn it on with one header, `X-AgentLab-Routing: auto`.

### Usage reports

Spend split by key, model, team and provider; export a month as CSV.

### What auto saved

Daily savings against always using the strongest model, net of the assessment's own cost.

### Cheaper-model suggestions

A sample shows whether each key's model fits its work; switch to a cheaper one in one click.

### Quality sampling

2% of requests answered by a cheaper model are rerun on the standard one and compared.

Real usage

## We run on it every day

Our own workspace, last 30 days as of 2026-10-29, numbers from the console's usage page.

**requests**

31.3K

99.8% routed by auto

**tokens**

5.5B

92.8% of input from the prompt cache

**success rate**

97.2%

887 failed in 30 days

**P95 latency**

22 s

P50 5.8 s, end to end, model time included

Illustration redrawn from the console's usage page. The figures and shares are real; the daily columns only show the layout, not each day's data.

Used for a public benchmark

### Seven models tested through one key

- CAAC Go's November 2026 agent benchmark ran Claude Opus 5.5, Claude Sonnet 5.5 and GPT-6.1 Sol through Agent Lab AI Gateway, in the same table as the models it already had on Azure OpenAI and Vertex AI.
- No new vendor accounts: one key served the Anthropic Messages API and the OpenAI Responses API, and every call came back with the token counts the cost table is built from.
- [Read the benchmark on caacgo.cresclab.com](https://caacgo.cresclab.com/benchmark/november-2026)

04 Govern

## Who may spend what, and who changed which rule, on record

Workspace

- Monthly spend
- Tokens and requests
- Rate limits
- Allowed models
- Guardrails

Team

- Monthly ceiling
- 80% and 100% alerts

Key

- Model allowlist
- Source IPs
- Expiry and rotation
- RPM / TPM
- Concurrency

End user

- Monthly cap

Each level holds the most a request could cost before it is sent; if it does not fit, it is not sent and nothing is charged. Where several limits apply, the tightest wins.

Illustration. The policy register lists each kind of setting with its current value and every change, before and after.

### Hold, then send

Cost is held up front, so concurrent calls cannot overspend together.

### Key controls

Expiry, allowed models, source IPs, rotation without downtime; stored as hashes.

### Directory groups

Groups from your enterprise SSO (OIDC) carry allowed models and a monthly budget.

### Alerts and spikes

Alerts at 80% and 100%, and when spend jumps to three times a usual day.

Roadmap Next: policy that follows agents

- Allowed models, budgets and routing rules per agent.

Multi-region companies

## Offices in several regions, one set of rules

When a company has offices in several regions, IT and compliance ask first: where is data kept, who may use which models, and can we trace what happened.

### Where data is kept

Settings, usage and the audit trail are kept by Agent Lab; models run in each provider's cloud.

### Sensitive data stopped before it leaves

Guardrails check both what is sent and what the model replies, and can mask or refuse; off by default, turned on by IT.

### Each office, its own access

Set the models and budget each office may use, per key or per enterprise SSO group.

### More than one model family

The same key reaches open models such as DeepSeek, Llama, Mistral and gpt-oss, and you can self-host a model in your own project.

Planned Next: pin where each class of data may run

- Let IT choose which models and regions a class of data may go to. Today the Agent Lab team sets this up for you.

Each model provider has its own supported regions and usage policies; book a demo and we will confirm with you which models apply where your company operates.

Plans and pricing

## Models at list price, with no markup

Turning it on costs nothing and needs no credit card. Add credit before calling a model; the fee is charged once, on the top-up.

|  | How you pay | Included | Model spend | With no balance |
| --- | --- | --- | --- | --- |
| Free | Nothing, and no credit card | No model credit | Add credit or redeem a code before calling a model | Requests are refused; after a top-up the same key works at once |
| Pay as you go | Prepaid credit, plus a 5% fee on each top-up | No request limit | List price, no markup, drawn from the balance | A request the balance cannot cover is refused; the balance never goes negative |
| Enterprise | By contract; talk to us | By contract | List price, no markup | Refused when the balance runs out |

**Paying in US dollars**

US$5 to US$25,000 per top-up, 5% fee, at least US$0.80. Example: US$100 of credit costs US$105

**Paying in NT$**

NT$300 to NT$800,000 per top-up, 5% fee, plus 5% business tax, with a uniform invoice (統一發票). Example: NT$1,000 of credit costs NT$1,103

**Expiry**

Credit does not expire

**Low balance and auto top-up**

A notice when it falls below the amount you set; workspaces that pay in NT$ can turn on automatic top-up

**After the first top-up or code**

The workspace is pay as you go; at a zero balance requests are refused, and it does not return to Free

**Codes**

Redeemed in the console; the credit joins the same balance as a top-up

Compare

## How it compares with other AI gateways

|  | Hosting | Budget enforcement | Auto model choice | Best for |
| --- | --- | --- | --- | --- |
| Agent Lab AI Gateway | Run by us, or self-hosted per project | Held before sending | Auto by difficulty | Companies that do not want to run it |
| LiteLLM | Open source, self-hosted | Reserved before sending | Auto Router (beta) | Engineering teams that self-host |
| Portkey (now Prisma AIRS AI Gateway) | SaaS, hybrid, self-hosted | Per-key limits | Rule-based | Request logs and observability first |
| Cloudflare AI Gateway | Run by Cloudflare; core free | Charged after completion | Auto Router | Teams already on Cloudflare |
| Kong / Azure APIM | Extends existing API management | Kong: next request; Azure: tokens only | Load balancing | Teams already on them |
| TrueFoundry | SaaS; enterprise self-host | Per user or model | Weight, latency | Teams that also host models |

From each vendor's docs, checked 2026-10-21: [LiteLLM](https://docs.litellm.ai/docs/proxy/users), [Portkey](https://portkey.ai/docs/product/ai-gateway/virtual-keys/budget-limits), [Cloudflare](https://developers.cloudflare.com/ai-gateway/features/spend-limits/), [Kong](https://developer.konghq.com/plugins/ai-rate-limiting-advanced/), [Azure](https://learn.microsoft.com/en-us/azure/api-management/llm-token-limit-policy), [TrueFoundry](https://www.truefoundry.com/docs/ai-gateway/budgetlimiting).

FAQ

## AI gateway questions

**What is an AI gateway?**

An AI gateway (LLM gateway) is the control layer between a company and its AI models. Every tool calls models through it, and it decides where data may go, which model answers and how much may be spent, and records each call.

**How is AI Gateway priced?**

Model spend is charged at list price with no markup. Turning it on costs nothing and needs no credit card, but the Free plan includes no model credit: add credit or redeem a code before calling a model. Each top-up carries a 5% fee (at least US$0.80; US$5 to US$25,000 per top-up). Credit does not expire, and a request the balance cannot cover is refused, so the balance never goes negative.

**How is an AI gateway different from an API proxy?**

An API proxy only forwards requests. An AI gateway applies data rules, budgets and limits before sending, picks the model for the task, and records what every call cost.

**Does AI Gateway also govern tool calls and MCP?**

AI Gateway governs model calls. When an assistant calls a tool or an MCP connector, Agent Lab's Access & Connect governs it: the call runs with the asker's own permissions, waits for approval when one is needed, and is recorded. Both share the same identity, budgets and audit trail. [See how permissions, approvals and budgets work](https://agentlab.cresclab.com/en/platform#governance)

**What are the security risks of an AI gateway, and how do you reduce them?**

Every prompt passes through the gateway, so ask whether it stores content, how keys are kept and whether sensitive data can be stopped before sending. Agent Lab stores no prompts or replies; keys are kept as hashes and can be held to models, source IPs and an expiry; guardrails block or mask personal data in requests and replies. Requests go to Azure AI Foundry or Google Vertex AI, under their data terms.

**Can we control which data goes to which model?**

Yes, in three layers: guardrails block or mask emails, phone numbers, card numbers, national IDs and API keys; allowlists hold a workspace, a key or a directory group to set models; routing rules narrow the models a request may use by tag or user, or refuse it. Guardrails match text; images and files are not inspected.

**Which models does it support?**

Claude, GPT, Gemini and 80-plus models including DeepSeek, Llama and Mistral, served through Azure AI Foundry and Google Vertex AI; chat, embeddings, images and speech use the same key.

**Does the gateway store my prompts?**

No. Prompts and replies pass through memory only and never reach a database, log or trace; what is kept is metadata such as key, team, model, tokens, cost and duration.

**Can it connect on-prem models?**

The hosted gateway serves cloud models. When data cannot leave your network, the gateway can be installed in your environment per project and connected to on-prem models with an OpenAI-compatible API, such as vLLM or Ollama.

**How do I control Claude Code spend?**

Point ANTHROPIC_BASE_URL at the gateway and set ANTHROPIC_AUTH_TOKEN to a gateway key. Cost is held before each call and a call past the workspace, team or key limit is not sent; add X-AgentLab-Routing: auto and simple steps go to cheaper models.

**Can offices in different countries share one AI gateway?**

Yes. Offices share one workspace, and keys or enterprise SSO groups set the models and budget each may use. Which models are available where follows each provider's policy, which we confirm with you before turning it on.

**How do I choose an AI gateway?**

Ask five things: does it store prompts, can it control where data goes, does it stop spend before sending or count it after, can it pick models by task, and do you run it or does someone run it for you. Agent Lab stores no prompts, holds cost before sending and is run for you; if you want open source you run yourself, LiteLLM fits better.

**How do I monitor and trace LLM activity through the AI gateway?**

The usage page records every request: time, key, team, end user, tag, model, the reason it was chosen, tokens, cost and duration, and splits the same records by key, model, team, end user, tag and prompt. Export it as CSV, send traces to your own OpenTelemetry collector, and read the append-only audit log of who changed which rule. Prompt and reply text is never stored.

**How do I run a proof of concept for an AI gateway?**

Start with one tool and one team: turn it on, add a small amount of credit, create a key for that team with a limit, and point Claude Code or Codex at the gateway. After a week or two, read the usage page: where spend went, what auto saved and how quality sampling came out.

## Start free, or book a demo

[Start free](https://app.agentlab.cresclab.com/?lang=en&intent=gateway)

Self-hosting, a contract, or questions first? Book a demo:

- Which models and tools you use
- Rough monthly usage
- Whether you need self-hosting or on-prem models

Already have a workspace? Open the [AI Gateway page](https://app.agentlab.cresclab.com/) in the console. The [docs](https://agentlab.cresclab.com/docs/en/gateway) cover keys, limits and errors.
