​

LLM Gateway Architecture

​

Decision

• Accept explicit model IDs exactly as before, or omi:auto:* lane IDs.
• Resolve an auto lane to a versioned route artifact.
• Validate request capabilities before execution.
• Execute the primary provider/model and compatible fallbacks.
• Return a normal provider-compatible response while recording bounded route/fallback terminal telemetry.
• Write a durable, cache-aware usage ledger for each gateway provider attempt without retaining prompts or response bodies.

​

Chat-Structured Lane

omi:auto:chat-structured

​

Deployment

omi:auto:chat-fast-memory
omi:auto:chat-smart-tools

​

Why A Separate Service

• backend and pusher both run LLM workloads and should share one runtime policy;
• route artifacts need deploy-time validation and runtime fallback independent of product code;
• route observability needs one consistent set of labels;
• future surfaces can call the same OpenAI-compatible gateway without learning provider details;
• rollback must be config-only and not require desktop or mobile releases.

​

Language And Framework

• The backend is already Python 3.11 and FastAPI.
• Existing auth, logging, sanitizer, executor, test, and deployment patterns are Python.
• Existing LLM provider knowledge lives in Python modules.
• Pydantic is already the natural schema/validation tool for FastAPI.
• httpx.AsyncClient matches backend async I/O rules and avoids blocking the event loop.

backend/scripts/dev-serve-llm-gateway.sh
backend/scripts/validate-llm-gateway-env.py
backend/scripts/smoke-llm-gateway.py
backend/llm_gateway/
  main.py
  config/
    lanes.yaml
    route_artifacts.yaml
    feature_bundles.yaml
  routers/
    health.py
    openai_compatible.py
    dependencies.py
  gateway/
    schemas.py
    config_loader.py
    auth.py
    credentials.py
    errors.py
    validator.py
    resolver.py
    executor.py
    metrics.py

​

Deployment Shape

1. Start with no live product traffic while service startup, readiness, config validation, and service auth are verified.
2. Route non-embedding features through the gateway in shadow mode (OMI_LLM_GATEWAY_DEV_SHADOW_ALL_* and feature shadow flags).
3. Promote dev to gateway-serving with OMI_LLM_GATEWAY_FEATURE_MODE=gateway. When serving is on, turn shadow flags off so callers do not double-hit the gateway. Keep hard direct surfaces (BYOK, Anthropic agentic chat, Assistants/file chat, omni realtime) on providers via OMI_LLM_GATEWAY_ALLOW_DIRECT_MODEL_EXCEPTION=true. Backend wraps gateway clients with legacy fallback on hard transport failures (timeout / 5xx / connection).
4. Run prod shadow before prod serving when the feature touches user content or user-visible behavior. Prod serving also requires OMI_LLM_GATEWAY_ALLOW_PROD_FEATURE_MODE=true and an explicit gateway URL.
5. Promote prod serving only after rollback has been tested and the route artifact has suitable eval evidence.

cd backend && ./scripts/dev-serve-llm-gateway.sh

​

Deployment Configuration

<env>-omi-llm-gateway

dev:  http://dev-omi-llm-gateway.dev-omi-backend.svc.cluster.local:8080
prod: http://prod-omi-llm-gateway.prod-omi-backend.svc.cluster.local:8080

• gateway chart: backend/charts/llm-gateway/
• manual workflow: .github/workflows/gcp_llm_gateway.yml
• backend caller env: backend/charts/backend-listen/{dev,prod}_omi_backend_listen_values.yaml
• shared secret mapping: backend/charts/backend-secrets/{dev,prod}_omi_backend_secrets_values.yaml

dev:  based-hardware-dev / OMI_LLM_GATEWAY_SERVICE_TOKEN
prod: based-hardware     / OMI_LLM_GATEWAY_SERVICE_TOKEN

python backend/scripts/validate-llm-gateway-env.py \
  backend/charts/backend-listen/dev_omi_backend_listen_values.yaml \
  backend/charts/llm-gateway/dev_omi_llm_gateway_values.yaml

python backend/scripts/validate-backend-runtime-env.py --env dev

python backend/scripts/validate-backend-runtime-env.py --env dev --check-live-cloud-run

python backend/scripts/smoke-llm-gateway.py \
  --url http://dev-omi-llm-gateway.dev-omi-backend.svc.cluster.local:8080 \
  --token "$OMI_LLM_GATEWAY_SERVICE_TOKEN"

1. Create or verify OMI_LLM_GATEWAY_SERVICE_TOKEN in based-hardware-dev Secret Manager and set the LLM_GATEWAY_GSA environment variable after its Workload Identity / Vertex IAM binding exists.
2. Run the values validation command above.
3. Run focused gateway unit tests and preflight checks.
4. Manually dispatch gcp_llm_gateway.yml to development.
5. Run /ready and chat-completions smoke checks.
6. Deploy backend-listen values only after gateway smoke passes, so backend callers do not point at a missing service.

1. Create or verify an independent OMI_LLM_GATEWAY_SERVICE_TOKEN in based-hardware Secret Manager and the prod LLM_GATEWAY_GSA Workload Identity / Vertex IAM binding.
2. Confirm DEV smoke passed with the same image/commit.
3. Run the prod values validation command.
4. Manually dispatch gcp_llm_gateway.yml to prod.
5. Run prod /ready and chat-completions smoke checks from the prod GKE network.
6. Deploy backend-listen and the Cloud Run callers after gateway prod smoke passes; keep OMI_LLM_GATEWAY_ALLOW_DIRECT_MODEL_EXCEPTION=true so acknowledged direct surfaces retain their existing provider routes.

​

Major Library Choices

• FastAPI for HTTP endpoints and service lifecycle.
• Pydantic for lane, artifact, request, and validation schemas.
• httpx.AsyncClient for provider HTTP calls and gateway-to-provider calls.
• OpenAI Python SDK only where it materially reduces request/stream parsing risk. Prefer direct httpx for the gateway core so we preserve request/response metadata, timeouts, headers, and streaming behavior consistently.
• Prometheus-compatible metrics following existing backend observability conventions.
• pytest for deterministic resolver, validator, and executor tests.

• LangChain inside the gateway execution path. Existing product code can keep using LangChain, but the gateway should speak provider HTTP contracts directly so it can preserve OpenAI-compatible request/response shapes and streaming semantics.
• Firestore/Redis as routing config stores. All lane and route artifacts live in repo files for v1.
• LiteLLM, Portkey, Envoy AI Gateway, or Kong as the gateway foundation.
• Runtime benchmark fetching, including Artificial Analysis calls, inside request handling.

​

Open Source Position

• LiteLLM
• LiteLLM proxy docs
• Portkey Gateway
• Envoy AI Gateway supported endpoints

​

API Surface

POST /v1/chat/completions

{
  "model": "gpt-4.1-mini",
  "messages": []
}

{
  "model": "omi:auto:chat-structured",
  "messages": [],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "RequiresContext",
      "strict": true,
      "schema": {}
    }
  },
  "metadata": {
    "omi_feature": "chat_extraction.requires_context",
    "prompt_version": "chat_extraction.requires_context.v1",
    "parser_version": "RequiresContext.v1"
  }
}

{
  "model": "omi:auto:chat-structured",
  "choices": []
}

X-Omi-Lane-Id: omi:auto:chat-structured
X-Omi-Route-Id: route.chat_structured.2026_06_27.001
X-Omi-Fallback-Used: false

​

Config Model

backend/llm_gateway/config/lanes.yaml
backend/llm_gateway/config/route_artifacts.yaml
backend/llm_gateway/config/feature_bundles.yaml

lane_id: omi:auto:chat-structured
surface: openai.chat_completions
capabilities:
  text_input: true
  streaming: false
  structured_output: json_schema
  tools: false
objective:
  quality: 0.60
  latency: 0.20
  cost: 0.20
active_route: route.chat_structured.2026_06_27.001
last_known_good: route.chat_structured.2026_06_20.001

route_artifact_id: route.chat_structured.2026_06_27.001
lane_id: omi:auto:chat-structured
primary:
  provider: openai
  model: gpt-4.1-mini
fallbacks:
  - provider: openai
    model: gpt-4.1-mini
timeouts:
  request_ms: 8000
retry:
  max_attempts: 1
capabilities:
  structured_output: json_schema
  streaming: false
evidence:
  benchmark_snapshot: bench.omi.chat_structured.2026_06_27
  eval_report: eval.chat_extraction_requires_context.2026_06_27
rollout:
  stage: shadow
  percent: 0
credential_policy:
  mode: omi_paid
  allow_byok_to_omi_paid_fallback: false
fallback_policy:
  fallback_on:
    - timeout_before_output
    - provider_429_omi_paid
    - provider_5xx_omi_paid
  never_fallback_on:
    - byok_auth
    - byok_quota
    - byok_rate_limit
    - missing_byok_key
    - capability_mismatch
    - invalid_config
artifact_digest: sha256:<computed-by-validation>

feature: chat_extraction.requires_context
lane_id: omi:auto:chat-structured
prompt_version: chat_extraction.requires_context.v1
parser_version: RequiresContext.v1
eval_suite: chat_extraction_requires_context.v1
promotion_gates:
  schema_valid_rate: ">= 99.5%"
  classification_agreement_delta: ">= 0"
  false_negative_regression: "none"
  p95_latency_ms: "<= LKG + 10%"
  cost_per_success: "<= LKG + 15%"

​

Integration With Existing Backend Code

@dataclass(frozen=True)
class ExplicitRouteRef:
    feature: str
    model: str
    provider: str
    options: dict[str, object]

@dataclass(frozen=True)
class AutoLaneRouteRef:
    feature: str
    lane_id: str

• get_model(feature)
• get_provider(feature)
• get_llm(feature)
• get_route_options(feature, model, provider)

get_route_ref(feature) -> ExplicitRouteRef | AutoLaneRouteRef
is_auto_lane_id(model: str) -> bool

​

BYOK Policy

• backend forwards a short-lived BYOK credential envelope to the gateway over service-authenticated transport;
• gateway receives a key reference and resolves it through an approved internal secret path.

​

Service Auth

​

Request Validation And Route Resolution

• is_auto_lane_id(model) recognizes only the omi:auto: namespace.
• omi:auto:chat-structured is the only supported auto lane.
• unknown omi:auto:* lanes return a typed model-not-found error.
• bare provider model names such as gpt-4o-mini are not direct gateway routes in v1 and return a typed unsupported-model error.
• the resolver maps the supported lane to its configured active_route and last_known_good route artifact.
• runtime route checks defensively reject active/LKG lane, surface, capability, and credential-mode mismatches as invalid config.

• model: "omi:auto:chat-structured";
• non-empty text messages;
• stream absent or false;
• no tool use;
• response_format.type: "json_schema";
• a response_format.json_schema.schema object.

​

HTTP Surface

• requires service auth;
• accepts the same request shape validated by the resolver;
• builds an Omi-managed request credential context for the current v1 lane;
• resolves model: "omi:auto:chat-structured" to the checked-in active route;
• calls the executor and returns the provider response payload with model rewritten to the requested lane ID;
• forwards supported OpenAI chat-completions controls such as temperature, max_tokens, max_completion_tokens, seed, top_p, penalties, stop, and user;
• maps typed gateway exceptions to OpenAI-style error envelopes with error.message, error.type, error.param, and error.code;
• rejects unknown auto lanes, bare provider model names, streaming, tools, unsupported capabilities, and unknown top-level request parameters before provider execution.

​

Provider Execution

• the caller passes a resolved route plus a request-level CredentialContext;
• the executor sends an OpenAI-compatible chat-completions payload to the active route primary provider first;
• the provider-facing payload replaces the lane model with the selected provider model and forces stream: false;
• the caller-facing response payload keeps the normal provider response shape but reports model as the requested lane ID, such as omi:auto:chat-structured;
• selected route artifact, provider, provider model, fallback reason, and LKG usage are returned only on the executor result metadata, not embedded into the response payload.

• active route fallbacks are attempted only when the route policy allows the normalized failure class;
• LKG is attempted only through select_lkg_route_for_failure, so active route policy remains the single gate for runtime LKG;
• BYOK missing key, auth, quota, rate limit, unsupported provider, capability mismatch, and invalid config failures fail visibly and do not fall back by default;
• Omi-paid timeout before output, provider 429, and provider 5xx may fall back only when the route artifact allows those classes;
• streaming-after-first-token recovery is unsupported for MVP. The executor assumes no partial output exists and does not try to recover or replay partial responses.

• OpenAICompatibleChatCompletionProvider for live non-streaming OpenAI-compatible calls;
• in-memory fake providers for unit tests.

​

Observability

llm_gateway_requests_total
llm_gateway_request_latency_seconds

​

Shadow Safety

• feature-owner/privacy approval for the feature being shadowed;
• sampling limits and a kill switch;
• cost cap;
• no BYOK shadow by default;
• no provider expansion beyond the current production provider class without approval;
• no persistence of raw prompts or raw responses;
• metrics-only comparison unless an approved eval store exists.

​

Explicit Non-Goals

• No desktop Settings UI.
• No quality/latency/cost sliders.
• No per-user routing preferences.
• No Firestore or Redis routing prefs.
• No desktop-side model/route cache.
• No public /v1/auto-router/pick.
• No new public /v1/auto/model-pick.
• No request-path benchmark fetching.
• No production route from mock benchmark data.
• No benchmark-only promotion.
• No wholesale LiteLLM, Portkey, Envoy AI Gateway, or Kong adoption.

​

Maintainer Checklist

• explicit model routing remains backward compatible;
• model_config.py still owns feature-to-route mapping;
• gateway config validation fails closed on invalid prod config;
• active route has valid LKG;
• route artifacts are immutable;
• BYOK failure does not silently fall back to Omi-paid traffic;
• /v1/chat/completions requires internal service auth;
• LKG/fallback is limited to artifact-approved failure classes;
• mock benchmarks cannot load in prod;
• Omi eval report exists;
• shadow/canary completed;
• rollback is config-only;
• observability includes route, fallback, latency, errors, and cost.