​

LLM Gateway Architecture

Status: v1 foundation implemented; first backend pilot is code-on by default with application-level fallback. Implemented so far: separate FastAPI service skeleton, unauthenticated /health, service-authenticated /ready, service-authenticated /v1/chat/completions, repo-local config schemas/loader, checked-in omi:auto:chat-structured lane artifacts, internal service-auth helpers, request-level credential context helpers, typed gateway errors, OpenAI chat-completions request validation, route resolution for the v1 auto lane, the non-streaming provider executor, fake providers for deterministic tests, a live OpenAI-compatible provider adapter, local service URL configuration for backend callers, and code-on structured-output caller pilots for chat_extraction.requires_context, conversation_discard.should_discard, and conversation_action_items.extract.shadow with application-level fallback.

​

Decision

• Accept explicit model IDs exactly as before, or omi:auto:* lane IDs.
• Resolve an auto lane to a versioned route artifact.
• Validate request capabilities before execution.
• Execute the primary provider/model and compatible fallbacks.
• Return a normal provider-compatible response while recording route/fallback/cost metrics.

​

First Pilot

omi:auto:chat-structured

omi:auto:chat-fast-memory
omi:auto:chat-smart-tools

​

Why A Separate Service

• backend and pusher both run LLM workloads and should share one runtime policy;
• route artifacts need deploy-time validation and runtime fallback independent of product code;
• route observability needs one consistent set of labels;
• future surfaces can call the same OpenAI-compatible gateway without learning provider details;
• rollback must be config-only and not require desktop or mobile releases.

​

Language And Framework

• The backend is already Python 3.11 and FastAPI.
• Existing auth, logging, sanitizer, executor, test, and deployment patterns are Python.
• Existing LLM provider knowledge lives in Python modules.
• Pydantic is already the natural schema/validation tool for FastAPI.
• httpx.AsyncClient matches backend async I/O rules and avoids blocking the event loop.

backend/scripts/dev-serve-llm-gateway.sh
backend/scripts/validate-llm-gateway-env.py
backend/scripts/smoke-llm-gateway.py
backend/llm_gateway/
  main.py
  config/
    lanes.yaml
    route_artifacts.yaml
    feature_bundles.yaml
  routers/
    health.py
    openai_compatible.py
    dependencies.py
  gateway/
    schemas.py
    config_loader.py
    auth.py
    credentials.py
    errors.py
    validator.py
    resolver.py
    executor.py
    metrics.py

​

Deployment Shape

1. Add the app entrypoint and unit tests without any production traffic.
2. Add a local run command and Docker/Helm wiring for a separately named service.
3. Add /ready startup/readiness validation that fails closed unless active and LKG route config is valid.
4. Add service-to-service auth from backend and pusher to the gateway.
5. Route one structured feature through the gateway in shadow mode.
6. Promote only after the route artifact has an Omi eval report and rollback has been tested.

cd backend && ./scripts/dev-serve-llm-gateway.sh

​

Deployment Configuration

<env>-omi-llm-gateway

dev:  http://dev-omi-llm-gateway.dev-omi-backend.svc.cluster.local:8080
prod: http://prod-omi-llm-gateway.prod-omi-backend.svc.cluster.local:8080

• gateway chart: backend/charts/llm-gateway/
• manual workflow: .github/workflows/gcp_llm_gateway.yml
• backend caller env: backend/charts/backend-listen/{dev,prod}_omi_backend_listen_values.yaml
• shared secret mapping: backend/charts/backend-secrets/{dev,prod}_omi_backend_secrets_values.yaml

dev:  based-hardware-dev / OMI_LLM_GATEWAY_SERVICE_TOKEN
prod: based-hardware     / OMI_LLM_GATEWAY_SERVICE_TOKEN

python backend/scripts/validate-llm-gateway-env.py \
  backend/charts/backend-listen/dev_omi_backend_listen_values.yaml \
  backend/charts/llm-gateway/dev_omi_llm_gateway_values.yaml

python backend/scripts/smoke-llm-gateway.py \
  --url http://dev-omi-llm-gateway.dev-omi-backend.svc.cluster.local:8080 \
  --token "$OMI_LLM_GATEWAY_SERVICE_TOKEN"

1. Create or verify OMI_LLM_GATEWAY_SERVICE_TOKEN in based-hardware-dev Secret Manager.
2. Run the values validation command above.
3. Run focused gateway unit tests and preflight checks.
4. Manually dispatch gcp_llm_gateway.yml to development.
5. Run /ready and chat-completions smoke checks.
6. Deploy backend-listen values only after gateway smoke passes, so backend callers do not point at a missing service.

1. Create or verify an independent OMI_LLM_GATEWAY_SERVICE_TOKEN in based-hardware Secret Manager.
2. Confirm DEV smoke passed with the same image/commit.
3. Run the prod values validation command.
4. Manually dispatch gcp_llm_gateway.yml to prod.
5. Run prod /ready and chat-completions smoke checks from the prod GKE network.
6. Deploy backend-listen prod values after gateway prod smoke passes.

​

Major Library Choices

• FastAPI for HTTP endpoints and service lifecycle.
• Pydantic for lane, artifact, request, and validation schemas.
• httpx.AsyncClient for provider HTTP calls and gateway-to-provider calls.
• OpenAI Python SDK only where it materially reduces request/stream parsing risk. Prefer direct httpx for the gateway core so we preserve request/response metadata, timeouts, headers, and streaming behavior consistently.
• Prometheus-compatible metrics following existing backend observability conventions.
• pytest for deterministic resolver, validator, and executor tests.

• LangChain inside the gateway execution path. Existing product code can keep using LangChain, but the gateway should speak provider HTTP contracts directly so it can preserve OpenAI-compatible request/response shapes and streaming semantics.
• Firestore/Redis as routing config stores. All lane and route artifacts live in repo files for v1.
• LiteLLM, Portkey, Envoy AI Gateway, or Kong as the gateway foundation.
• Runtime benchmark fetching, including Artificial Analysis calls, inside request handling.

​

Open Source Position

• LiteLLM
• LiteLLM proxy docs
• Portkey Gateway
• Envoy AI Gateway supported endpoints

​

API Surface

POST /v1/chat/completions

{
  "model": "gpt-4.1-mini",
  "messages": []
}

{
  "model": "omi:auto:chat-structured",
  "messages": [],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "RequiresContext",
      "strict": true,
      "schema": {}
    }
  },
  "metadata": {
    "omi_feature": "chat_extraction.requires_context",
    "prompt_version": "chat_extraction.requires_context.v1",
    "parser_version": "RequiresContext.v1"
  }
}

{
  "model": "omi:auto:chat-structured",
  "choices": []
}

X-Omi-Lane-Id: omi:auto:chat-structured
X-Omi-Route-Id: route.chat_structured.2026_06_27.001
X-Omi-Fallback-Used: false

​

Config Model

backend/llm_gateway/config/lanes.yaml
backend/llm_gateway/config/route_artifacts.yaml
backend/llm_gateway/config/feature_bundles.yaml

lane_id: omi:auto:chat-structured
surface: openai.chat_completions
capabilities:
  text_input: true
  streaming: false
  structured_output: json_schema
  tools: false
objective:
  quality: 0.60
  latency: 0.20
  cost: 0.20
active_route: route.chat_structured.2026_06_27.001
last_known_good: route.chat_structured.2026_06_20.001

route_artifact_id: route.chat_structured.2026_06_27.001
lane_id: omi:auto:chat-structured
primary:
  provider: openai
  model: gpt-4.1-mini
fallbacks:
  - provider: openai
    model: gpt-4.1-mini
timeouts:
  request_ms: 8000
retry:
  max_attempts: 1
capabilities:
  structured_output: json_schema
  streaming: false
evidence:
  benchmark_snapshot: bench.omi.chat_structured.2026_06_27
  eval_report: eval.chat_extraction_requires_context.2026_06_27
rollout:
  stage: shadow
  percent: 0
credential_policy:
  mode: omi_paid
  allow_byok_to_omi_paid_fallback: false
fallback_policy:
  fallback_on:
    - timeout_before_output
    - provider_429_omi_paid
    - provider_5xx_omi_paid
  never_fallback_on:
    - byok_auth
    - byok_quota
    - byok_rate_limit
    - missing_byok_key
    - capability_mismatch
    - invalid_config
artifact_digest: sha256:<computed-by-validation>

feature: chat_extraction.requires_context
lane_id: omi:auto:chat-structured
prompt_version: chat_extraction.requires_context.v1
parser_version: RequiresContext.v1
eval_suite: chat_extraction_requires_context.v1
promotion_gates:
  schema_valid_rate: ">= 99.5%"
  classification_agreement_delta: ">= 0"
  false_negative_regression: "none"
  p95_latency_ms: "<= LKG + 10%"
  cost_per_success: "<= LKG + 15%"

​

Integration With Existing Backend Code

@dataclass(frozen=True)
class ExplicitRouteRef:
    feature: str
    model: str
    provider: str
    options: dict[str, object]

@dataclass(frozen=True)
class AutoLaneRouteRef:
    feature: str
    lane_id: str

• get_model(feature)
• get_provider(feature)
• get_llm(feature)
• get_route_options(feature, model, provider)

get_route_ref(feature) -> ExplicitRouteRef | AutoLaneRouteRef
is_auto_lane_id(model: str) -> bool

​

BYOK Policy

• backend forwards a short-lived BYOK credential envelope to the gateway over service-authenticated transport;
• gateway receives a key reference and resolves it through an approved internal secret path.

​

Service Auth

​

Request Validation And Route Resolution

• is_auto_lane_id(model) recognizes only the omi:auto: namespace.
• omi:auto:chat-structured is the only supported auto lane.
• unknown omi:auto:* lanes return a typed model-not-found error.
• bare provider model names such as gpt-4o-mini are not direct gateway routes in v1 and return a typed unsupported-model error.
• the resolver maps the supported lane to its configured active_route and last_known_good route artifact.
• runtime route checks defensively reject active/LKG lane, surface, capability, and credential-mode mismatches as invalid config.

• model: "omi:auto:chat-structured";
• non-empty text messages;
• stream absent or false;
• no tool use;
• response_format.type: "json_schema";
• a response_format.json_schema.schema object.

​

HTTP Surface

• requires service auth;
• accepts the same request shape validated by the resolver;
• builds an Omi-managed request credential context for the current v1 lane;
• resolves model: "omi:auto:chat-structured" to the checked-in active route;
• calls the executor and returns the provider response payload with model rewritten to the requested lane ID;
• forwards supported OpenAI chat-completions controls such as temperature, max_tokens, max_completion_tokens, seed, top_p, penalties, stop, and user;
• maps typed gateway exceptions to OpenAI-style error envelopes with error.message, error.type, error.param, and error.code;
• rejects unknown auto lanes, bare provider model names, streaming, tools, unsupported capabilities, and unknown top-level request parameters before provider execution.

​

Provider Execution

• the caller passes a resolved route plus a request-level CredentialContext;
• the executor sends an OpenAI-compatible chat-completions payload to the active route primary provider first;
• the provider-facing payload replaces the lane model with the selected provider model and forces stream: false;
• the caller-facing response payload keeps the normal provider response shape but reports model as the requested lane ID, such as omi:auto:chat-structured;
• selected route artifact, provider, provider model, fallback reason, and LKG usage are returned only on the executor result metadata, not embedded into the response payload.

• active route fallbacks are attempted only when the route policy allows the normalized failure class;
• LKG is attempted only through select_lkg_route_for_failure, so active route policy remains the single gate for runtime LKG;
• BYOK missing key, auth, quota, rate limit, unsupported provider, capability mismatch, and invalid config failures fail visibly and do not fall back by default;
• Omi-paid timeout before output, provider 429, and provider 5xx may fall back only when the route artifact allows those classes;
• streaming-after-first-token recovery is unsupported for MVP. The executor assumes no partial output exists and does not try to recover or replay partial responses.

• OpenAICompatibleChatCompletionProvider for live non-streaming OpenAI-compatible calls;
• in-memory fake providers for unit tests.

​

Observability

llm_gateway_requests_total
llm_gateway_request_latency_seconds

​

Build Status

1. Add this architecture doc and temporary implementation tickets. Done.
2. Add llm_gateway service skeleton with health endpoint. Done.
3. Add Pydantic schemas for lane config, route artifacts, feature bundles, capability declarations, retries, timeouts, rollout, evidence, credential policy, fallback policy, and LKG. Done.
4. Add service-auth and credential-policy contracts. Done.
5. Add resolver tests with no network calls. Done.
6. Add capability validator tests for chat-structured. Done.
7. Add provider executor with fake providers and fallback/error normalization tests. Done.
8. Add /v1/chat/completions non-streaming support behind service auth. Done.
9. Add /ready, local run command, service URL configuration, and separate-process deployment wiring before shadow traffic. Done.
10. Add route refs and gateway URL helpers in main backend so selected model_config.py features can call the gateway later. Done.
11. Route one low-risk structured feature to omi:auto:chat-structured in shadow mode. Done for non-BYOK chat_extraction.requires_context, protected by application-level fallback.
12. Add gateway-side metrics and authenticated metrics endpoint. Done.
13. Add Omi eval reports and promote only after schema validity, extraction quality, latency, and cost gates pass.

​

Shadow Safety

• feature-owner/privacy approval for the feature being shadowed;
• sampling limits and a kill switch;
• cost cap;
• no BYOK shadow by default;
• no provider expansion beyond the current production provider class without approval;
• no persistence of raw prompts or raw responses;
• metrics-only comparison unless an approved eval store exists.

​

Explicit Non-Goals

• No desktop Settings UI.
• No quality/latency/cost sliders.
• No per-user routing preferences.
• No Firestore or Redis routing prefs.
• No desktop-side model/route cache.
• No public /v1/auto-router/pick.
• No new public /v1/auto/model-pick.
• No request-path benchmark fetching.
• No production route from mock benchmark data.
• No benchmark-only promotion.
• No wholesale LiteLLM, Portkey, Envoy AI Gateway, or Kong adoption.

​

Maintainer Checklist

• explicit model routing remains backward compatible;
• model_config.py still owns feature-to-route mapping;
• gateway config validation fails closed on invalid prod config;
• active route has valid LKG;
• route artifacts are immutable;
• BYOK failure does not silently fall back to Omi-paid traffic;
• /v1/chat/completions requires internal service auth;
• LKG/fallback is limited to artifact-approved failure classes;
• mock benchmarks cannot load in prod;
• Omi eval report exists;
• shadow/canary completed;
• rollback is config-only;
• observability includes route, fallback, latency, errors, and cost.