Real-time Transcription

Overview

Omi’s transcription system provides real-time speech-to-text conversion with speaker identification, multiple language support, and seamless integration with the conversation processing pipeline.

Quick Start
Full Documentation
Key Concepts

Connect to /v4/listen WebSocket with your user token and start streaming audio. Transcripts arrive in real-time as JSON.

WebSocket Endpoint

WebSocket connections require Firebase authentication. The uid parameter must be a valid user ID obtained through Firebase Auth.

Endpoint URL

wss://api.omi.me/v4/listen?uid={uid}&language={lang}&sample_rate={rate}&codec={codec}

Query Parameters

uid (required)

Type: stringUser ID obtained from Firebase authentication. Required for all connections.

language

Type: string | Default: 'en'Language code for transcription. Supports:

Standard codes: 'en', 'es', 'fr', 'de', 'ja', 'zh', etc.
Multi-language: 'multi' for automatic language detection

sample_rate

Type: integer | Default: 8000Audio sample rate in Hz. Common values: 8000, 16000, 44100, 48000

codec

Type: string | Default: 'pcm8'Audio codec. Supported options:

pcm8 - 8-bit PCM (default)
pcm16 - 16-bit PCM
opus - Opus codec (16kHz)
opus_fs320 - Opus with 320 frame size
aac - AAC codec
lc3 - LC3 codec
lc3_fs1030 - LC3 with 1030 frame size

channels

Type: integer | Default: 1Number of audio channels. Use 1 for mono, 2 for stereo.

include_speech_profile

Type: boolean | Default: trueEnable speaker identification using the user’s stored speech profile. When enabled, the system extracts a speaker embedding from the user’s speech profile and uses it to identify the user’s voice via biometric matching.

conversation_timeout

Type: integer | Default: 120 | Range: 2-14400Seconds of silence before the conversation is automatically processed. After this timeout, the conversation is saved and LLM processing begins.

stt_service

Type: string | OptionalExplicitly specify STT service. Options: deepgram. If not specified, Deepgram is used.

custom_stt

Type: string | Default: 'disabled'Enable custom STT mode. When set to 'enabled', the backend accepts app-provided transcripts instead of using STT services. Useful for apps with their own transcription.

source

Type: string | OptionalConversation source identifier. Examples: 'omi', 'openglass', 'phone'

Audio Codecs

The system supports multiple audio codecs with automatic decoding:

Codec	Sample Rate	Description	Use Case
`pcm8`	8kHz	8-bit PCM	Default, low bandwidth
`pcm16`	16kHz	16-bit PCM	Better quality
`opus`	16kHz	Opus encoded	Efficient compression
`opus_fs320`	16kHz	Opus 320 frame	Alternative frame size
`aac`	Variable	AAC encoded	iOS compatibility
`lc3`	Variable	LC3 codec	Bluetooth audio
`lc3_fs1030`	Variable	LC3 1030 frame	Alternative LC3

All audio is internally converted to 16-bit linear PCM before being sent to STT providers.

STT Service Selection

The selected streaming provider comes from the ordered STT_SERVICE_MODELS configuration. The first configured model that supports the requested language wins. Pre-recorded voice and sync routes use the corresponding STT_PRERECORDED_MODEL selection contract.

Provider Capabilities

Provider	Languages	Model	Best For
Deepgram	50+	`nova-3`	Broad language coverage
Modulate	Provider-dependent	`velma-2`	Supported configured languages
Parakeet	Deployment-dependent	`parakeet`	Self-hosted streaming and batch STT

Transcription outcome contract

Transport success is not transcription success. The backend uses one bounded semantic vocabulary across voice upload, voice-message SSE, offline sync, and live provider failures:

Outcome	Meaning	Retryable
`success`	Speech-eligible audio produced non-empty normalized text	No
`expected_silence`	An explicit speech gate found no eligible speech	No
`empty_unexpected`	Eligible audio reached STT but produced no usable text	Yes
`timeout`	The selected provider timed out	Yes
`upstream_error`	The provider or response parser failed	Yes
`config_error`	The selected provider is not deployable on this runtime	No
`invalid_input`	The audio cannot be decoded or violates the route contract	No

POST /v2/voice-message/transcribe returns outcome alongside transcript, stt_provider, and stt_model. True silence remains HTTP 200 with an empty transcript and outcome=expected_silence. All failure outcomes use a non-2xx status and a fixed response body containing only error, outcome, provider, retryable, and a public-safe message; provider response bodies, audio identifiers, and exception text are never exposed. The /v2/voice-messages stream emits the same safe failure object in a terminal error: SSE frame. A normal empty stream is reserved for explicit expected_silence. For /v4/listen, an unusable initial or mid-session STT socket emits this event before the client connection closes with WebSocket code 1011:

{
  "type": "service_status",
  "status": "stt_failed",
  "outcome": "upstream_error",
  "provider": "parakeet",
  "retryable": true,
  "reason": "connection_lost"
}

The backend retains any buffered audio it could not hand to the provider. It does not keep a green client WebSocket while discarding later audio; the close activates the client’s existing reconnect or local-recovery path.

Production transcription candidate gate

The manual backend workflow deploys a no-traffic, uniquely tagged backend candidate and proves the real /v2/voice-message/transcribe route before shifting any Cloud Run traffic. The same path runs for an explicit development deployment first, which is the required identity-and-route dry run before a production release. The tag resolver requires the exact candidate revision recorded in Cloud Run status.traffic; it cannot accidentally exercise the current serving revision or latest-created revision. backend/testing/release_fixtures/transcription-release-probe.wav and its versioned JSON manifest provide the known audio, language, expected transcript, SHA-256 digest, and CC-BY-4.0 LibriSpeech provenance. The gate requires HTTP 200, outcome=success, and the exact normalized transcript. It intentionally does not assert provider or model identity: this is a semantic capability gate, not an unreviewed routing-policy setting. The shared transcription-release-candidate-probe action uses the existing authenticated deploy identity to read the existing FIREBASE_API_KEY Secret Manager secret, sign a five-minute Firebase custom token for the dedicated non-human omi-release-probe UID, and exchange it immediately for an ID token. It writes that token only to an owner-only temporary runner file, never exposes it through workflow outputs or evidence, and deletes it at step exit. The workflow makes no IAM changes: the existing deploy identity must be reviewed to have only the Secret Manager read and iam.serviceAccounts.signJwt access this action needs. Missing access fails closed before promotion. Reports contain only redacted booleans and status codes. The Cloud Run candidate itself makes the private VPC hop to Parakeet. After a passing or failed candidate gate, the workflow removes its temporary tag; a failed gate never shifts traffic.

Automatic development candidate acceptance

gcp_backend_auto_dev.yml keeps the development backend mutation lock while it deploys the four Cloud Run services (backend, backend-sync, backend-sync-backfill, and backend-integration) with the service-scoped candidate tag. It resolves each exact tag URL from Cloud Run status, runs the source-owned backend/deploy/dev_candidate_acceptance.json manifest, and only then permits the one traffic-promotion step. The backend uses the authenticated What Matters Now contract; the worker services use bounded /v1/health checks. The evidence artifact records only service, bounded contract category, and outcome — never URLs, tokens, user data, or response bodies. Candidate requests carry an OIDC token in X-Serverless-Authorization while the application keeps its own Authorization header. The token audience is the canonical Cloud Run service URL as required by Cloud Run; the HTTP target remains the exact no-traffic tagged candidate URL. Development pusher is deliberately GKE-only (pusher.omiapi.com). A legacy Cloud Run pusher service is not a deploy, candidate, or health-report surface; do not publish an image merely to make that retired surface appear ready. During the direct-URL retirement window, the public dev Cloud Run backend and legacy backend-listen services must use http://pusher.omiapi.com. Other dev Cloud Run services and jobs must not define HOSTED_PUSHER_API_URL. Retire those public endpoints only after they show no /v4/listen traffic.

Deepgram Configuration

When using Deepgram, the following options are configured:

Option	Value	Purpose
`punctuate`	`true`	Automatic punctuation insertion
`no_delay`	`true`	Minimize latency for real-time feedback
`endpointing`	`300`	300ms silence to detect sentence boundaries
`interim_results`	`false`	Only return final transcripts
`smart_format`	`true`	Format numbers, dates, currencies
`profanity_filter`	`false`	Keep all words unfiltered
`diarize`	`true`	Enable speaker identification
`filler_words`	`false`	Remove “um”, “uh”, etc.
`encoding`	`linear16`	16-bit PCM encoding

External Custom STT Service

Build your own transcription/diarization WebSocket service that integrates with Omi.

Your Service Receives

Message	Format	Description
Audio frames	Binary	Raw audio bytes (codec configured by app, typically `opus` 16kHz)
`{"type": "CloseStream"}`	JSON	End of audio stream

Your Service Sends

Format: JSON object with segments array

{
  "segments": [
    {
      "text": "Hello, how are you?",
      "speaker": "SPEAKER_00",
      "start": 0.0,
      "end": 1.5
    },
    {
      "text": "I'm doing great, thanks!",
      "speaker": "SPEAKER_01",
      "start": 1.6,
      "end": 3.2
    }
  ]
}

Segment Fields

Field	Type	Required	Description
`text`	`string`	Yes	Transcribed text
`speaker`	`string`	No	Speaker label (`SPEAKER_00`, `SPEAKER_01`, etc.)
`start`	`float`	No	Start time in seconds
`end`	`float`	No	End time in seconds

Requirements

Response must be an object with segments key. Raw arrays [{...}] will fail.
Do not include a type field, or set it to "Results". Other values are ignored.
Connection closes after 90 seconds of inactivity.

Speech Profile & Speaker Embedding

When a user has a speech profile, the system uses speaker embedding comparison to identify the user’s voice in real-time.

How It Works

Speech Profile Benefits

User Identification: Speaker embedding comparison identifies the device owner by voice biometrics
No Startup Delay: Transcription begins immediately (no profile audio prepending)
Single Socket: One Deepgram connection per session (reduced API costs)

Transcription Flow

Connection Established

WebSocket connection accepted, user validated, STT provider selected based on language.

Audio Streaming

App sends binary audio chunks. Backend decodes based on codec parameter.

STT Processing

Decoded audio sent to Deepgram. Provider returns word-level transcripts with speaker IDs.

Segment Creation

Words grouped into segments. Same-speaker consecutive words merged. Timing adjusted for speech profile offset.

Real-time Delivery

JSON segments streamed back to app immediately. UI updates as user speaks.

Conversation Lifecycle

Background task monitors silence. After conversation_timeout, conversation is processed and saved.

Message Formats

Incoming Messages (App → Backend)

Audio Data
Speaker Assignment
Custom Transcript
Image Chunk

Format: BinaryRaw audio bytes encoded according to the codec parameter. Sent continuously during recording.

[Binary audio chunk - varies by codec]

Keep-alive: Messages of 2 bytes or less are treated as heartbeat pings.

Format: JSONAssign a known person to detected speakers:

{
  "type": "speaker_assigned",
  "speaker_id": 1,
  "person_id": "person-uuid-here",
  "person_name": "John",
  "segment_ids": ["seg-uuid-1", "seg-uuid-2"]
}

Format: JSONWhen custom_stt=enabled, apps can provide their own transcripts:

{
  "type": "suggested_transcript",
  "segments": [
    {
      "text": "Hello there",
      "speaker": "SPEAKER_00",
      "speaker_id": 0,
      "start": 0.0,
      "end": 1.5,
      "is_user": true,
      "person_id": "known-person-uuid-or-null"
    }
  ],
  "stt_provider": "custom-provider-name"
}

See External Custom STT Service for building your own transcription service.

Format: JSONFor OpenGlass and visual captures:

{
  "type": "image_chunk",
  "id": "temp-image-id",
  "index": 0,
  "total": 3,
  "data": "base64-encoded-chunk"
}

Outgoing Messages (Backend → App)

Transcript Segments
Service Status
Speaker Suggestion
Conversation Created
Translations

Format: JSON ArrayReal-time transcript segments as they’re detected:

[
  {
    "id": "uuid-string",
    "text": "Hello there",
    "speaker": "SPEAKER_00",
    "speaker_id": 0,
    "is_user": true,
    "person_id": null,
    "start": 0.0,
    "end": 1.5,
    "speech_profile_processed": true,
    "stt_provider": "deepgram"
  }
]

Format: JSONConnection and service status updates:

{
  "type": "service_status",
  "status": "ready",
  "status_text": "Service Ready"
}

Format: JSONSystem suggests a known person for a detected speaker:

{
  "type": "speaker_label_suggestion",
  "speaker_id": 1,
  "person_id": "person-uuid",
  "person_name": "John",
  "segment_id": "segment-uuid"
}

Format: JSONSent when conversation timeout triggers processing:

{
  "type": "memory_created",
  "memory": {
    "id": "conversation-uuid",
    "structured": {
      "title": "Meeting Discussion",
      "overview": "..."
    }
  },
  "messages": []
}

Format: JSONWhen translation is enabled:

{
  "type": "translation",
  "segments": [
    {
      "id": "segment-uuid",
      "translations": [
        {"lang": "es", "text": "Hola ahí"}
      ]
    }
  ]
}

Transcript Segment Model

Each transcript segment contains:

Field	Type	Description
`id`	`string`	Unique UUID for the segment
`text`	`string`	Transcribed text content
`speaker`	`string`	Speaker label (`"SPEAKER_00"`, `"SPEAKER_01"`, etc.)
`speaker_id`	`integer`	Numeric speaker ID (0, 1, 2…)
`is_user`	`boolean`	`true` if spoken by device owner
`person_id`	`string?`	UUID of identified person (if matched)
`start`	`float`	Start time in seconds
`end`	`float`	End time in seconds
`speech_profile_processed`	`boolean`	Whether speech profile was used for identification
`stt_provider`	`string?`	Name of STT provider used

Connection Lifecycle

Lifecycle Events

Open

WebSocket accepted
User authentication verified
Language/STT service selected
STT connections initialized (with retry logic)
Speech profile loaded in background
Heartbeat task started (10s interval)

Stream

Audio received and decoded
Sent to STT provider(s)
Results collected in buffers
Processed every 600ms
Segments sent to client
Speaker suggestions generated

Usage statistics recorded
All STT sockets closed
Client WebSocket closed (code 1000/1001)
Buffers and collections cleared

Error Handling & Retry Logic

The system includes robust error handling:

Error Type	Handling
STT Connection Failed	Before live audio is accepted, bounded connection retry may select a configured provider. After a live provider becomes unusable, emit `service_status(stt_failed)` and close the client WebSocket with `1011`; do not discard audio as successful.
Provider Error	Pre-recorded/sync work classifies the provider failure and remains retryable. Live `/v4/listen` fails the session visibly; the desktop/mobile client retains local retry material and reconnects.
Decode Error	Log and skip corrupted audio chunk
WebSocket Error	Clean close with appropriate code

Live STT never silently downgrades a terminal provider failure to a successful transcription. The app receives the bounded failure status before the 1011 close and must retain unconfirmed local audio until a later successful sync.

Key File Locations

Component	Path
WebSocket Handler	`backend/routers/transcribe.py`
Deepgram Integration	`backend/utils/stt/streaming.py`
Audio Decoding	`backend/routers/transcribe.py`
Speech Profile	`backend/utils/stt/speech_profile.py`
VAD (Voice Activity)	`backend/utils/stt/vad.py`
Transcript Model	`backend/models/transcript_segment.py`

Backend Deep Dive

Complete backend architecture overview

Storing Conversations

How conversations and memories are stored

Chat System

How the AI chat system uses transcriptions

Backend Setup

Environment setup and configuration

Get Started

Core Development

DIY Guide

Info

Real-time Transcription

Overview

WebSocket Endpoint

Endpoint URL

Query Parameters

Audio Codecs

STT Service Selection

Provider Capabilities

Transcription outcome contract

Production transcription candidate gate

Automatic development candidate acceptance

Deepgram Configuration

External Custom STT Service

Your Service Receives

Your Service Sends

Segment Fields

Requirements

Speech Profile & Speaker Embedding

How It Works

Speech Profile Benefits

Transcription Flow

Message Formats

Incoming Messages (App → Backend)

Outgoing Messages (Backend → App)

Transcript Segment Model

Connection Lifecycle

Lifecycle Events

Error Handling & Retry Logic

Key File Locations

Backend Deep Dive

Storing Conversations

Chat System

Backend Setup

​Overview

​WebSocket Endpoint

​Endpoint URL

​Query Parameters

​Audio Codecs

​STT Service Selection

​Provider Capabilities

​Transcription outcome contract

​Production transcription candidate gate

​Automatic development candidate acceptance

​Deepgram Configuration

​External Custom STT Service

​Your Service Receives

​Your Service Sends

​Segment Fields

​Requirements

​Speech Profile & Speaker Embedding

​How It Works

​Speech Profile Benefits

​Transcription Flow

​Message Formats

​Incoming Messages (App → Backend)

​Outgoing Messages (Backend → App)

​Transcript Segment Model

​Connection Lifecycle

​Lifecycle Events

​Error Handling & Retry Logic

​Key File Locations

​Related Documentation

Backend Deep Dive

Storing Conversations

Chat System

Backend Setup

Overview

WebSocket Endpoint

Endpoint URL

Query Parameters

Audio Codecs

STT Service Selection

Provider Capabilities

Transcription outcome contract

Production transcription candidate gate

Automatic development candidate acceptance

Deepgram Configuration

External Custom STT Service

Your Service Receives

Your Service Sends

Segment Fields

Requirements

Speech Profile & Speaker Embedding

How It Works

Speech Profile Benefits

Transcription Flow

Message Formats

Incoming Messages (App → Backend)

Outgoing Messages (Backend → App)

Transcript Segment Model

Connection Lifecycle

Lifecycle Events

Error Handling & Retry Logic

Key File Locations

Related Documentation