Backend Deep Dive

​

Understanding the Omi Ecosystem

• Processing and analyzing data: Converting audio to text, extracting meaning, and creating structured information from user interactions.
• Storing conversations and extracting memories: Building a rich knowledge base of user conversations and extracted facts that Omi can draw upon to provide context and insights.
• Facilitating intelligent chat: Understanding user requests, retrieving relevant conversations, and generating personalized responses using an agentic AI system.
• Integrating with external services: Extending Omi’s capabilities and connecting it to other tools and platforms.

​

System Architecture

​

Quick Reference

​

The Flow of Information

​

What Gets Extracted

​

Core Components

​

1. database/conversations.py: The Conversation Guardian

• upsert_conversation: Creates or updates a conversation document in Firestore, ensuring efficient storage and handling of updates.
• get_conversation: Retrieves a specific conversation by its ID.
• get_conversations: Fetches a list of conversations for a user, allowing for filtering, pagination, and optional inclusion of discarded conversations.
• Photo Functions: Handles the storage and retrieval of photos associated with conversations.

class Conversation(BaseModel):
    id: str  # Unique ID
    created_at: datetime  # Creation timestamp
    started_at: Optional[datetime]
    finished_at: Optional[datetime]

    source: Optional[ConversationSource]  # omi, phone, desktop, openglass, etc.
    language: Optional[str]
    status: ConversationStatus  # in_progress, processing, completed, failed

    structured: Structured  # Contains extracted title, overview, action items, etc.
    transcript_segments: List[TranscriptSegment]
    geolocation: Optional[Geolocation]
    photos: List[ConversationPhoto]

    apps_results: List[AppResult]
    external_data: Optional[Dict]

    discarded: bool
    deleted: bool
    visibility: str  # private, shared, public

    # See StoringConversations.mdx for complete field reference

​

Conversation Lifecycle

• Manual: App sends POST /v1/conversations with empty body to trigger immediate processing
• Automatic: Backend automatically processes conversations after a timeout period (conversation_timeout parameter, default 120 seconds of silence)
• Both paths use the same process_conversation() function to extract structure, memories, and embeddings

​

2. database/vector_db.py: The Embedding Expert

• upsert_vector: Adds or updates a conversation embedding in Pinecone.
• upsert_vectors: Efficiently adds or updates multiple embeddings.
• query_vectors: Performs similarity search to find conversations relevant to a user query.
• delete_vector: Removes a conversation embedding.

• Contextual Retrieval: Finding conversations that are semantically related to a user’s request, even if they don’t share exact keywords.
• Efficient Search: Quickly retrieving relevant conversations from a large collection.
• Scalability: Handling the growing number of conversation embeddings as the user creates more conversations.

​

3. utils/llm/ Directory: The AI Maestro

• clients.py: LLM client configurations and embedding models
• chat.py: Chat-related prompts and processing
• conversation_processing.py: Conversation analysis and structuring

• Conversation Processing:
  • Determines if a conversation should be discarded.
  • Extracts structured information from transcripts (title, overview, categories, etc.).
  • Runs apps on conversation data.
• External Integration Processing:
  • Creates structured summaries from photos and descriptions (OpenGlass).
  • Processes data from external sources to generate conversations.
• Chat and Retrieval:
  • Generates initial chat messages.
  • Analyzes chat conversations to determine if context is needed.
  • Extracts relevant topics and dates from chat history.
  • Retrieves and summarizes relevant conversation content for chat responses.
• Emotional Processing:
  • Analyzes conversation transcripts for user emotions.
  • Generates emotionally aware responses based on context and user facts.
• Fact Extraction: Identifies and extracts new facts (memories) about the user from conversation transcripts.

• The LLM modules leverage OpenAI’s models (GPT-4o and others) for language understanding, generation, and reasoning.
• text-embedding-3-large is used to generate vector embeddings for conversations and user queries.

• The Brain of Omi: These modules enable Omi’s core AI capabilities, including natural language understanding, content generation, and context-aware interactions.
• Conversation Enhancement: They enrich raw data by extracting meaning and creating structured information.
• Personalized Responses: They help Omi provide responses tailored to individual users, incorporating their unique facts, conversations, and emotional states.
• Extensibility: The app system and integration with external services make Omi highly versatile.

​

4. utils/other/storage.py: The Cloud Storage Manager

• upload_profile_audio(file_path: str, uid: str):
  • Uploads a user’s speech profile audio recording to the GCS bucket specified by the BUCKET_SPEECH_PROFILES environment variable.
  • Organizes audio files within the bucket using the user’s ID (uid).
  • Returns the public URL of the uploaded file.
• get_profile_audio_if_exists(uid: str) -> str:
  • Checks if a speech profile already exists for a given user ID in the GCS bucket.
  • Downloads the speech profile audio to a local temporary file if it exists and returns the file path.
  • Returns None if the profile does not exist.

• The upload_profile_audio function is called when a user uploads a new speech profile recording through the /v3/upload-audio endpoint (defined in routers/speech_profile.py).
• The get_profile_audio_if_exists function is used to retrieve a user’s speech profile when needed, for example, during speaker identification in real-time transcription or post-processing.

​

5. database/redis_db.py: The Data Speedster

• User Speech Profile Metadata:
  • Storage: When a user uploads a speech profile, the audio file is stored in Google Cloud Storage, while the duration is cached in Redis for quick access.
  • Retrieval: During real-time transcription or post-processing, the speech profile duration is retrieved from Redis cache, while the actual audio file is loaded from Google Cloud Storage when needed for speaker identification.
• Enabled Apps:
  • Storage: A set of app IDs is stored for each user, representing the apps they have enabled.
  • Retrieval: When processing a conversation or handling a chat request, the backend checks Redis to see which apps are enabled for the user.
• App Reviews:
  • Storage: Reviews for each app (score, review text, date) are stored in Redis, organized by app ID and user ID.
  • Retrieval: When displaying app information, the backend retrieves reviews from Redis.
• Cached User Names:
  • Storage: User names are cached in Redis to avoid repeated lookups from Firebase.
  • Retrieval: The backend first checks Redis for a user’s name before querying Firestore, improving performance.

• Performance: Caching data in Redis significantly improves the backend’s speed, as frequently accessed data can be retrieved from memory very quickly.
• User Data Management: Redis provides a flexible and efficient way to manage user-specific metadata, such as app preferences, speech profile durations, and enabled apps.
• Real-time Features: The low-latency nature of Redis makes it ideal for supporting real-time features like live transcription and instant app interactions.
• Scalability: As the number of users grows, Redis helps maintain performance by reducing the load on primary databases (Firestore) and storage systems (GCS).

​

6. routers/transcribe.py: The Real-Time Transcription Engine

​

1. WebSocket Communication

• /v4/listen Endpoint: The Omi app initiates a WebSocket connection with the backend at the /v4/listen endpoint, which is defined in the websocket_endpoint function of routers/transcribe.py.
• Bidirectional Communication: WebSockets enable a two-way communication channel, allowing:
  • The Omi app to stream audio data to the backend continuously.
  • The backend to send back transcribed text segments as they become available from Deepgram.
• Real-Time Feedback: This constant back-and-forth ensures that users see their words being transcribed in real-time, creating a more interactive and engaging experience.

​

2. STT Service Integration

• Deepgram (Nova-2, Nova-3 models) - Primary service for most languages
• Soniox (stt-rt-preview model) - Used for specific language combinations
• Speechmatics - Additional fallback option

• process_audio_dg() - Manages interaction with Deepgram API (found in utils/stt/streaming.py)
• process_audio_soniox() - Manages interaction with Soniox API
• process_audio_speechmatics() - Manages interaction with Speechmatics API

​

3. Transcription Flow

1. App Streams Audio: The Omi app captures audio from the user’s device and continuously sends chunks of audio data through the WebSocket to the backend’s /v4/listen endpoint.
2. Backend Receives and Selects STT Service: The backend’s websocket_endpoint function receives the audio chunks and selects the appropriate STT service (Deepgram, Soniox, or Speechmatics) based on the language using get_stt_service_for_language().
3. Backend Forwards to STT: The backend immediately forwards audio chunks to the selected STT service using the corresponding processing function (process_audio_dg, process_audio_soniox, or process_audio_speechmatics).
4. STT Processes: The STT service’s speech recognition models transcribe the audio data in real-time.
5. Results Sent Back: The STT service sends the transcribed text segments back to the backend as they become available.
6. Backend Relays to App: The backend immediately sends these transcription results back to the Omi app over the WebSocket connection.
7. App Displays Transcript: The Omi app updates the user interface with the newly transcribed text, providing instant feedback.

​

4. Key Considerations

• Speaker Identification: The code uses the STT service’s speaker diarization feature (available in Deepgram, Soniox, and Speechmatics) to identify different speakers in the audio. This information is included in the transcription results, allowing the app to display who said what.
• User Speech Profile Integration: If a user has uploaded a speech profile, the backend can use this information (duration cached in Redis, audio file stored in Google Cloud Storage) to improve the accuracy of speaker identification.
• Latency Management: Real-time transcription requires careful attention to latency to ensure a seamless user experience. The no_delay option in Deepgram and the efficient handling of data in the backend are essential for minimizing delays.
• Error Handling: The code includes error handling mechanisms to gracefully handle any issues that may occur during the WebSocket connection or STT service transcription process.

​

5. Example Code Snippet (Simplified):

from fastapi import APIRouter, WebSocket

# ... other imports ...

router = APIRouter()


@router.websocket("/v4/listen")
async def websocket_endpoint(websocket: WebSocket, uid: str, language: str = 'en', ...):
    await websocket.accept()  # Accept the WebSocket connection

    # Start STT transcription (service selected automatically based on language)
    transcript_socket = await process_audio_dg(uid, websocket, language, ...)
    # Note: Could also be process_audio_soniox() or process_audio_speechmatics()
    # depending on language support

    # Receive and process audio chunks from the app
    async for data in websocket.iter_bytes():
        transcript_socket.send(data)

        # ... other logic for speaker identification, error handling, etc. 

​

Related Documentation

​

Need Help?