Storing Conversations & Memories

Overview

Omi uses a dual-collection architecture for storing user data:

Conversations

Primary storage for recorded interactions - transcripts, audio, structured summaries

Memories

Secondary storage for extracted facts/learnings FROM conversations

This separation allows for efficient retrieval of both full conversation context and quick access to key facts about the user.

On-device summaries in desktop reads

The app conversation response may carry a display-only client_processing projection. Binding it does not modify canonical structured or create authoritative tasks or memories. The macOS ServerConversation adapter selects the projection once for every display consumer: only schema v1 is understood, and a response containing transcript segments must match the projection’s transcript_sha256 using encoding v5, including speaker identity and attribution. A list response omitting the transcript relies on the server’s transactional binding and invalidation when the transcript changes; it cannot independently prove freshness. An enriched canonical overview, sections, events, or non-default category wins over a retained projection, including after an upgrade and cloud reprocessing. processing_state is not a cloud-origin discriminator: it can be absent with a projection, or remain local_pending after late binding. Title-only/default-field cloud results are indistinguishable from the deterministic minimum in this wire contract, so that ambiguous shape follows the projection-first rule. Canonical action items win exact-description collisions, preserving both completed and reopened state; unmatched projected items are display-only additions with no canonical task identity. Selected provenance remains available as ServerConversation.localSummary and is cached in transcription_sessions.localSummaryJson. The producer’s clientProcessingJson retry bytes are separate. Revision-aware sync replaces the selected summary atomically; older or unversioned snapshots cannot fill fields across projection/canonical sources. A cached transcript whose hash differs from the selected projection is left unloaded until detail refresh. Run the desktop regressions from the repository root:

xcrun swift test --package-path desktop/macos/Desktop --filter 'ConversationProjection(Rendering|Persistence)Tests'

These hermetic tests exercise decode, reconciliation, and production SQLite storage. Named-bundle end-to-end acceptance remains a separate check.

Architecture Diagram

Firestore Structure

users/
├── {uid}/
│   ├── conversations/                    # PRIMARY - Recorded interactions
│   │   └── {conversation_id}/
│   │       ├── id
│   │       ├── created_at
│   │       ├── started_at
│   │       ├── finished_at
│   │       ├── source
│   │       ├── language
│   │       ├── status
│   │       ├── structured
│   │       ├── transcript_segments
│   │       ├── geolocation
│   │       ├── photos/ (subcollection)
│   │       ├── audio_files
│   │       ├── apps_results
│   │       ├── discarded
│   │       ├── visibility
│   │       ├── is_locked
│   │       └── data_protection_level
│   │
│   ├── memories/                         # SECONDARY - Extracted facts
│   │   └── {memory_id}/
│   │       ├── id
│   │       ├── uid
│   │       ├── conversation_id
│   │       ├── content
│   │       ├── category
│   │       ├── tags
│   │       ├── visibility
│   │       ├── created_at
│   │       ├── updated_at
│   │       ├── reviewed
│   │       ├── user_review
│   │       ├── scoring
│   │       └── data_protection_level
│   │
│   └── action_items/                     # Standalone action items
│       └── {action_item_id}/
│           ├── description
│           ├── completed
│           ├── conversation_id
│           ├── created_at
│           ├── due_at
│           └── completed_at

Part 1: Storing Conversations

Processing Flow

API Request

The app sends a POST request to /v1/conversations with transcript data

Processing

process_conversation() in utils/conversations/process_conversation.py handles the logic

Structure Extraction

LLM extracts title, overview, action items, and events from the transcript

Storage

upsert_conversation() in database/conversations.py saves to Firestore

Vector Embedding

Conversation is embedded and stored in Pinecone for semantic search

Conversation Model Fields

Field	Type	Description
`id`	string	Unique conversation identifier
`created_at`	datetime	When the conversation record was created
`started_at`	datetime	When the actual conversation started
`finished_at`	datetime	When the conversation ended
`source`	enum	Source device (omi, phone, desktop, openglass, etc.)
`language`	string	Language code of the conversation
`status`	enum	Processing status: `in_progress`, `processing`, `completed`, `failed`
`structured`	object	Extracted structured information (see below)
`transcript_segments`	array	List of transcript segments
`geolocation`	object	Location data (latitude, longitude, address)
`photos`	array	Photos captured during conversation
`audio_files`	array	Audio file references
`apps_results`	array	Results from summarization apps
`external_data`	object	Data from external integrations
`discarded`	boolean	Whether conversation was marked as low-quality
`visibility`	enum	`private`, `shared`, or `public`
`is_locked`	boolean	Whether conversation is locked from editing
`data_protection_level`	string	`standard` or `enhanced` (encrypted)

Structured Information

The structured field contains LLM-extracted information:

Field	Type	Description
`title`	string	Short descriptive title for the conversation
`overview`	string	Summary of key points discussed
`emoji`	string	Emoji representing the conversation
`category`	enum	Category (personal, work, health, etc.)
`action_items`	array	Tasks or to-dos mentioned
`events`	array	Calendar events to be created

Transcript Segments

Each segment in transcript_segments includes:

Field	Type	Description
`text`	string	Transcribed text content
`speaker`	string	Speaker label (e.g., “SPEAKER_00”)
`start`	float	Start time in seconds
`end`	float	End time in seconds
`is_user`	boolean	Whether spoken by the device owner
`person_id`	string	ID of identified person (if matched)

Action Items

Action items are stored both inline (in structured.action_items) and in a standalone collection:

Field	Type	Description
`description`	string	The action item text
`completed`	boolean	Whether the item is done
`created_at`	datetime	When extracted
`due_at`	datetime	Optional due date
`completed_at`	datetime	When marked complete
`conversation_id`	string	Source conversation

Events

Calendar events extracted from conversations:

Field	Type	Description
`title`	string	Event title
`description`	string	Event description
`start`	datetime	Start date/time
`duration`	integer	Duration in minutes
`created`	boolean	Whether added to calendar

Part 2: Extracting & Storing Memories

Memories are facts about the user extracted from conversations. They represent learnings, preferences, habits, and other personal information.

Memory Extraction Process

During process_conversation(), the system:

Analyze Transcript

Reviews the conversation transcript for personal information

Extract Facts

Identifies facts worth remembering about the user (~15 words max)

Store with Link

Saves to memories collection with a link back to the source conversation

Memory Model Fields

Field	Type	Description
`id`	string	Unique memory identifier
`uid`	string	User ID
`conversation_id`	string	Source conversation (links back)
`content`	string	The actual fact/learning (max ~15 words)
`category`	enum	`interesting`, `system`, or `manual`
`tags`	array	Categorization tags
`visibility`	string	`private` or `public`
`created_at`	datetime	When memory was created
`updated_at`	datetime	Last modification time
`reviewed`	boolean	Whether user has reviewed
`user_review`	boolean	User’s approval (true/false/null)
`edited`	boolean	Whether user edited the content
`scoring`	string	Ranking score for retrieval
`manually_added`	boolean	Whether user created manually
`is_locked`	boolean	Prevent automatic deletion
`app_id`	string	Source app (if from integration)
`data_protection_level`	string	Encryption level

Memory Categories

Interesting

Notable facts about the user: hobbies, opinions, stories

System

Preferences and patterns: work habits, sleep schedule

Manual

User-created memories: explicitly added facts

Legacy categories (core, hobbies, lifestyle, interests, habits, work, skills, learnings, other) are automatically mapped to the new primary categories for backward compatibility.

Memory Extraction Rules

The system follows these guidelines when extracting memories:

Maximum ~15 words per memory
Must pass the “shareability test” - would this be worth telling someone?
Maximum 2 interesting + 2 system memories per conversation
No duplicate or near-duplicate facts
Skip mundane details (eating, sleeping, commuting)

Part 3: Data Protection & Encryption

Both conversations and memories support encryption for sensitive data.

Standard
Enhanced

Standard Protection Level

No encryption, stored as plaintext. This is the default for most users.

Fastest read/write performance
Data visible in Firestore console
Suitable for general use

Enhanced Protection Level

AES encryption for sensitive fields. Provides additional security for sensitive conversations.Encrypted Fields:

Conversations: transcript_segments (the actual transcript text)
Memories: content (the memory text)

Enhanced encryption adds processing overhead to read/write operations.

Implementation

# Conversations: database/conversations.py
def _prepare_conversation_for_write(conversation_data, data_protection_level):
    if data_protection_level == 'enhanced':
        # Encrypt transcript_segments before storage
        ...

def _prepare_conversation_for_read(conversation_data, data_protection_level):
    if data_protection_level == 'enhanced':
        # Decrypt transcript_segments after retrieval
        ...

Part 4: Vector Embeddings

Conversations are also stored as vector embeddings in Pinecone for semantic search.

What Gets Embedded

Data	Embedded?	Stored in Metadata?
Title	Yes	No
Overview	Yes	No
Action Items	Yes	No
Full Transcript	No (too large)	No
People Mentioned	No	Yes
Topics	No	Yes
Entities	No	Yes
created_at	No	Yes

Vector Creation

Vectors are created in a background thread after conversation processing:

# utils/conversations/process_conversation.py
threading.Thread(
    target=save_structured_vector,
    args=(uid, conversation)
).start()

The save_structured_vector() function:

Generates embedding from conversation.structured (title + overview + action_items + events)
Extracts metadata via LLM (people, topics, entities, dates)
Upserts to Pinecone with metadata filters

Vectors are created ONCE during initial processing. Reprocessed conversations do NOT update their vectors.

Key Code Locations

Component	File Path
Conversation Model	`backend/models/conversation.py`
Memory Model	`backend/models/memories.py`
Process Conversation	`backend/utils/conversations/process_conversation.py`
Database - Conversations	`backend/database/conversations.py`
Database - Memories	`backend/database/memories.py`
Router - Conversations	`backend/routers/conversations.py`
Router - Memories	`backend/routers/memories.py`
Vector Database	`backend/database/vector_db.py`

API Endpoints

Conversations

Method	Endpoint	Description
POST	`/v1/conversations`	Process and store a new conversation
GET	`/v1/conversations`	List user’s conversations
GET	`/v1/conversations/{id}`	Get specific conversation
PATCH	`/v1/conversations/{id}/title`	Update conversation title
DELETE	`/v1/conversations/{id}`	Delete a conversation

Memories

Method	Endpoint	Description
POST	`/v3/memories`	Create a manual memory
GET	`/v3/memories`	List user’s memories
PATCH	`/v3/memories/{id}`	Edit a memory
DELETE	`/v3/memories/{id}`	Delete a memory
PATCH	`/v3/memories/{id}/visibility`	Change memory visibility

Chat System Architecture

How conversations are retrieved for chat using LangGraph

Real-time Transcription

WebSocket-based real-time speech-to-text

Backend Deep Dive

General backend architecture overview

Backend Setup

Environment setup and configuration

Manual speaker attribution during concurrent writes

Manual speaker decisions live in a protected sparse receipt alongside the transcript. Live STT merging and both processing-result writers read that receipt in their commit transaction. Live merging preserves every manually covered speech span, including speaker-wide defaults, explicit clears, and training opt-outs. The committed transcript and deleted-ID delta are returned together for client delivery. Translation and inference-only writers patch current IDs without owning the segment set. Privacy-level migration re-encodes the transcript and receipt together. Changing attribution does not automatically regenerate a summary; refresh remains an explicit user action. Enrollment remains best-effort.

​Overview

Conversations

Memories

​On-device summaries in desktop reads

​Architecture Diagram

​Firestore Structure

​Part 1: Storing Conversations

​Processing Flow

​Conversation Model Fields

​Structured Information

​Transcript Segments

​Action Items

​Events

​Part 2: Extracting & Storing Memories

​Memory Extraction Process

​Memory Model Fields

​Memory Categories

Interesting

System

Manual

​Memory Extraction Rules

​Part 3: Data Protection & Encryption

​Standard Protection Level

​Enhanced Protection Level

​Implementation

​Part 4: Vector Embeddings

​What Gets Embedded

​Vector Creation

​Key Code Locations

​API Endpoints

​Conversations

​Memories

​Related Documentation

Chat System Architecture

Real-time Transcription

Backend Deep Dive

Backend Setup

​Manual speaker attribution during concurrent writes

Overview

On-device summaries in desktop reads

Architecture Diagram

Firestore Structure

Part 1: Storing Conversations

Processing Flow

Conversation Model Fields

Structured Information

Transcript Segments

Action Items

Events

Part 2: Extracting & Storing Memories

Memory Extraction Process

Memory Model Fields

Memory Categories

Memory Extraction Rules

Part 3: Data Protection & Encryption

Standard Protection Level

Enhanced Protection Level

Implementation

Part 4: Vector Embeddings

What Gets Embedded

Vector Creation

Key Code Locations

API Endpoints

Conversations

Memories

Related Documentation

Manual speaker attribution during concurrent writes