# Create Action Item
Source: https://docs.omi.me/api-reference/action-items/create-action-item
/api-reference/openapi.json post /v1/dev/user/action-items
Create a new action item for the authenticated user.
- **description**: The action item description (1-500 characters)
- **completed**: Whether the action item is completed (default: False)
- **due_at**: Optional due date in ISO 8601 format with timezone
# Create Action Items Batch
Source: https://docs.omi.me/api-reference/action-items/create-action-items-batch
/api-reference/openapi.json post /v1/dev/user/action-items/batch
Create multiple action items in a batch.
- **action_items**: List of action items to create (max 50)
# Delete Action Item
Source: https://docs.omi.me/api-reference/action-items/delete-action-item
/api-reference/openapi.json delete /v1/dev/user/action-items/{action_item_id}
Delete an action item by ID.
- **action_item_id**: The ID of the action item to delete
# Get Action Items
Source: https://docs.omi.me/api-reference/action-items/get-action-items
/api-reference/openapi.json get /v1/dev/user/action-items
Get action items with optional filters. Locked action items are excluded.
- **conversation_id**: Filter by conversation ID (None for standalone items)
- **completed**: Filter by completion status
- **start_date**: Filter by start date (inclusive)
- **end_date**: Filter by end date (inclusive)
- **limit**: Maximum number of items to return
- **offset**: Number of items to skip
# Update Action Item
Source: https://docs.omi.me/api-reference/action-items/update-action-item
/api-reference/openapi.json patch /v1/dev/user/action-items/{action_item_id}
Update an action item.
- **action_item_id**: The ID of the action item to update
- **description**: New description (optional)
- **completed**: New completion status (optional)
- **due_at**: New due date (optional, set to null to remove)
# Create Key
Source: https://docs.omi.me/api-reference/api-keys/create-key
/api-reference/openapi.json post /v1/dev/keys
Create a new Developer API key with optional scopes.
- **name**: Descriptive name for the key
- **scopes**: Optional list of scopes. If not provided, defaults to read-only access.
Available scopes:
- conversations:read
- conversations:write
- memories:read
- memories:write
- action_items:read
- action_items:write
- goals:read
- goals:write
# Delete Key
Source: https://docs.omi.me/api-reference/api-keys/delete-key
/api-reference/openapi.json delete /v1/dev/keys/{key_id}
# Get calling API key
Source: https://docs.omi.me/api-reference/api-keys/get-calling-api-key
/api-reference/openapi.json get /v1/dev/key
Return only the calling key's metadata and effective scopes. No data scope is required.
# Get Keys
Source: https://docs.omi.me/api-reference/api-keys/get-keys
/api-reference/openapi.json get /v1/dev/keys
# Ask Conversations
Source: https://docs.omi.me/api-reference/conversations/ask-conversations
/api-reference/openapi.json post /v1/dev/user/ask
Answer a natural-language question grounded in the user's own conversations.
Semantically searches the user's conversations for the question, then synthesizes a
cited answer from the most relevant ones — the same retrieval + RAG the chat surface
uses, exposed for headless / Developer-API callers (CLI, CI, scripts). Read-only:
it never writes; discarded conversations are excluded from retrieval and locked
conversations are re-checked on the authoritative record before any go to the LLM.
# Create Conversation
Source: https://docs.omi.me/api-reference/conversations/create-conversation
/api-reference/openapi.json post /v1/dev/user/conversations
Create a new conversation from text for the authenticated user.
This endpoint processes the provided text through the full conversation pipeline:
- Generates structured data (title, overview, category, emoji)
- Extracts action items (with deduplication)
- Extracts memories (with quality filtering)
- Determines if conversation should be discarded
- Triggers app integrations
- Triggers webhooks
**Request Parameters:**
- **text**: The conversation text/transcript (1-100,000 characters)
- **text_source**: Source type - audio_transcript, message, or other_text (default: other_text)
- **text_source_spec**: Additional source info (e.g., 'email', 'slack')
- **started_at**: When conversation started (defaults to now)
- **finished_at**: When conversation finished (defaults to started_at + 5 minutes)
- **language**: Language code (default: 'en')
- **geolocation**: Optional geolocation data
**Response:**
- Returns the created conversation ID and status
- Use GET /v1/dev/user/conversations/{id} to retrieve full details
# Create Conversation From Segments
Source: https://docs.omi.me/api-reference/conversations/create-conversation-from-segments
/api-reference/openapi.json post /v1/dev/user/conversations/from-segments
Create a new conversation from structured transcript segments.
This endpoint is for advanced integrations that have speaker diarization and timing information.
It processes the transcript segments through the full conversation pipeline.
**Transcript Segments:**
- **text**: The text spoken (required)
- **speaker**: Speaker identifier like 'SPEAKER_00', 'SPEAKER_01' (default: 'SPEAKER_00')
- **speaker_id**: Numeric speaker ID (auto-calculated from speaker if not provided)
- **is_user**: Whether this segment is from the user (default: False)
- **person_id**: ID of known person speaking (optional)
- **start**: Start time in seconds, e.g., 0.0, 1.5, 60.2 (required)
- **end**: End time in seconds, e.g., 1.5, 3.0, 65.8 (required)
**Other Parameters:**
- **source**: Source of conversation (default: external_integration). Options:
- omi, friend, openglass, phone, desktop, apple_watch, bee, plaud, frame, etc.
- **started_at**: When conversation started (defaults to now)
- **finished_at**: When conversation finished (calculated from last segment if not provided)
- **language**: Language code (default: 'en')
- **geolocation**: Optional geolocation data
**Example:**
```json
{
"transcript_segments": [
{
"text": "Hey, how are you doing?",
"speaker": "SPEAKER_00",
"is_user": true,
"start": 0.0,
"end": 2.5
},
{
"text": "I'm doing great, thanks!",
"speaker": "SPEAKER_01",
"is_user": false,
"start": 2.8,
"end": 5.2
}
],
"source": "phone",
"language": "en"
}
```
# Delete Conversation Endpoint
Source: https://docs.omi.me/api-reference/conversations/delete-conversation-endpoint
/api-reference/openapi.json delete /v1/dev/user/conversations/{conversation_id}
Delete a conversation by ID.
This also deletes any associated photos in the conversation's subcollection.
- **conversation_id**: The ID of the conversation to delete
# Get Conversation Endpoint
Source: https://docs.omi.me/api-reference/conversations/get-conversation-endpoint
/api-reference/openapi.json get /v1/dev/user/conversations/{conversation_id}
Get a single conversation by ID.
- **conversation_id**: The ID of the conversation to retrieve
- **include_transcript**: If True, includes full transcript_segments in the response
# Get Conversations
Source: https://docs.omi.me/api-reference/conversations/get-conversations
/api-reference/openapi.json get /v1/dev/user/conversations
Get conversations with optional transcript inclusion.
- **include_transcript**: If True, includes full transcript_segments in the response
- **folder_id**: Filter by folder ID (must be a non-empty string if provided)
- **starred**: Filter by starred status (true/false)
# Update Conversation Endpoint
Source: https://docs.omi.me/api-reference/conversations/update-conversation-endpoint
/api-reference/openapi.json patch /v1/dev/user/conversations/{conversation_id}
Update a conversation's title or discard status.
- **conversation_id**: The ID of the conversation to update
- **title**: New title for the conversation (optional)
- **discarded**: Whether the conversation is discarded (optional)
# Get Developer Daily Summaries
Source: https://docs.omi.me/api-reference/daily-summaries/get-developer-daily-summaries
/api-reference/openapi.json get /v1/dev/user/daily-summaries
List the authenticated user's stored daily recaps.
Daily summaries are derived from conversations, so this read uses the same
`conversations:read` scope and aggregate read budget as conversation lists.
# Get Developer Daily Summary
Source: https://docs.omi.me/api-reference/daily-summaries/get-developer-daily-summary
/api-reference/openapi.json get /v1/dev/user/daily-summaries/{summary_id}
Get one stored daily recap by ID.
# Search Memories Vector
Source: https://docs.omi.me/api-reference/developer/search-memories-vector
/api-reference/openapi.json get /v1/dev/user/memories/vector/search
Search developer-readable default memory memory through hydrated vector candidates.
This narrow developer API vector endpoint fails closed unless the authenticated
Developer API app/key has a verified memories.read scope and a persisted app/key
default-read grant. Vector hits are hydrated through the universal repository
against authoritative
`users/{uid}/memory_items` before returning results, so stale Short-term and
Archive remain unavailable by default.
# Create Action Item
Source: https://docs.omi.me/api-reference/endpoint/action-items/create
POST /v1/dev/user/action-items
Create a new action item for the authenticated user.
- **description**: The action item description (1-500 characters)
- **completed**: Whether the action item is completed (default: False)
- **due_at**: Optional due date in ISO 8601 format with timezone
# Create Action Items (Batch)
Source: https://docs.omi.me/api-reference/endpoint/action-items/create-batch
POST /v1/dev/user/action-items/batch
Create multiple action items in a batch.
- **action_items**: List of action items to create (max 50)
# Delete Action Item
Source: https://docs.omi.me/api-reference/endpoint/action-items/delete
DELETE /v1/dev/user/action-items/{action_item_id}
Delete an action item by ID.
- **action_item_id**: The ID of the action item to delete
# List Action Items
Source: https://docs.omi.me/api-reference/endpoint/action-items/list
GET /v1/dev/user/action-items
Get action items with optional filters. Locked action items are excluded.
- **conversation_id**: Filter by conversation ID (None for standalone items)
- **completed**: Filter by completion status
- **start_date**: Filter by start date (inclusive)
- **end_date**: Filter by end date (inclusive)
- **limit**: Maximum number of items to return
- **offset**: Number of items to skip
# Update Action Item
Source: https://docs.omi.me/api-reference/endpoint/action-items/update
PATCH /v1/dev/user/action-items/{action_item_id}
Update an action item.
- **action_item_id**: The ID of the action item to update
- **description**: New description (optional)
- **completed**: New completion status (optional)
- **due_at**: New due date (optional, set to null to remove)
# Create Conversation
Source: https://docs.omi.me/api-reference/endpoint/conversations/create
POST /v1/dev/user/conversations
Create a new conversation from text for the authenticated user.
This endpoint processes the provided text through the full conversation pipeline:
- Generates structured data (title, overview, category, emoji)
- Extracts action items (with deduplication)
- Extracts memories (with quality filtering)
- Determines if conversation should be discarded
- Triggers app integrations
- Triggers webhooks
**Request Parameters:**
- **text**: The conversation text/transcript (1-100,000 characters)
- **text_source**: Source type - audio_transcript, message, or other_text (default: other_text)
- **text_source_spec**: Additional source info (e.g., 'email', 'slack')
- **started_at**: When conversation started (defaults to now)
- **finished_at**: When conversation finished (defaults to started_at + 5 minutes)
- **language**: Language code (default: 'en')
- **geolocation**: Optional geolocation data
**Response:**
- Returns the created conversation ID and status
- Use GET /v1/dev/user/conversations/{id} to retrieve full details
# Create from Transcript Segments
Source: https://docs.omi.me/api-reference/endpoint/conversations/create-from-segments
POST /v1/dev/user/conversations/from-segments
Create a new conversation from structured transcript segments.
This endpoint is for advanced integrations that have speaker diarization and timing information.
It processes the transcript segments through the full conversation pipeline.
**Transcript Segments:**
- **text**: The text spoken (required)
- **speaker**: Speaker identifier like 'SPEAKER_00', 'SPEAKER_01' (default: 'SPEAKER_00')
- **speaker_id**: Numeric speaker ID (auto-calculated from speaker if not provided)
- **is_user**: Whether this segment is from the user (default: False)
- **person_id**: ID of known person speaking (optional)
- **start**: Start time in seconds, e.g., 0.0, 1.5, 60.2 (required)
- **end**: End time in seconds, e.g., 1.5, 3.0, 65.8 (required)
**Other Parameters:**
- **source**: Source of conversation (default: external_integration). Options:
- omi, friend, openglass, phone, desktop, apple_watch, bee, plaud, frame, etc.
- **started_at**: When conversation started (defaults to now)
- **finished_at**: When conversation finished (calculated from last segment if not provided)
- **language**: Language code (default: 'en')
- **geolocation**: Optional geolocation data
**Example:**
```json
{
"transcript_segments": [
{
"text": "Hey, how are you doing?",
"speaker": "SPEAKER_00",
"is_user": true,
"start": 0.0,
"end": 2.5
},
{
"text": "I'm doing great, thanks!",
"speaker": "SPEAKER_01",
"is_user": false,
"start": 2.8,
"end": 5.2
}
],
"source": "phone",
"language": "en"
}
```
# Delete Conversation
Source: https://docs.omi.me/api-reference/endpoint/conversations/delete
DELETE /v1/dev/user/conversations/{conversation_id}
Delete a conversation by ID.
This also deletes any associated photos in the conversation's subcollection.
- **conversation_id**: The ID of the conversation to delete
# Get Conversation
Source: https://docs.omi.me/api-reference/endpoint/conversations/get
GET /v1/dev/user/conversations/{conversation_id}
Get a single conversation by ID.
- **conversation_id**: The ID of the conversation to retrieve
- **include_transcript**: If True, includes full transcript_segments in the response
# List Conversations
Source: https://docs.omi.me/api-reference/endpoint/conversations/list
GET /v1/dev/user/conversations
Get conversations with optional transcript inclusion.
- **include_transcript**: If True, includes full transcript_segments in the response
- **folder_id**: Filter by folder ID (must be a non-empty string if provided)
- **starred**: Filter by starred status (true/false)
# Update Conversation
Source: https://docs.omi.me/api-reference/endpoint/conversations/update
PATCH /v1/dev/user/conversations/{conversation_id}
Update a conversation's title or discard status.
- **conversation_id**: The ID of the conversation to update
- **title**: New title for the conversation (optional)
- **discarded**: Whether the conversation is discarded (optional)
# List Folders
Source: https://docs.omi.me/api-reference/endpoint/folders/list
GET /v1/dev/user/folders
Get all folders for the authenticated user.
This endpoint is strictly read-only and returns an empty list if the user has no folders.
Unlike the internal `/v1/folders` endpoint, it does NOT call `initialize_system_folders`,
because doing so under a `conversations:read` scope would silently write to Firestore
(violating the read-only contract) and opens a TOCTOU window where concurrent first
requests can race past the outer empty-check and create duplicate system folders.
System folders (Work, Personal, Social) are still initialized lazily through other paths:
- The mobile app calls the internal `GET /v1/folders` whenever the conversations screen
is rendered (`app/lib/pages/conversations/conversations_page.dart`), which triggers
`initialize_system_folders` on first access.
- The conversation post-processing pipeline calls `initialize_system_folders` whenever
a new conversation is created (`backend/utils/conversations/process_conversation.py`).
In practice, any user who can issue a Developer API key has already gone through one of
those paths, so the empty-list case here only affects users who have never opened the
conversations tab nor created a single conversation.
# Create API Key
Source: https://docs.omi.me/api-reference/endpoint/keys/create
POST /v1/dev/keys
Create a new Developer API key with optional scopes.
- **name**: Descriptive name for the key
- **scopes**: Optional list of scopes. If not provided, defaults to read-only access.
Available scopes:
- conversations:read
- conversations:write
- memories:read
- memories:write
- action_items:read
- action_items:write
- goals:read
- goals:write
# Calling API Key
Source: https://docs.omi.me/api-reference/endpoint/keys/current
GET /v1/dev/key
Return only the calling key's metadata and effective scopes. No data scope is required.
# List API Keys
Source: https://docs.omi.me/api-reference/endpoint/keys/list
GET /v1/dev/keys
# Revoke API Key
Source: https://docs.omi.me/api-reference/endpoint/keys/revoke
DELETE /v1/dev/keys/{key_id}
# Create Memory
Source: https://docs.omi.me/api-reference/endpoint/memories/create
POST /v1/dev/user/memories
Create a new memory for the authenticated user.
- **content**: The content of the memory (1-500 characters)
- **category**: Memory category (auto-categorized if not provided)
- **visibility**: Visibility: public or private (default: private)
- **tags**: List of tags associated with the memory
A memory is identified by its text. Sending text the user already has (ignoring surrounding
whitespace; case-sensitive) returns that memory unchanged, so retries are safe without extra
headers. Use PATCH to change an existing memory's fields. After a delete, the same text creates
a new memory.
# Create Memories (Batch)
Source: https://docs.omi.me/api-reference/endpoint/memories/create-batch
POST /v1/dev/user/memories/batch
Create multiple memories in a batch.
- **memories**: List of memories to create (max 25)
Items follow the single-create rule: text the user already has returns the existing memory at
that position, so a retried batch creates nothing twice. `created_count` is the number of
memories returned.
# Delete Memory
Source: https://docs.omi.me/api-reference/endpoint/memories/delete
DELETE /v1/dev/user/memories/{memory_id}
Delete a memory by ID.
- **memory_id**: The ID of the memory to delete
# List Memories
Source: https://docs.omi.me/api-reference/endpoint/memories/list
GET /v1/dev/user/memories
# Update Memory
Source: https://docs.omi.me/api-reference/endpoint/memories/update
PATCH /v1/dev/user/memories/{memory_id}
Update a memory's content, visibility, tags, or category.
- **memory_id**: The ID of the memory to update
- **content**: New content for the memory (optional)
- **visibility**: New visibility: public or private (optional)
- **tags**: New tags for the memory (optional)
- **category**: New category for the memory (optional)
# Get User Folders
Source: https://docs.omi.me/api-reference/folders/get-user-folders
/api-reference/openapi.json get /v1/dev/user/folders
Get all folders for the authenticated user.
This endpoint is strictly read-only and returns an empty list if the user has no folders.
Unlike the internal `/v1/folders` endpoint, it does NOT call `initialize_system_folders`,
because doing so under a `conversations:read` scope would silently write to Firestore
(violating the read-only contract) and opens a TOCTOU window where concurrent first
requests can race past the outer empty-check and create duplicate system folders.
System folders (Work, Personal, Social) are still initialized lazily through other paths:
- The mobile app calls the internal `GET /v1/folders` whenever the conversations screen
is rendered (`app/lib/pages/conversations/conversations_page.dart`), which triggers
`initialize_system_folders` on first access.
- The conversation post-processing pipeline calls `initialize_system_folders` whenever
a new conversation is created (`backend/utils/conversations/process_conversation.py`).
In practice, any user who can issue a Developer API key has already gone through one of
those paths, so the empty-list case here only affects users who have never opened the
conversations tab nor created a single conversation.
# Create Goal
Source: https://docs.omi.me/api-reference/goals/create-goal
/api-reference/openapi.json post /v1/dev/user/goals
Create a durable goal. Metrics are optional and other goals are never changed implicitly.
- **title**: The goal title/description (1-500 characters)
- **goal_type**: Optional metric type: boolean, scale, or numeric
- **target_value**: Optional target value
- **current_value**: Optional current progress
- **min_value**: Optional minimum scale value
- **max_value**: Optional maximum scale value
- **unit**: Optional unit label (e.g., 'users', 'points')
Omit all metric fields to create a qualitative goal.
# Delete Goal
Source: https://docs.omi.me/api-reference/goals/delete-goal
/api-reference/openapi.json delete /v1/dev/user/goals/{goal_id}
Delete a goal by ID.
- **goal_id**: The ID of the goal to delete
# Get Goal
Source: https://docs.omi.me/api-reference/goals/get-goal
/api-reference/openapi.json get /v1/dev/user/goals/{goal_id}
Get a single goal by ID.
- **goal_id**: The ID of the goal to retrieve
# Get Goal History
Source: https://docs.omi.me/api-reference/goals/get-goal-history
/api-reference/openapi.json get /v1/dev/user/goals/{goal_id}/history
Get progress history for a goal.
- **goal_id**: The ID of the goal
- **days**: Number of days of history to return (max 365, default 30)
# Get Goals
Source: https://docs.omi.me/api-reference/goals/get-goals
/api-reference/openapi.json get /v1/dev/user/goals
Get user goals.
- **limit**: Maximum number of goals to return
- **include_inactive**: If True, includes inactive/completed goals
# Update Goal
Source: https://docs.omi.me/api-reference/goals/update-goal
/api-reference/openapi.json patch /v1/dev/user/goals/{goal_id}
Update a goal.
- **goal_id**: The ID of the goal to update
- **title**: New title (optional)
- **target_value**: New target value (optional)
- **current_value**: New progress value (optional)
- **min_value**: New minimum value (optional)
- **max_value**: New maximum value (optional)
- **unit**: New unit label (optional)
# Update Goal Progress
Source: https://docs.omi.me/api-reference/goals/update-goal-progress
/api-reference/openapi.json patch /v1/dev/user/goals/{goal_id}/progress
Update the progress value of a goal.
- **goal_id**: The ID of the goal to update
- **current_value**: New progress value (query parameter)
# API Reference
Source: https://docs.omi.me/api-reference/introduction
Complete reference for the Omi Developer API. Build integrations with memories, conversations, action items, and more.
## Base URL
```
https://api.omi.me
```
## Authentication
All API requests require a Developer API key in the `Authorization` header:
```http theme={null}
Authorization: Bearer omi_dev_your_api_key_here
```
Get your key from **Settings → Developer → Create Key** in the Omi app.
MCP keys (`omi_mcp_...`) are only for the MCP server at `/v1/mcp`. They are not accepted by REST
Developer API endpoints. Use a Developer API key (`omi_dev_...`) for endpoints such as
`/v1/dev/user/memories` and `/v1/dev/user/conversations`.
## Rate Limits
| Limit | Value |
| - | - |
| Per minute | 100 requests per API key |
| Per day | 10,000 requests per user |
Rate limit headers are included in every response:
```http theme={null}
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1642694400
```
## Resources
Timeless facts about the user — preferences, relationships, insights.
Conversation transcripts with AI-generated summaries and action items.
Tasks and to-dos extracted from conversations or created manually.
Create, list, and revoke developer API keys.
# Create Memories Batch
Source: https://docs.omi.me/api-reference/memories/create-memories-batch
/api-reference/openapi.json post /v1/dev/user/memories/batch
Create multiple memories in a batch.
- **memories**: List of memories to create (max 25)
Items follow the single-create rule: text the user already has returns the existing memory at
that position, so a retried batch creates nothing twice. `created_count` is the number of
memories returned.
# Create Memory
Source: https://docs.omi.me/api-reference/memories/create-memory
/api-reference/openapi.json post /v1/dev/user/memories
Create a new memory for the authenticated user.
- **content**: The content of the memory (1-500 characters)
- **category**: Memory category (auto-categorized if not provided)
- **visibility**: Visibility: public or private (default: private)
- **tags**: List of tags associated with the memory
A memory is identified by its text. Sending text the user already has (ignoring surrounding
whitespace; case-sensitive) returns that memory unchanged, so retries are safe without extra
headers. Use PATCH to change an existing memory's fields. After a delete, the same text creates
a new memory.
# Delete Memory
Source: https://docs.omi.me/api-reference/memories/delete-memory
/api-reference/openapi.json delete /v1/dev/user/memories/{memory_id}
Delete a memory by ID.
- **memory_id**: The ID of the memory to delete
# Get Memories
Source: https://docs.omi.me/api-reference/memories/get-memories
/api-reference/openapi.json get /v1/dev/user/memories
# Update Memory
Source: https://docs.omi.me/api-reference/memories/update-memory
/api-reference/openapi.json patch /v1/dev/user/memories/{memory_id}
Update a memory's content, visibility, tags, or category.
- **memory_id**: The ID of the memory to update
- **content**: New content for the memory (optional)
- **visibility**: New visibility: public or private (optional)
- **tags**: New tags for the memory (optional)
- **category**: New category for the memory (optional)
# Building the Device
Source: https://docs.omi.me/doc/assembly/Build_the_device
Step-by-step guide to assemble your own Omi device from parts.
## Before You Start
Make sure you have all required components
Download and print the case STL files
If you don't have access to a 3D printer, check [Makerspace.com](https://makerspace.com/) for printing locations near you, or use an online 3D printing service.
***
## Assembly Instructions
Gather all your parts: XIAO board, battery, switch, wires, and 3D printed case.
Cut the black wire approximately **2/3 inch (17mm)** from the base of the battery connector.
Remove a small portion of insulation from both ends of the cut wire using a wire stripper.
Use the **28 AWG** notch on your wire stripper for best results.
Follow the soldering diagram to connect the battery to the board. The switch goes in-line with the black (ground) wire.
Be careful with soldering! Use proper ventilation and safety equipment.
Insert the switch securely into the battery connector circuit.
Carefully place the battery and board into the 3D-printed case.
Insert the switch into the notch next to the USB-C slot.
Twist the longer red wire gently to help organize it within the case.
Carefully curl the twisted wire and place it to the side to ensure it fits within the case.
Align the lid with the case using the ridges as a guide and snap it into place.
Apply even pressure around the edges to ensure the seams snap securely into place.
***
## Charging Instructions
**Important:** Omi can only charge when it is powered **ON**. The device will not charge if powered off.
* Use a standard USB-C cable to charge
* The LED will indicate charging status
* A full charge takes approximately 1-2 hours
***
## Congratulations!
You now have a fully assembled Omi device!
Install the latest firmware on your device
Download and configure the Omi app
Having issues? Get help from our community or contact support
Get help from the community on Discord
# Parts List
Source: https://docs.omi.me/doc/assembly/Buying_Guide
Everything you need to build your own Omi device. Availability and prices may vary by region.
## Required Components
Seeed Studio XIAO nRF52840 Sense
Rechargeable LiPo battery
Slider switch for power control
3D printed enclosure
***
## Parts List
### Seeed Studio XIAO nRF52840 Sense Board
The brain of your Omi device - a compact Bluetooth LE board with built-in microphone.
| Region | Store | Price |
| - | - | - |
| Global | [Seeed Studio](https://www.seeedstudio.com/Seeed-XIAO-BLE-Sense-nRF52840-p-5253.html) | \$15 |
| US | [Amazon US](https://amzn.to/3TZD1pO) | \$24 |
| EU | [Amazon EU](https://amzn.eu/d/3eG6gaA) | Varies |
| India | [Robu.in](https://robu.in/product/seeed-studio-xiao-nrf52840-sense-tinyml-tensorflow-lite-imu-microphone-bluetooth5-0/) | Varies |
The Seeed Studio direct store offers the best price at \$15/unit.
***
### Rechargeable Battery
A small LiPo battery to power your device.
| Region | Store | Price |
| - | - | - |
| US | [Amazon US](https://amzn.to/3TXlE9f) | \$7 |
***
### Slider Switch
Power switch to turn your device on/off. Model: **SS12D00-G3, 3mm**
| Region | Store | Price |
| - | - | - |
| US | [Amazon US](https://www.amazon.com/dp/B099MRCDG8) | \$8 for 30+ units |
| EU | [Amazon EU](https://www.amazon.de/dp/B09TVFF6KW) | Varies |
| UK | [Amazon UK](https://www.amazon.co.uk/dp/B09TVFF6KW) | Varies |
You only need one switch per device. The packs include many extras.
***
### Wires
Thin wires for connecting components.
| Region | Store | Price |
| - | - | - |
| US | [Amazon US](https://www.amazon.com/dp/B09X4629C1) | Varies |
***
### 3D Printed Case
The enclosure for your Omi device.
Get the 3D print files from GitHub
Options for getting your case printed:
* **Local makerspaces** - Check [Makerspace.com](https://makerspace.com/) for locations near you
* **Online services** - Use services like Shapeways, Craftcloud, or PCBWay
* **Local libraries** - Many public libraries now have 3D printers available
* **University labs** - Students may have access to campus maker labs
* **Material**: PLA or PETG recommended
* **Layer height**: 0.2mm for good detail
* **Infill**: 20% is sufficient
* **Supports**: May be needed depending on orientation
***
## Important Notes
The provided links are to third-party vendors. We do not guarantee product availability or quality.
* Pricing is approximate and may vary by location and shipping fees
* Some components may have long shipping times from certain vendors
* Consider buying extras of small components like switches
***
## Next Steps
Step-by-step assembly instructions
Install firmware on your device
Set up the Omi app
# Build Your Own Omi Device
Source: https://docs.omi.me/doc/assembly/introduction
As an open-source community, we empower enthusiasts to create their own Omi devices. This guide focuses on building the DevKit 1, as later versions use custom PCBs not suited for DIY assembly.
## Overview
Building your own Omi device is a rewarding experience that gives you complete control over your hardware. This guide covers everything you need to know to build the **DevKit 1**.
All designs are freely available
Customize your build as needed
Gain hands-on electronics skills
Get help from fellow builders
***
## Is This Guide for You?
Building your own Omi device requires **advanced knowledge** of soldering and basic electronics. If you're new to electronics, consider purchasing a pre-built device instead.
| Skill Level | Recommendation |
| - | - |
| Beginner | Purchase from [Omi Shop](https://omi.me) |
| Intermediate | Follow this guide with extra caution |
| Advanced | This guide is perfect for you |
***
## Is It Cheaper?
While building your own device is a great learning experience, it may not always be cheaper than purchasing from our [Shop](https://omi.me).
Consider these factors before deciding:
Component prices vary by region
Multiple vendors mean multiple shipping fees
You may need to purchase tools
Assembly takes several hours
Compare the total cost of parts, tools, and shipping against the shop price before deciding to build your own.
***
## Requirements
### Skills Needed
Through-hole and surface-mount experience
Basic PCB and circuit understanding
Familiarity with programming boards
### Tools & Materials
| Tool | Purpose |
| - | - |
| Soldering iron | Connecting components |
| Solder | Joining connections |
| Fine-tip tweezers | Handling small parts |
| Wire stripper | Preparing wires |
| 3D printed case | Device enclosure |
***
## Important Notes
**Safety First:** Always follow safety precautions when working with electronics and soldering equipment. Work in a well-ventilated area and wear appropriate protective equipment.
**Custom PCBs:** The DevKit 2 and Omi CV1 use custom PCBs, making them unsuitable for DIY assembly without specialized manufacturing equipment. This guide focuses on DevKit 1 only.
***
## Ready to Build?
Review and purchase all components from the parts list
3D print or order the enclosure
Follow the step-by-step build guide
Install the latest firmware
***
## Next Steps
See all required components and where to buy them
Step-by-step assembly instructions
Install firmware on your device
Purchase a ready-made device instead
# App Setup
Source: https://docs.omi.me/doc/developer/AppSetup
Set up the Omi Flutter app for development. Build automatically against the local backend harness, or manually with your own.
## Overview
There are two ways to set up the Omi app for development:
**Recommended for most developers**
One command setup against the local backend harness
**For custom backends**
Full control over configuration and backend
Don't want to build from source? Download the official app from the [App Store](https://apps.apple.com/us/app/friend-ai-wearable/id6502156163) or [Google Play](https://play.google.com/store/apps/details?id=com.friend.ios).
***
## Prerequisites
Before starting, make sure you have the following installed:
Includes Dart - the core framework
Required for iOS development
Required for Android development
iOS dependency manager
You'll also need [NDK](https://developer.android.com/ndk/downloads) to build Opus for ARM devices.
The local backend harness additionally needs **Python 3.11** (not 3.12+ — the
backend pins 3.11), a **Java runtime** for the Firestore emulator, and
**firebase-tools** (or `npx`). `make dev-up` names any of these that are missing.
***
## Build the App Automatically
This is the **recommended** way to get started. `setup.sh` builds the `dev` flavor
against the **local backend harness** — the Python API on port `8000` and the
Firebase Auth emulator on port `9099`, using the `demo-omi-local` Firebase
project. iOS builds address them as `127.0.0.1`; Android builds default to the
emulator's host alias `10.0.2.2`.
`setup.sh` does **not** start the backend or the emulator — start the harness
first (step 1 below), or the app builds and launches but every request fails to
connect.
Do not point this build at `https://api.omiapi.com/`. That API verifies Firebase
ID tokens against Omi's production Firebase project, and Firebase tokens are
project-scoped, so a `demo-omi-local` token is rejected with **401 Unauthorized**
on every call — sign-in appears to succeed and nothing else works. Production
data requires the explicit `beta` profile (see [Mobile beta](https://github.com/BasedHardware/omi/blob/main/app/README.md#mobile-beta--dogfood)).
### Video Walkthrough
### Setup Steps
From the **root of the repository**, one-time setup then start the services:
```bash theme={null}
make dev-init # once: creates backend/.venv and copies the env template
make dev-up # starts the Firestore + Auth emulators and the Python API
```
`make dev-init` builds `backend/.venv` from whatever `python3` resolves to,
and the backend requires **Python 3.11** — make sure that's what you get, or
the harness fails later with import errors.
No provider API keys? Use fake providers instead:
```bash theme={null}
PROVIDER_MODE=offline make dev-up
```
Check what came up with `make dev-status`, and stop it later with
`make dev-down`. Ports, seeded local users, and troubleshooting live in the
[local emulator runbook](https://github.com/BasedHardware/omi/blob/main/backend/docs/runbooks/local-emulator-manual-qa.md).
```bash theme={null}
cd app
```
```bash theme={null}
bash setup.sh ios
```
```bash theme={null}
bash setup.sh android
```
Open the app in your IDE and hit run:
* **Xcode**: Open `app/ios` folder
* **Android Studio**: Open `app/android` folder
Or run from terminal:
```bash theme={null}
flutter run --flavor dev
```
The automatic setup runs entirely on your machine — local API, local emulators,
no production data — making it the fastest way to start building apps and making
changes. On a physical iPhone, set `OMI_DEV_HOST` to your Mac's LAN or Tailscale
address before running both `setup.sh ios` and `make dev-up` (in the same shell,
or exported so both commands see it) — this now makes the harness itself listen
there too, not just the app you build. Only a private address works (LAN, e.g.
`192.168.x.x`, or Tailscale/CGNAT, e.g. `100.x.x.x`); a public address is
rejected. Use `OMI_DEV_BIND_HOST` instead if you want the harness to bind a
different address than the one the app is compiled to reach.
***
## Build the App Manually
Manual setup gives you full control, allowing you to use your own backend.
Ensure Flutter is installed by following the official [Flutter Installation Guide](https://docs.flutter.dev/get-started/install).
Verify your setup:
```bash theme={null}
flutter doctor -v
```
```
[✓] Flutter (Channel stable, 3.44.5, on macOS 15.4.1)
[✓] Android toolchain - develop for Android devices (Android SDK version 36.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 16.4)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2025.1)
[✓] VS Code (version 1.101.0)
[✓] Connected device (4 available)
[✓] Network resources
```
This project is tested with specific tool versions. See [`app/setup.sh`](https://github.com/BasedHardware/omi/blob/main/app/setup.sh) for recommended versions:
* Flutter 3.44.5
* Xcode 16.4
* Android SDK Platform 35
* NDK 28.2.13676358
* JDK 21
To set a specific JDK on macOS:
```bash theme={null}
flutter config --jdk-dir /Library/Java/JavaVirtualMachines/jdk-21.jdk/Contents/Home
```
From the `app` directory, install packages:
```bash theme={null}
cd app
flutter pub get
```
Navigate to iOS directory and install CocoaPods dependencies:
```bash theme={null}
cd ios
pod install
pod repo update
```
Create your environment file from the template:
```bash theme={null}
cd ..
cat .env.template > .dev.env
```
Edit `.dev.env` and add your API keys:
| Key | Description |
| - | - |
| `API_BASE_URL` | Your backend URL — `http://127.0.0.1:8000/` for the local harness, or [set up your own](/doc/developer/backend/Backend_Setup) |
| `GOOGLE_MAPS_API_KEY` | Optional - for location features |
Be sure to include the trailing `/` in `API_BASE_URL` or you'll get malformed URLs. If you change this later, delete the builds folder and recreate the runner.
Whatever backend you point at must verify Firebase ID tokens against the
**same Firebase project** the app signs into. Tokens are project-scoped, so a
mismatch returns **401 Unauthorized** on every authenticated call while
sign-in still appears to succeed. Omi's shared `https://api.omiapi.com/`
verifies against the production project and will reject tokens from the
`demo-omi-local` emulator or from your own Firebase project.
Generate necessary files:
```bash theme={null}
dart pub run build_runner clean
dart pub run build_runner build
```
Firebase is **mandatory** for the app to run.
`setup.sh` installs **local emulator Firebase configs** (the `demo-omi-local`
project) for the `dev` flavor. If you're using the local backend harness (the
common case), they're already in place — skip to the next step. These configs
only work against the Firebase Auth emulator; they are not credentials for
any hosted Firebase project.
**Never run `flutterfire configure`** against Omi's bundle IDs — it overwrites the prebuilt prod credentials in `app/ios/Config/Prod/`, `app/lib/firebase_options_prod.dart`, and `app/android/app/src/prod/`.
If you need your **own** Firebase project (custom backend):
1. Follow the official [Firebase Flutter Setup](https://firebase.google.com/docs/flutter/setup) through Step 1
2. For Apple login, [create an identifier](https://developer.apple.com/account/resources/identifiers/list) first
3. Configure `flutterfire config` using **your own** bundle IDs and **your own** project — not Omi's
4. Generate SHA1/SHA256 keys for your keystore and add them to Firebase ([StackOverflow guide](https://stackoverflow.com/a/56091158) | [Official Docs](https://support.google.com/firebase/answer/9137403?hl=en))
If you're facing auth issues, enable Google/Apple sign-in in the Firebase Console under **Authentication → Sign-in method**.
Select your target device and run:
```bash theme={null}
flutter run -v --flavor dev
```
To build an APK:
```bash theme={null}
flutter build apk --flavor dev
```
***
## Code Formatting
We use `dart format` with a line length of 120 characters.
To automatically format code on commit, install the pre-commit hook:
```bash theme={null}
# From the root of the repository
ln -s -f ../../scripts/pre-commit .git/hooks/pre-commit
```
***
## Troubleshooting
* Run `flutter doctor -v` for detailed output
* Follow the suggestions to fix each issue
* Make sure all required SDKs are installed
* Ensure CocoaPods is installed: `sudo gem install cocoapods`
* Run `pod install` in the `ios` directory
* Try `pod repo update` if dependencies fail
* Check NDK is installed via Android Studio SDK Manager
* Verify JDK version matches requirements (JDK 21)
* Accept all Android licenses: `flutter doctor --android-licenses`
* Enable Google/Apple sign-in in Firebase Console
* Verify SHA1/SHA256 keys are added to Firebase
* Check bundle IDs match your Firebase configuration
`setup.sh` builds against `http://127.0.0.1:8000` but does not start anything.
Run `make dev-up` from the repo root first, and confirm with `make dev-status`.
On a physical device, `127.0.0.1` is the phone — set `OMI_DEV_HOST` to your
Mac's LAN or Tailscale address before running **both** `setup.sh` (so the
build points at your machine) and `make dev-up` (so the harness actually
listens there — it defaults to loopback-only otherwise, which is why a
device build alone used to reach nothing). Export it in the same shell so
both commands see it. The Android emulator uses `10.0.2.2` by default.
The app and the backend are on **different Firebase projects**. Firebase ID
tokens are project-scoped: a token minted for one project cannot be verified
by a backend initialized for another, so authentication succeeds locally and
every authenticated API call is rejected.
This shows up on the manual path, where you choose both sides: your own
Firebase project signs the user in while `API_BASE_URL` points at a backend
initialized for a different project. Point `API_BASE_URL` at a backend that
verifies against *your* project.
Omi's shared `https://api.omiapi.com/` is never a valid target for a
self-configured build — it verifies against Omi's own Firebase project, and
the `demo-omi-local` configs `setup.sh` installs are emulator-only fakes that
no hosted backend can verify. Production data requires the explicit `beta`
profile.
**Symptoms:**
* The dev build runs fine while `flutter run` is attached, then crashes instantly
when you open it from the Home Screen after disconnecting (or iOS relaunches it
in the background for Bluetooth/VoIP). The phone's crash report ends in
`SwiftAwesomeNotificationsPlugin.register(with:)` with `KERN_INVALID_ADDRESS at 0x0`.
* `flutter run` itself prints:
```
error: Unable to flip between RX and RW memory protection on pages
```
**Cause:**
Debug builds run the Dart VM in JIT mode, and iOS only lets Flutter tooling
start a JIT VM on a physical device. Without it, `FlutterEngine` init returns
nil, the app's `FlutterViewController` has no engine, and plugin registration
has nothing to register against. The app now shows a notice explaining this
instead of crashing, but it still cannot run.
**Solutions:**
1. **Install an AOT build for untethered use (physical devices):**
```bash theme={null}
OMI_MOBILE_BUILD_MODE=profile bash setup.sh ios
```
Profile (or `release`) builds open from the Home Screen on their own. You
lose hot reload; press `d` in `flutter run` to detach and keep the app running.
`setup.sh ios` warns whenever a debug build is about to land on a physical iPhone.
2. **Use the iOS Simulator for debug-mode development:**
* The simulator has no JIT restriction, so `flutter run --flavor dev` works
from the Home Screen too.
***
## Need Help?
Search the help channel or ask questions
Report bugs or browse existing issues
***
## Related Documentation
Set up your own Omi backend
Create Omi apps and integrations
Flash and update device firmware
How to contribute to Omi
## Testing mobile voice enrollment
First-run speech-profile enrollment asks for a short sample about any topic.
Its progress uses approximately five seconds of user transcript spans; the
server still validates the uploaded WAV. Settings redo uses its existing
sentence target. Run the provider and onboarding widget regressions from `app/`:
```bash theme={null}
flutter test test/providers/speech_profile_provider_test.dart test/widgets/speech_profile_onboarding_test.dart
```
For device verification, use the dev flavor and dev backend. Check speaking,
Skip, and upload failure on the actual onboarding step. Advancing through All
Done is a navigation event; verify the upload result separately.
# Real-Time Audio Streaming
Source: https://docs.omi.me/doc/developer/AudioStreaming
Stream audio bytes from your Omi device to any backend service in real time.
## Overview
Omi streams raw audio bytes from your device to any webhook endpoint you configure. Use this to build custom audio processing, live transcription, or recording workflows.
Stream to your own server, cloud function, or local dev environment
Choose how often you receive audio chunks (e.g. every 5, 10, or 30 seconds)
Receive raw PCM bytes — add a WAV header to save as audio files
***
## Setup
Build an endpoint that accepts POST requests with raw audio bytes.
Omi sends requests in this format:
```
POST /your-endpoint?sample_rate=16000&uid=user123
Content-Type: application/octet-stream
[raw audio bytes]
```
| Parameter | Description |
| - | - |
| `sample_rate` | Audio samples per second (see table below) |
| `uid` | The user's unique ID in Omi |
| Body | Raw PCM audio bytes (`application/octet-stream`) |
| Device | Sample Rate |
| - | - |
| Omi (consumer) | 16,000 Hz |
| DevKit 2 | 16,000 Hz |
| DevKit 1 (v1.0.4+) | 16,000 Hz |
| DevKit 1 (v1.0.2) | 8,000 Hz |
1. Open the Omi app
2. Go to **Settings → Developer Mode**
3. Scroll to **Realtime audio bytes**
4. Enter your webhook URL
5. Set the interval (e.g. `10` for every 10 seconds)
Audio bytes should start arriving at your endpoint. The data is raw PCM — to save as a playable audio file, prepend a WAV header to the accumulated bytes.
***
## Example: Save to Google Cloud Storage
A complete example that saves audio chunks as WAV files in GCS.
Follow the [Cloud Audio Storage](/doc/developer/savingaudio) guide to create a bucket and get credentials.
1. Fork [github.com/mdmohsin7/omi-audio-streaming](https://github.com/mdmohsin7/omi-audio-streaming)
2. Clone and deploy to any cloud provider (GCP, AWS, DigitalOcean) or run locally with [ngrok](https://ngrok.com/)
3. Set these environment variables:
| Variable | Value |
| - | - |
| `GOOGLE_APPLICATION_CREDENTIALS_JSON` | Base64-encoded GCP key |
| `GCS_BUCKET_NAME` | Your bucket name |
The repo includes a Dockerfile for easy deployment.
Set the endpoint in the Omi app under **Realtime audio bytes** to:
```
https://your-deployment.com/audio
```
Audio files will appear in your GCS bucket at the configured interval.
***
## Related
Set up GCS bucket and credentials
Create Omi apps with audio capabilities
# Contribution Guide
Source: https://docs.omi.me/doc/developer/Contribution
Contribute to Omi and earn rewards! Some tasks have paid bounties.
## Will this be accepted?
Before you write code, check product fit:
* Read [`PRODUCT.md`](https://github.com/BasedHardware/Omi/blob/main/PRODUCT.md) — memory-first loop, one shared chat mind, trust over cleverness, integrations harness, taste floor.
* Check the [product invariant registry](https://github.com/BasedHardware/Omi/tree/main/product/invariants). Locked invariants (for example one shared transcript across notch and main chat) are binding; PRs that touch their paths must name the invariant ID.
* **Large or ambiguous features** should start as a GitHub issue so maintainers can align on scope and direction before major implementation work begins.
Maintainers declining a PR for direction or taste will cite an invariant ID or open a proposed invariant the same week — so “project direction” stays written down, not tribal.
***
## How to Contribute
Go to [github.com/BasedHardware/Omi](https://github.com/BasedHardware/Omi) and click **Fork**.
Clone your fork, create a branch, and implement your changes.
Add focused tests and include the commands or manual steps you used to verify the change. Before pushing, draft the PR body and run
`make preflight`, then run `scripts/pr-preflight --pr-body-file /path/to/draft-pr-body.md` to validate invariant citations and any required failure-class declaration. Use `scripts/pr-preflight --suggest` before drafting the body to get the paste-ready invariant section and failure-class guidance. The manifest-backed checks are identical locally and in CI.
Submit a PR and specify which issue it relates to. Name any product invariants your change affects.
***
## Mobile capture changes
The [capture ownership contract](https://github.com/BasedHardware/Omi/blob/main/app/lib/services/capture/OWNERSHIP.md)
defines constructor seams, session generations and staged recovery/foreground
ownership handoffs. Declare completed file/rule adoption in
`app/contracts/capture/boundary-baseline.json`. Only adopted files and new files
are constrained; ordinary legacy-file growth stays unblocked. Diagnostics name
the request interface to use. Run `TZ=UTC bash app/test.sh` for the protected contracts. Pending tests
execute and fail on unexpected success; builders remove only their pending marker.
## Pull Request Quality
Good PRs are easy to review and come with evidence that the change works.
* **Bug fixes** include the regression test that would have caught the bug.
* **Classify reactive fixes.** A `fix:` commit responds to a triggering defect
and must declare `Failure-Class: FC- | new | none` in its PR body. A
`harden:` commit strengthens a boundary or contract without a triggering
incident; it may cite a class but needs no declaration. The validator is
offline and required; its `report` command is advisory and only identifies
classes eligible for a maintainer to mark `dormant`.
* **Keep instance fixes compact.** An existing-class declaration replaces the
root-cause and durable-guard narrative. Provide that extended narrative only
when declaring `new`, changing a class's canonical prevention primitive or
owner, or making a registry-only lifecycle transition. An instance-fix PR never
edits registry data. To mark a class dormant, a separate registry-only PR sets
`status` to `dormant` with a UTC ISO-8601 `dormant_since`. When `report` flags a
recurrence after that timestamp, merge a separate registry-only reopen PR that
sets `status` to `open` and removes `dormant_since`, then classify the instance
fix. IDs use stable semantic lower-kebab slugs such as `FC-malformed-doc-read`;
use `scripts/failure-class prepare` or `explain` for the exact field and
relevant context.
* **Prefer primitives to guards.** Before adding a check or ratchet, answer in
the PR body: “Why isn't this a shared primitive instead?” A new guard must
cite the real merged PR or incident it would have caught; no real instance
means the check does not land.
* **Small features** can go straight to PR when they are well scoped: test the core path and the main error path, and describe any manual verification you did.
* **Large or ambiguous features** should start as a GitHub issue so maintainers can align on scope and direction before major implementation work begins.
* **Show your verification.** List the commands you ran and what they showed in the PR description. "It compiles" is not evidence — exercising the real user-facing path is.
* **Tests that run in CI must be hermetic** — no live services, no network. Validation against a live service is welcome as extra evidence, but it can never be the only proof a PR works.
* **CI on fork PRs** — fork PRs run the cheap checks (Hygiene, Formatting, metadata, affected backend/Python units, Static Contracts, and OpenAPI). Slow backend guardrails and the Windows Python ACL check wait for full CI. When a contribution is ready for full review, ask a maintainer to add the `ci:full` label, then use **Re-run all jobs** on the PR's latest commit. CI reads the current labels on rerun; adding a label alone does not start new workflows or cancel an in-flight run. The label persists across new commits, which then get full CI automatically. Heavy aggregate checks name this flow when coverage is deferred.
* **Backend hermetic gate** — every PR reports `Backend Hermetic Merge Gate`. Affected full-tier PRs run the backend E2E harness; emulator gauntlets run for their integration inputs, main, and nightly. A cheap-tier fork reports deferral instead of a green heavy-check verdict.
* **Behavioral tests execute production behavior.** Source-string assertions are
static tripwires only; they do not prove lifecycle, concurrency, or integration
behavior.
* **Product invariants** — if you touch paths listed on a locked invariant, name the ID (for example `INV-CHAT-1`) in the PR body.
* **Fast local deterministic checks** — run `make preflight`. Selection includes staged, unstaged, and untracked files; commit-based checks still need a final run after committing. New deterministic diff-scoped checks belong in `.github/checks-manifest.yaml`, never directly in workflow YAML.
* **CI budget** — when adding or changing a workflow or job, run `python3 .github/scripts/check_ci_budget.py --refresh`, then review `.github/ci-budget.yaml`. Set a p50 runtime budget, expected PR runs per day, owner, and rationale for new jobs. A wider trigger, larger matrix, runner change, or raised budget needs an explicit budget and rationale edit in the same PR. After CI diet changes merge, re-seed measured fields with `--refresh --reseed --seed-dayruns --seed-dayjobs ` and review the diff. The checked-in caps cover PR Linux and macOS runner minutes per day and internal time-to-signal. The weekly audit samples completed PR runs with `GITHUB_TOKEN`, publishes a report artifact, and updates one tracking issue on overrun or insufficient samples.
* **PR metadata contract check** — draft the body, then run `scripts/pr-preflight --pr-body-file /path/to/draft-pr-body.md`.
Use `scripts/pr-preflight --suggest` when you need the paste-ready invariant section. Once the branch has a PR,
`scripts/pr-preflight` reads its current body automatically with `gh` (or set `OMI_PR_BODY_FILE`). Pre-push runs these
same contracts. When editing only the PR body, use
`scripts/pr-preflight --metadata-only --pr-body-file /path/to/draft-pr-body.md`.
This runs every applicable metadata-dependent check and reports all failures together.
CI uses this mode for title, body, and label edits; code updates still run the full preflight.
Metadata success validates the PR description and does not replace the source-code verdict.
Backend compatibility checks cover adopted released-client shapes in
`contracts/client-compat/`. Preserve fields and request forms those clients use;
new unused endpoints stay unrestricted. A failure identifies the release and
field: restore compatibility or version the endpoint, rather than editing a
historical fixture. Replay uses synthetic records in the existing backend E2E
suite; a successful build tag alone is not proof of store distribution.
For typed mobile analytics, add events and executable consumer references to
`contracts/analytics/events.json`, then run
`python3 scripts/check_event_registry.py --write` and commit the generated Dart
API and tracking plan. Preserve emitted names and properties during migration.
The raw-call check covers adopted files; new and existing unadopted files stay
unrestricted. See `app/lib/utils/analytics/registry/REGISTRY.md` for batch tests.
### Contributing with an AI agent
Omi is built to be agent-friendly. If you use an AI coding agent, it will automatically pick up [`AGENTS.md`](https://github.com/BasedHardware/Omi/blob/main/AGENTS.md) at the repo root (plus per-component guides in `backend/`, `app/`, and `desktop/macos/`). The root guide routes to component instructions and repository conventions. The PR quality requirements above apply to all contributions; maintainer-machine instructions apply only when using those machines.
***
## Documentation Contributions
Docs contributions are just as important as code contributions! Our docs are located in the [GitHub docs folder](https://github.com/BasedHardware/Omi/tree/main/docs) and sync in real-time with [docs.omi.me](https://docs.omi.me).
You can edit docs directly from GitHub: Fork the repo → [click the edit icon](https://share.cleanshot.com/dkBSffGr) → click "Preview" → create a pull request.
***
## Contribution Rewards
Get rewarded for approved PRs with significant contributions:
Free DevKit device (necklace or glasses, depending on contribution area)
\$100 of transcription credits
\$500 of transcription credits + special Contributor role in Discord
Trip to hang out with the Based Hardware team
### What Qualifies as a Significant Contribution?
Something you spent 5+ hours on. Can be multiple smaller contributions that add up.
Either a new feature or a major bug fix.
Quality code with clear explanations, focused tests, and verification notes.
To claim your reward, email [team@basedhardware.com](mailto:team@basedhardware.com) with:
* List of your contributions
* Your address
* Email and phone number
***
## Paid Bounties
Get paid to improve AI wearables!
Browse open issues with the "Paid Bounty" label
### Bounty Rules
| Rule | Description |
| - | - |
| Code must be merged | Your code must be merged into the master branch |
| Eligibility | Bounty eligibility is solely at our discretion |
| Locking a task | You can "lock" a bounty task, but only after your first PR is merged |
| Claiming payment | Email [team@basedhardware.com](mailto:team@basedhardware.com) with bounty link + PayPal account |
### No Bounties Available?
If there are no paid bounties available, you can suggest your own:
Filter by [open issues](https://github.com/BasedHardware/Omi/issues?q=is:open+is:issue) without the "bounty" label.
Comment on the issue and ping @josancamon19 or @kodjima33 with your bounty suggestion.
***
## Get Started
Browse all open issues
View issues with bounties
Join the community
Contribute to docs
# Cursor Configuration (.cursor)
Source: https://docs.omi.me/doc/developer/Cursor
How Omi uses the .cursor folder (rules, skills, commands, and agents) to guide AI assistance in this repository.
## Overview
This repo includes a `.cursor/` directory that configures Cursor + AI agents to work effectively across Omi’s codebase. It contains:
* Rules that apply based on file patterns (backend/flutter/web/etc.)
* Reusable “skills” and slash commands
* Specialized agents (e.g. backend API, Flutter, firmware)
* Internal reference docs for agents (not the same as the public `docs/` site)
If you’re contributing to Omi and using Cursor, understanding `.cursor/` makes it much easier to add consistent automation and avoid common mistakes.
***
## `.cursor/` vs `docs/` (important)
* **`docs/`** is the public Mintlify site at `docs.omi.me`. What belongs there is defined in `docs/AGENTS.md`: how to use the app, hardware/DIY, and how to build on Omi. Unlisted MDX or MD under `docs/` is still a public URL.
* **`.cursor/`** is **editor/agent configuration** for contributors and AI tooling inside this repo.
Do not dump a runbook, flag table, invariant, or agent-only rule into `docs/` "temporarily." Those live next to the owning code (`backend/docs/`, `desktop/macos/docs/`, `.github/agent-docs/`, `product/invariants/`) or in a private tracker. We document `.cursor/` publicly here so contributors can use it; `.cursor/docs/` is not the public site.
***
## What’s inside `.cursor/`
### Rules (`.cursor/rules/`)
Rules are small, focused guidance files (typically `*.mdc`) that apply automatically based on globs (e.g. `backend/**/*.py`, `app/**/*.dart`, `web/**/*.ts(x)`).
Typical rule topics:
* **Architecture**: module boundaries + import hierarchy
* **Testing & verification**: what to run before merging
* **Style and conventions**: formatting, localization requirements, common mistakes
### Skills (`.cursor/skills/`)
Skills are reusable “playbooks” for recurring workflows. A skill folder typically contains:
* `SKILL.md`: the skill’s instructions and checklists
* `commands/`: slash commands (e.g. `/backend-test`, `/pr`)
* `agents/`: specialized agents that focus on specific domains
### Commands (`.cursor/commands/`)
Legacy / general commands that show up in Cursor when you type `/`. (Many commands also live under `skills/*/commands/`.)
### Agents (`.cursor/agents/`)
Legacy / compatibility agent definitions. Most agents live under `skills/*/agents/`.
### Internal agent docs (`.cursor/docs/`)
Internal reference docs used by agents (architecture maps, component indexes, internal guidance).
**Note**: These are for agent operation and contributor context, not for end users.
### Hooks (`.cursor/hooks/` and `.cursor/hooks.json`)
Optional hook scripts/config to observe and extend the agent loop. Useful for enforcing workflows, validation, and automation.
### MCP config (`.cursor/mcp.json`)
Configuration for MCP servers that provide external tools (e.g. browser testing, Notion/Figma integration if enabled).
***
## Common contributor workflows
### Add a new rule
1. Add a focused rule file under `.cursor/rules/` (e.g. `backend-foo.mdc`)
2. Ensure it targets the correct file glob(s)
3. Keep it actionable: conventions + “how to verify”
### Add a new skill
1. Create `.cursor/skills//SKILL.md`
2. Put any related slash commands under `.cursor/skills//commands/`
3. Put any related agents under `.cursor/skills//agents/`
### Update public documentation
If your change affects how people use the app, hardware, or public developer APIs:
* Read `docs/AGENTS.md` first. Only then update `docs/` and `docs/docs.json`.
* Operator runbooks, flags, and invariants do not go on the public site.
***
## Safety & maintenance notes
* **Don’t put secrets** (tokens, keys, credentials) in `.cursor/` files.
* Keep rules **layer-aware** (e.g. don’t encourage importing from `routers/` inside `utils/`).
* Prefer linking to **public docs in `docs/`** only when the page fits `docs/AGENTS.md`. Keep `.cursor/docs/` for internal agent context.
# DevKit 2 Testing
Source: https://docs.omi.me/doc/developer/DevKit2Testing
Test storage, button, and SD card functionality on the Omi DevKit 2.
## Overview
Two areas to test on DevKit 2: **SD card storage** (offline audio recording) and **button behavior** (tap events via BLE).
**Never remove the SD card while the device is on.** It is not hot-swappable. Use the safe removal process described below.
***
## SD Card Storage
When the device is disconnected from the phone (LED is red), audio data is written to the SD card instead of streaming over Bluetooth.
### SD Card Commands
Commands are sent as 6-byte packets: `[command, file_number, offset_byte0, offset_byte1, offset_byte2, offset_byte3]`
| Command | Action |
| - | - |
| `0` | Read specified file from offset |
| `1` | Clear all bytes in specified file |
| `2` | Clear all files in directory |
| `3` | Stop transmission of specified file |
**Example:** Read file 1 starting at byte 255:
```
[0, 1, 255, 0, 0, 0]
```
### Response Codes
| Code | Meaning |
| - | - |
| `0` | Command successful |
| `100` | End of file transmission (last byte) |
| `200` | Delete process finished |
| `3` | Invalid file (file 0 never exists; files start at 1) |
| `4` | File size is 0 |
| `6` | Invalid command |
If you don't receive code `100` or `200`, the command was not completed.
***
## Testing Process
**Prerequisites:** Python packages from `requirements.txt`, plus `ffmpeg` and `opus` for audio decoding. The test scripts live in [`omi/firmware/scripts/devkit/`](https://github.com/BasedHardware/omi/tree/main/omi/firmware/scripts/devkit).
Run these scripts in order:
Run `discover_devices.py` to list all nearby Bluetooth devices. Look for **"Omi"** or **"Omi Devkit 2"** and note the device ID.
Run `get_info_list.py` — replace the `device_id` value with your device ID. This lists the size of each audio file on the SD card.
Run `get_audio_file.py` — replace the `device_id` value with your device ID. Audio data streams to `my_file.txt` in the current directory.
Run `decode_audio.py` to convert the downloaded data into a WAV file. If successful, you'll have a playable recording of what was spoken near the mic.
Run `delete_audio_files.py` — replace the `device_id` value with your device ID. This clears audio data from the specified file.
***
## Safe SD Card Removal
Double-tap the reset button inside the kit
Remove the card to inspect contents on your computer
Put the SD card back and tap the button to restart
***
## Device Behavior
| Connection State | LED | SD Card |
| - | - | - |
| No Bluetooth | Red | Writing audio to SD card |
| Bluetooth connected | Blue | Stops writing; streams over BLE |
### App-Side SD Card Import
When the device connects to the app with stored audio:
1. A green banner appears showing estimated download time
2. Tap the banner (or go to **Settings → Device Settings → Storage**) to open the import screen
3. Press the download button to start importing — a progress bar shows completion
The green banner only appears for DevKit 2 devices. DevKit 1 devices do not support SD card import.
***
## Button Events
After connecting to the BLE notification service, these events fire:
| Event Code | Action |
| - | - |
| `1` | Single tap |
| `2` | Double tap |
| `3` | Long press (\~3 seconds) |
| `4` | Button pressed down |
| `5` | Button released |
Verify that each physical action sends the correct BLE packet.
***
## Known Issues
Some devices create an empty `a01.txt` file, preventing audio from being written.
**Fix:**
1. Delete the audio file
2. Press the reset button to reboot
3. The device will auto-generate the required files on startup
# App-Device Protocol
Source: https://docs.omi.me/doc/developer/Protocol
BLE protocol specification for communicating with Omi wearable devices
## Overview
This document describes the Bluetooth Low Energy (BLE) protocol used to communicate between the Omi app and device. Use this reference when building custom apps or SDKs.
***
## BLE Discovery
The official app discovers the device by scanning for BLE devices with the name `Omi`.
***
## BLE Services
The Omi wearable implements three BLE services:
Standard BLE battery level monitoring
Device metadata and firmware version
Audio streaming and codec configuration
***
## Battery Service
Battery notifications require firmware v1.5 or later. Earlier versions (1.0.x) do not support notifications.
| Property | Value |
| - | - |
| Service UUID | `0x180F` (standard) |
| Characteristic UUID | `0x2A19` (Battery Level) |
| Supports Notifications | Yes (firmware v1.5+) |
This is the standard [BLE Battery Service](https://www.bluetooth.com/specifications/specs/battery-service).
***
## Device Information Service
Available since firmware version 1.0.3
| Property | Value |
| - | - |
| Service UUID | `0x180A` (standard) |
This is the standard [BLE Device Information Service](https://www.bluetooth.com/specifications/specs/device-information-service/).
### Characteristics
| Characteristic | UUID | Value |
| - | - | - |
| Manufacturer Name | `0x2A29` | "Based Hardware" |
| Model Number | `0x2A24` | "Omi" |
| Hardware Revision | `0x2A27` | "Seeed Xiao BLE Sense" |
| Firmware Revision | `0x2A26` | e.g., "1.0.3" |
***
## Audio Streaming Service
This is the main service for streaming audio from the device to the app.
| Property | Value |
| - | - |
| Service UUID | `19B10000-E8F2-537E-4F6C-D104768A1214` |
### Characteristics
| Characteristic | UUID | Purpose |
| - | - | - |
| Audio Data | `19B10001-E8F2-537E-4F6C-D104768A1214` | Audio stream from device |
| Codec Type | `19B10002-E8F2-537E-4F6C-D104768A1214` | Audio codec identifier |
***
## Codec Types
The codec type characteristic determines how to decode the audio data:
| Value | Codec | Sample Rate | Bit Depth |
| - | - | - | - |
| `0` | PCM | 16 kHz | 16-bit mono |
| `1` | PCM | 8 kHz | 16-bit mono |
| `10` | Mu-law | 16 kHz | 8-bit mono |
| `11` | Mu-law | 8 kHz | 8-bit mono |
| `20` | Opus | 16 kHz | 16-bit mono |
Starting with firmware v1.0.3, the default codec is **Opus**. Earlier versions used PCM 16-bit at 8 kHz.
***
## Audio Data Format
Audio data is sent as BLE notifications on the Audio Data characteristic.
### Packet Structure
```mermaid theme={null}
flowchart LR
subgraph Header["Header (3 bytes)"]
PN[Packet Number
2 bytes]
IDX[Index
1 byte]
end
subgraph Payload["Audio Payload"]
DATA[Audio Samples
160 samples/packet]
end
Header --> Payload
```
Each value update includes a 3-byte header:
| Bytes | Field | Description |
| - | - | - |
| 0-1 | Packet Number | Overall packet counter (0-65535, wraps around) |
| 2 | Index | Position of this value within the current packet |
Each audio packet contains 160 samples. If the packet exceeds `(negotiated BLE MTU - 3 bytes)`, it will be split across multiple value notifications.
**Example:** PCM 16-bit at 16 kHz
* 160 samples × 2 bytes = 320 bytes per packet
* On iOS devices, this typically results in:
* Notification 1: packet n, index 0, 251 bytes
* Notification 2: packet n+1, index 1, 75 bytes
All audio data is sent in **little-endian** format.
***
## Implementation Example
```python theme={null}
# Example: Parsing audio packet header
def parse_audio_header(data: bytes):
packet_number = int.from_bytes(data[0:2], 'little')
index = data[2]
audio_data = data[3:]
return packet_number, index, audio_data
```
***
## Related
Python SDK with Opus decoding
Native iOS SDK
Cross-platform mobile SDK
Compile custom firmware
# Action Items
Source: https://docs.omi.me/doc/developer/api/action-items
Create and retrieve action items (tasks/to-dos) via the Developer API
## Endpoints
Retrieve action items
Create action item
Update action item
Delete action item
Create up to 50
***
## Get Action Items
Retrieve your action items with optional filtering
| Parameter | Type | Default | Description |
| - | - | - | - |
| `limit` | integer | 100 | Maximum number of items to return |
| `offset` | integer | 0 | Number of items to skip |
| `completed` | boolean | - | Filter by completion status |
| `conversation_id` | string | - | Filter by conversation ID |
| `start_date` | datetime | - | Filter by creation start date (inclusive) |
| `end_date` | datetime | - | Filter by creation end date (inclusive) |
```bash theme={null}
curl -H "Authorization: Bearer $API_KEY" \
"https://api.omi.me/v1/dev/user/action-items?completed=false&limit=50"
```
```python theme={null}
import requests
response = requests.get(
"https://api.omi.me/v1/dev/user/action-items",
headers={"Authorization": f"Bearer {API_KEY}"},
params={"completed": False, "limit": 50}
)
action_items = response.json()
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/action-items?completed=false&limit=50",
{ headers: { Authorization: `Bearer ${API_KEY}` } }
);
const actionItems = await response.json();
```
```json theme={null}
[
{
"id": "action_101",
"description": "Review budget proposal",
"completed": false,
"created_at": "2025-01-20T10:15:00Z",
"updated_at": "2025-01-20T10:15:00Z",
"due_at": null,
"completed_at": null,
"conversation_id": "conv_202"
},
{
"id": "action_102",
"description": "Call dentist for appointment",
"completed": true,
"created_at": "2025-01-19T14:00:00Z",
"updated_at": "2025-01-20T09:00:00Z",
"due_at": "2025-01-25T00:00:00Z",
"completed_at": "2025-01-20T09:00:00Z",
"conversation_id": null
}
]
```
| Field | Type | Description |
| - | - | - |
| `id` | string | Unique identifier |
| `description` | string | The action item text |
| `completed` | boolean | Whether the item is completed |
| `created_at` | datetime | When the item was created |
| `updated_at` | datetime | When the item was last updated |
| `due_at` | datetime | Due date (null if not set) |
| `completed_at` | datetime | When completed (null if not completed) |
| `conversation_id` | string | Associated conversation ID (null for standalone items) |
***
## Create Action Item
Create a new action item (task/to-do)
| Parameter | Type | Required | Description |
| - | - | - | - |
| `description` | string | **Yes** | The action item description (1-500 characters) |
| `completed` | boolean | No | Whether completed (default: `false`) |
| `due_at` | datetime | No | Due date in ISO 8601 format with timezone |
```bash theme={null}
curl -X POST "https://api.omi.me/v1/dev/user/action-items" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"description": "Review pull request #123",
"completed": false,
"due_at": "2025-12-08T15:00:00+00:00"
}'
```
```python theme={null}
import requests
response = requests.post(
"https://api.omi.me/v1/dev/user/action-items",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
},
json={
"description": "Review pull request #123",
"completed": False,
"due_at": "2025-12-08T15:00:00+00:00"
}
)
action_item = response.json()
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/action-items",
{
method: "POST",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
description: "Review pull request #123",
completed: false,
due_at: "2025-12-08T15:00:00+00:00"
})
}
);
const actionItem = await response.json();
```
```json theme={null}
{
"id": "action_abc123",
"description": "Review pull request #123",
"completed": false,
"created_at": "2025-12-06T10:30:00Z",
"updated_at": "2025-12-06T10:30:00Z",
"due_at": "2025-12-08T15:00:00+00:00",
"completed_at": null,
"conversation_id": null
}
```
Push notifications are automatically sent to the Omi app if a `due_at` date is provided.
***
## Create Action Items (Batch)
Create multiple action items in a single request (max 50)
| Parameter | Type | Required | Description |
| - | - | - | - |
| `action_items` | array | **Yes** | List of action item objects (max 50) |
Each action item object accepts the same fields as the single create endpoint.
```bash theme={null}
curl -X POST "https://api.omi.me/v1/dev/user/action-items/batch" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"action_items": [
{"description": "Review Q1 report", "due_at": "2025-12-10T17:00:00Z"},
{"description": "Schedule team meeting"}
]
}'
```
```python theme={null}
import requests
response = requests.post(
"https://api.omi.me/v1/dev/user/action-items/batch",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
},
json={
"action_items": [
{"description": "Review Q1 report", "due_at": "2025-12-10T17:00:00Z"},
{"description": "Schedule team meeting"}
]
}
)
result = response.json()
print(f"Created {result['created_count']} action items")
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/action-items/batch",
{
method: "POST",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
action_items: [
{ description: "Review Q1 report", due_at: "2025-12-10T17:00:00Z" },
{ description: "Schedule team meeting" }
]
})
}
);
const result = await response.json();
console.log(`Created ${result.created_count} action items`);
```
```json theme={null}
{
"action_items": [
{ "id": "action_001", "description": "Review Q1 report", ... },
{ "id": "action_002", "description": "Schedule team meeting", ... }
],
"created_count": 2
}
```
***
## Update Action Item
Update an action item's description, completion status, or due date
| Parameter | Type | Description |
| - | - | - |
| `action_item_id` | string | The ID of the action item to update |
| Parameter | Type | Required | Description |
| - | - | - | - |
| `description` | string | No | New description (1-500 characters) |
| `completed` | boolean | No | New completion status |
| `due_at` | datetime | No | New due date (ISO 8601 format) |
At least one field must be provided. Setting `completed: true` automatically sets `completed_at`.
```bash theme={null}
curl -X PATCH "https://api.omi.me/v1/dev/user/action-items/action_abc123" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"completed": true
}'
```
```python theme={null}
import requests
# Mark action item as completed
response = requests.patch(
"https://api.omi.me/v1/dev/user/action-items/action_abc123",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
},
json={"completed": True}
)
action_item = response.json()
print(f"Completed at: {action_item['completed_at']}")
```
```javascript theme={null}
// Mark action item as completed
const response = await fetch(
"https://api.omi.me/v1/dev/user/action-items/action_abc123",
{
method: "PATCH",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({ completed: true })
}
);
const actionItem = await response.json();
console.log(`Completed at: ${actionItem.completed_at}`);
```
```json theme={null}
{
"id": "action_abc123",
"description": "Review pull request #123",
"completed": true,
"created_at": "2025-12-06T10:30:00Z",
"updated_at": "2025-12-25T14:00:00Z",
"due_at": "2025-12-08T15:00:00+00:00",
"completed_at": "2025-12-25T14:00:00Z",
"conversation_id": null
}
```
When updating `due_at`, a push notification is sent to the Omi app to remind the user.
***
## Delete Action Item
Delete an action item permanently
| Parameter | Type | Description |
| - | - | - |
| `action_item_id` | string | The ID of the action item to delete |
```bash theme={null}
curl -X DELETE "https://api.omi.me/v1/dev/user/action-items/action_abc123" \
-H "Authorization: Bearer $API_KEY"
```
```python theme={null}
import requests
response = requests.delete(
"https://api.omi.me/v1/dev/user/action-items/action_abc123",
headers={"Authorization": f"Bearer {API_KEY}"}
)
result = response.json()
if result["success"]:
print("Action item deleted successfully")
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/action-items/action_abc123",
{
method: "DELETE",
headers: { Authorization: `Bearer ${API_KEY}` }
}
);
const result = await response.json();
if (result.success) {
console.log("Action item deleted successfully");
}
```
```json theme={null}
{
"success": true
}
```
This action is permanent. Deleted action items cannot be recovered.
***
## Use Case: Import Tasks
Import tasks from an external task management system into Omi:
```python theme={null}
import requests
API_KEY = "omi_dev_your_api_key"
# Your tasks from another app
external_tasks = [
{"title": "Complete project proposal", "due": "2025-12-15T10:00:00Z", "done": False},
{"title": "Send weekly report", "due": "2025-12-12T17:00:00Z", "done": False},
{"title": "Review team feedback", "due": None, "done": True}
]
# Convert to Omi action items format
action_items = [
{
"description": task["title"],
"due_at": task["due"],
"completed": task["done"]
}
for task in external_tasks
]
# Create batch
response = requests.post(
"https://api.omi.me/v1/dev/user/action-items/batch",
headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
json={"action_items": action_items}
)
print(f"Imported {response.json()['created_count']} action items")
```
```javascript theme={null}
const API_KEY = process.env.OMI_API_KEY;
// Your tasks from another app
const externalTasks = [
{ title: "Complete project proposal", due: "2025-12-15T10:00:00Z", done: false },
{ title: "Send weekly report", due: "2025-12-12T17:00:00Z", done: false },
{ title: "Review team feedback", due: null, done: true }
];
// Convert to Omi action items format
const actionItems = externalTasks.map(task => ({
description: task.title,
due_at: task.due,
completed: task.done
}));
// Create batch
const response = await fetch(
"https://api.omi.me/v1/dev/user/action-items/batch",
{
method: "POST",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({ action_items: actionItems })
}
);
const result = await response.json();
console.log(`Imported ${result.created_count} action items`);
```
***
## Use Case: Get Pending Tasks
```bash theme={null}
# Get all incomplete action items
curl -H "Authorization: Bearer $API_KEY" \
"https://api.omi.me/v1/dev/user/action-items?completed=false"
```
```python theme={null}
import requests
API_KEY = "omi_dev_your_api_key"
# Get pending tasks
response = requests.get(
"https://api.omi.me/v1/dev/user/action-items",
headers={"Authorization": f"Bearer {API_KEY}"},
params={"completed": False, "limit": 100}
)
pending_tasks = response.json()
print(f"You have {len(pending_tasks)} pending tasks")
for task in pending_tasks:
due = task.get("due_at", "No due date")
print(f"- {task['description']} (Due: {due})")
```
```javascript theme={null}
const API_KEY = process.env.OMI_API_KEY;
const response = await fetch(
"https://api.omi.me/v1/dev/user/action-items?completed=false&limit=100",
{ headers: { Authorization: `Bearer ${API_KEY}` } }
);
const pendingTasks = await response.json();
console.log(`You have ${pendingTasks.length} pending tasks`);
pendingTasks.forEach(task => {
const due = task.due_at || "No due date";
console.log(`- ${task.description} (Due: ${due})`);
});
```
# Ask
Source: https://docs.omi.me/doc/developer/api/ask
Ask a question over your conversations
Use a Developer API key with `conversations:read`. See [Quick Start](/doc/developer/api/overview) for authentication and rate limits.
| Method | Endpoint |
| - | - |
| `POST` | `/v1/dev/user/ask` |
# Conversations
Source: https://docs.omi.me/doc/developer/api/conversations
Create and retrieve conversations via the Developer API
## Endpoints
Retrieve conversations
Get one conversation
Create from text
Update conversation
Delete conversation
Create from transcript
***
## Get Conversations
Retrieve your conversation transcripts (only completed, non-discarded)
| Parameter | Type | Default | Description |
| - | - | - | - |
| `limit` | integer | 25 | Maximum number to return |
| `offset` | integer | 0 | Number to skip |
| `start_date` | datetime | - | Filter by start date (inclusive) |
| `end_date` | datetime | - | Filter by end date (inclusive) |
| `include_transcript` | boolean | false | Include full transcript segments |
| `categories` | string | - | Comma-separated list of categories to filter by (e.g. `work,personal`). Matches conversations whose category is **any** of the listed values (OR semantics). |
| `folder_id` | string | - | Filter by folder ID. Each conversation belongs to at most one folder, so this is a single-value filter. Must be a non-empty string if provided. Use `GET /v1/dev/user/folders` to retrieve folder IDs. |
| `starred` | boolean | - | Filter by starred status. `true` for starred only, `false` for non-starred only. Omit to return all. |
```bash theme={null}
curl -H "Authorization: Bearer $API_KEY" \
"https://api.omi.me/v1/dev/user/conversations?limit=10&include_transcript=true"
```
```python theme={null}
import requests
response = requests.get(
"https://api.omi.me/v1/dev/user/conversations",
headers={"Authorization": f"Bearer {API_KEY}"},
params={"limit": 10, "include_transcript": True}
)
conversations = response.json()
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/conversations?limit=10&include_transcript=true",
{ headers: { Authorization: `Bearer ${API_KEY}` } }
);
const conversations = await response.json();
```
```json theme={null}
[
{
"id": "conv_202",
"created_at": "2025-01-20T13:50:00Z",
"started_at": "2025-01-20T13:50:00Z",
"finished_at": "2025-01-20T14:10:00Z",
"language": "en",
"source": "omi",
"structured": {
"title": "Feature Discussion",
"overview": "Brainstorming session for new features",
"emoji": "💼",
"category": "business",
"action_items": [
{
"description": "Create mockups for new UI",
"completed": false,
"created_at": "2025-01-20T14:10:00Z",
"updated_at": "2025-01-20T14:10:00Z",
"due_at": null,
"completed_at": null
}
],
"events": []
},
"folder_id": "folder_uuid",
"folder_name": "Work",
"geolocation": {
"latitude": 37.7749295,
"longitude": -122.4194155,
"address": "1 Market St, San Francisco, CA 94105, USA",
"google_place_id": "ChIJIQBpAG2ahYAR_6128GcTUEo",
"location_type": "street_address"
}
}
]
```
```json theme={null}
[
{
"id": "conv_202",
"created_at": "2025-01-20T13:50:00Z",
"started_at": "2025-01-20T13:50:00Z",
"finished_at": "2025-01-20T14:10:00Z",
"language": "en",
"source": "phone",
"structured": {
"title": "Feature Discussion",
"overview": "Brainstorming session for new features",
"emoji": "💼",
"category": "business",
"action_items": [],
"events": []
},
"transcript_segments": [
{
"id": "seg_001",
"text": "Let's discuss the new feature",
"speaker_id": 0,
"speaker_name": "John",
"start": 0.0,
"end": 2.5
},
{
"id": "seg_002",
"text": "I think we should focus on the user interface first",
"speaker_id": 1,
"speaker_name": "Speaker 1",
"start": 2.5,
"end": 5.8
}
],
"folder_id": "folder_uuid",
"folder_name": "Work",
"geolocation": {
"latitude": 37.7749295,
"longitude": -122.4194155,
"address": "1 Market St, San Francisco, CA 94105, USA",
"google_place_id": "ChIJIQBpAG2ahYAR_6128GcTUEo",
"location_type": "street_address"
}
}
]
```
| Field | Type | Description |
| - | - | - |
| `id` | string | Unique identifier |
| `created_at` | datetime | When the conversation record was created |
| `started_at` | datetime | When the conversation started |
| `finished_at` | datetime | When the conversation ended |
| `language` | string | Language code (e.g., "en") |
| `source` | string | Source device/app |
| `structured` | object | AI-generated structured data |
| `transcript_segments` | array | Transcript segments (if `include_transcript=true`) |
| `folder_id` | string | Folder ID the conversation belongs to (optional) |
| `folder_name` | string | Folder name, resolved from the user's folder settings. Null if no folder is assigned. (optional) |
| `geolocation` | object | Location where conversation occurred (if available) |
| Field | Type | Description |
| - | - | - |
| `id` | string | Segment identifier (optional) |
| `text` | string | The transcribed text |
| `speaker_id` | integer | Numeric speaker identifier (optional) |
| `speaker_name` | string | Human-readable speaker name. Resolved from user profiles or the People database. Falls back to a generic name like "Speaker 1" if unidentified. (optional) |
| `start` | float | Start timestamp in seconds |
| `end` | float | End timestamp in seconds |
| Field | Type | Description |
| - | - | - |
| `latitude` | float | Latitude coordinate |
| `longitude` | float | Longitude coordinate |
| `address` | string | Human-readable address (optional) |
| `google_place_id` | string | Google Places API ID (optional) |
| `location_type` | string | Type of location, e.g., "premise", "street\_address" (optional) |
| Field | Type | Description |
| - | - | - |
| `title` | string | AI-generated title |
| `overview` | string | Summary of the conversation |
| `emoji` | string | Representative emoji |
| `category` | string | Category (work, personal, etc.) |
| `action_items` | array | Extracted action items |
| `events` | array | Extracted calendar events |
***
## Create Conversation from Text
Create a new conversation from text with full AI processing
| Parameter | Type | Required | Description |
| - | - | - | - |
| `text` | string | **Yes** | The conversation text/transcript (1-100,000 characters) |
| `text_source` | string | No | Source type: `audio_transcript`, `message`, or `other_text` (default: `other_text`) |
| `text_source_spec` | string | No | Additional source info (e.g., 'email', 'slack', 'whatsapp') |
| `started_at` | datetime | No | When conversation started (defaults to now) |
| `finished_at` | datetime | No | When conversation finished (defaults to `started_at` + 5 minutes) |
| `language` | string | No | Language code (default: 'en') |
| `geolocation` | object | No | Geolocation data with `latitude` and `longitude` |
```bash theme={null}
curl -X POST "https://api.omi.me/v1/dev/user/conversations" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"text": "Meeting notes: Discussed Q1 goals. Action item: prepare budget by Friday.",
"text_source": "other_text",
"text_source_spec": "meeting_notes",
"started_at": "2025-12-06T14:00:00+00:00",
"language": "en"
}'
```
```python theme={null}
import requests
response = requests.post(
"https://api.omi.me/v1/dev/user/conversations",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
},
json={
"text": "Meeting notes: Discussed Q1 goals. Action item: prepare budget by Friday.",
"text_source": "other_text",
"text_source_spec": "meeting_notes",
"started_at": "2025-12-06T14:00:00+00:00",
"language": "en"
}
)
conversation = response.json()
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/conversations",
{
method: "POST",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
text: "Meeting notes: Discussed Q1 goals. Action item: prepare budget by Friday.",
text_source: "other_text",
text_source_spec: "meeting_notes",
started_at: "2025-12-06T14:00:00+00:00",
language: "en"
})
}
);
const conversation = await response.json();
```
```json theme={null}
{
"id": "conv_456",
"status": "completed",
"discarded": false
}
```
When you create a conversation, the following happens automatically:
* **Discard Detection**: Determines if content is meaningful enough to save
* **Structured Generation**: Creates title, overview, category, and emoji
* **Action Item Extraction**: Identifies and creates action items (with deduplication against past 2 days)
* **Memory Extraction**: Extracts interesting facts and insights (up to 4 per conversation)
* **App Integration**: Triggers enabled summarization apps
* **Webhooks**: Notifies external systems
Memories and action items are extracted asynchronously. Use the GET endpoints to retrieve them after creation.
***
## Create Conversation from Transcript Segments
Create from structured transcript with speaker diarization
When a conversation created with a stable client session ID is deleted, retries return HTTP 200
with the same `id`, `status: "deleted"`, and `discarded: true`, without recreating or processing
content. Clients should resolve the upload and purge its local transcript rather than retrying
or fetching the deleted conversation. Requests without a client session ID create a fresh random
identity and cannot be linked to an earlier deleted upload.
Deletion intent is retained as content-free metadata under the account and is removed by account
deletion. It commits before derived-data cleanup; a failed delete remains retryable while this
intent prevents backlog recreation. No automatic tombstone expiry is currently configured.
| Parameter | Type | Required | Description |
| - | - | - | - |
| `transcript_segments` | array | **Yes** | List of transcript segments (1-500 segments) |
| `client_session_id` | string | No | Stable upload identity (aliases: `client_conversation_id`, `session_id`, `client_id`); retries return the same conversation ID |
| `source` | string | No | Source of conversation (default: `phone`); segments use transcript processing for every source |
| `started_at` | datetime | No | When conversation started (defaults to now) |
| `finished_at` | datetime | No | When conversation finished (calculated from last segment if not provided) |
| `language` | string | No | Language code (default: 'en') |
| `geolocation` | object | No | Geolocation data |
| Field | Type | Required | Description |
| - | - | - | - |
| `text` | string | **Yes** | The spoken text |
| `speaker` | string | No | Speaker identifier (e.g., 'SPEAKER\_00', 'SPEAKER\_01'). Default: 'SPEAKER\_00' |
| `speaker_id` | integer | No | Numeric speaker ID (auto-calculated if not provided) |
| `is_user` | boolean | No | Whether this segment is from the user. Default: `false` |
| `person_id` | string | No | ID of known person speaking |
| `start` | float | **Yes** | Start time in seconds (e.g., 0.0, 1.5, 60.2) |
| `end` | float | **Yes** | End time in seconds (e.g., 1.5, 3.0, 65.8) |
`omi`, `friend`, `openglass`, `phone`, `desktop`, `apple_watch`, `bee`, `plaud`, `frame`, `screenpipe`, `workflow`, `sdcard`, `external_integration`
```bash theme={null}
curl -X POST "https://api.omi.me/v1/dev/user/conversations/from-segments" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"transcript_segments": [
{
"text": "Let'\''s review the quarterly results",
"speaker": "SPEAKER_00",
"is_user": true,
"start": 0.0,
"end": 3.5
},
{
"text": "Revenue is up 25% from last quarter",
"speaker": "SPEAKER_01",
"is_user": false,
"start": 4.0,
"end": 7.2
}
],
"source": "phone",
"language": "en"
}'
```
```python theme={null}
import requests
response = requests.post(
"https://api.omi.me/v1/dev/user/conversations/from-segments",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
},
json={
"transcript_segments": [
{
"text": "Let's review the quarterly results",
"speaker": "SPEAKER_00",
"is_user": True,
"start": 0.0,
"end": 3.5
},
{
"text": "Revenue is up 25% from last quarter",
"speaker": "SPEAKER_01",
"is_user": False,
"start": 4.0,
"end": 7.2
}
],
"source": "phone",
"language": "en"
}
)
conversation = response.json()
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/conversations/from-segments",
{
method: "POST",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
transcript_segments: [
{
text: "Let's review the quarterly results",
speaker: "SPEAKER_00",
is_user: true,
start: 0.0,
end: 3.5
},
{
text: "Revenue is up 25% from last quarter",
speaker: "SPEAKER_01",
is_user: false,
start: 4.0,
end: 7.2
}
],
source: "phone",
language: "en"
})
}
);
const conversation = await response.json();
```
```json theme={null}
{
"id": "conv_789",
"status": "completed",
"discarded": false
}
```
Unlike text-based conversations, transcript segments ARE stored in the `transcript_segments` field and can be retrieved with `include_transcript=true`.
***
## Get Single Conversation
Retrieve a single conversation by ID
| Parameter | Type | Description |
| - | - | - |
| `conversation_id` | string | The ID of the conversation to retrieve |
| Parameter | Type | Default | Description |
| - | - | - | - |
| `include_transcript` | boolean | false | Include full transcript segments |
```bash theme={null}
curl -H "Authorization: Bearer $API_KEY" \
"https://api.omi.me/v1/dev/user/conversations/conv_456?include_transcript=true"
```
```python theme={null}
import requests
response = requests.get(
"https://api.omi.me/v1/dev/user/conversations/conv_456",
headers={"Authorization": f"Bearer {API_KEY}"},
params={"include_transcript": True}
)
conversation = response.json()
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/conversations/conv_456?include_transcript=true",
{ headers: { Authorization: `Bearer ${API_KEY}` } }
);
const conversation = await response.json();
```
```json theme={null}
{
"id": "conv_456",
"created_at": "2025-01-20T13:50:00Z",
"started_at": "2025-01-20T13:50:00Z",
"finished_at": "2025-01-20T14:10:00Z",
"language": "en",
"source": "external_integration",
"structured": {
"title": "Q1 Goals Discussion",
"overview": "Meeting to discuss quarterly goals and budget preparation",
"emoji": "📊",
"category": "business",
"action_items": [
{
"description": "Prepare budget by Friday",
"completed": false
}
],
"events": []
},
"geolocation": {
"latitude": 37.7749295,
"longitude": -122.4194155,
"address": "1 Market St, San Francisco, CA 94105, USA",
"google_place_id": "ChIJIQBpAG2ahYAR_6128GcTUEo",
"location_type": "street_address"
}
}
```
***
## Update Conversation
Update a conversation's title or discard status
| Parameter | Type | Description |
| - | - | - |
| `conversation_id` | string | The ID of the conversation to update |
| Parameter | Type | Required | Description |
| - | - | - | - |
| `title` | string | No | New title (1-500 characters) |
| `discarded` | boolean | No | Whether the conversation is discarded |
At least one field must be provided.
```bash theme={null}
curl -X PATCH "https://api.omi.me/v1/dev/user/conversations/conv_456" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"title": "Q1 Planning Meeting - Budget Discussion"
}'
```
```python theme={null}
import requests
response = requests.patch(
"https://api.omi.me/v1/dev/user/conversations/conv_456",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
},
json={"title": "Q1 Planning Meeting - Budget Discussion"}
)
conversation = response.json()
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/conversations/conv_456",
{
method: "PATCH",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
title: "Q1 Planning Meeting - Budget Discussion"
})
}
);
const conversation = await response.json();
```
```json theme={null}
{
"id": "conv_456",
"created_at": "2025-01-20T13:50:00Z",
"started_at": "2025-01-20T13:50:00Z",
"finished_at": "2025-01-20T14:10:00Z",
"language": "en",
"source": "external_integration",
"structured": {
"title": "Q1 Planning Meeting - Budget Discussion",
"overview": "Meeting to discuss quarterly goals and budget preparation",
"emoji": "📊",
"category": "business",
"action_items": [],
"events": []
},
"geolocation": {
"latitude": 37.7749295,
"longitude": -122.4194155,
"address": "1 Market St, San Francisco, CA 94105, USA",
"google_place_id": "ChIJIQBpAG2ahYAR_6128GcTUEo",
"location_type": "street_address"
}
}
```
Use `discarded: true` to hide a conversation from the main list without permanently deleting it.
***
## Delete Conversation
Delete a conversation permanently
| Parameter | Type | Description |
| - | - | - |
| `conversation_id` | string | The ID of the conversation to delete |
```bash theme={null}
curl -X DELETE "https://api.omi.me/v1/dev/user/conversations/conv_456" \
-H "Authorization: Bearer $API_KEY"
```
```python theme={null}
import requests
response = requests.delete(
"https://api.omi.me/v1/dev/user/conversations/conv_456",
headers={"Authorization": f"Bearer {API_KEY}"}
)
result = response.json()
if result["success"]:
print("Conversation deleted successfully")
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/conversations/conv_456",
{
method: "DELETE",
headers: { Authorization: `Bearer ${API_KEY}` }
}
);
const result = await response.json();
if (result.success) {
console.log("Conversation deleted successfully");
}
```
```json theme={null}
{
"success": true
}
```
This action is permanent. Deleted conversations and their associated photos cannot be recovered. Consider using `PATCH` with `discarded: true` to hide a conversation instead.
***
## Use Case: Upload Meeting Transcript
```python theme={null}
import requests
from datetime import datetime, timezone
API_KEY = "omi_dev_your_api_key"
# Your meeting transcript with speaker diarization
transcript = [
{"speaker": "SPEAKER_00", "text": "Welcome everyone to the standup", "start": 0.0, "end": 2.5},
{"speaker": "SPEAKER_01", "text": "Thanks. I finished the API integration yesterday", "start": 3.0, "end": 6.2},
{"speaker": "SPEAKER_00", "text": "Great work! Any blockers?", "start": 6.5, "end": 8.0},
{"speaker": "SPEAKER_01", "text": "Just waiting on the design review", "start": 8.3, "end": 10.5},
]
# Convert to Omi format
segments = [
{
"text": seg["text"],
"speaker": seg["speaker"],
"is_user": seg["speaker"] == "SPEAKER_00",
"start": seg["start"],
"end": seg["end"]
}
for seg in transcript
]
# Create conversation
response = requests.post(
"https://api.omi.me/v1/dev/user/conversations/from-segments",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
},
json={
"transcript_segments": segments,
"source": "phone",
"started_at": datetime.now(timezone.utc).isoformat(),
"language": "en"
}
)
result = response.json()
print(f"Created conversation: {result['id']}")
print(f"Status: {result['status']}")
print(f"Discarded: {result['discarded']}")
```
```javascript theme={null}
const API_KEY = process.env.OMI_API_KEY;
// Your meeting transcript with speaker diarization
const transcript = [
{ speaker: "SPEAKER_00", text: "Welcome everyone to the standup", start: 0.0, end: 2.5 },
{ speaker: "SPEAKER_01", text: "Thanks. I finished the API integration yesterday", start: 3.0, end: 6.2 },
{ speaker: "SPEAKER_00", text: "Great work! Any blockers?", start: 6.5, end: 8.0 },
{ speaker: "SPEAKER_01", text: "Just waiting on the design review", start: 8.3, end: 10.5 },
];
// Convert to Omi format
const segments = transcript.map(seg => ({
text: seg.text,
speaker: seg.speaker,
is_user: seg.speaker === "SPEAKER_00",
start: seg.start,
end: seg.end
}));
// Create conversation
const response = await fetch(
"https://api.omi.me/v1/dev/user/conversations/from-segments",
{
method: "POST",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
transcript_segments: segments,
source: "phone",
started_at: new Date().toISOString(),
language: "en"
})
}
);
const result = await response.json();
console.log(`Created conversation: ${result.id}`);
console.log(`Status: ${result.status}`);
console.log(`Discarded: ${result.discarded}`);
```
***
## Use Case: Export Conversations
```bash theme={null}
#!/bin/bash
API_KEY="omi_dev_your_api_key"
# Export all conversations with transcripts
curl -H "Authorization: Bearer $API_KEY" \
"https://api.omi.me/v1/dev/user/conversations?limit=1000&include_transcript=true" \
> conversations_backup.json
echo "Exported conversations to conversations_backup.json"
```
```python theme={null}
import requests
import json
API_KEY = "omi_dev_your_api_key"
# Paginate through all conversations
all_conversations = []
offset = 0
limit = 100
while True:
response = requests.get(
"https://api.omi.me/v1/dev/user/conversations",
headers={"Authorization": f"Bearer {API_KEY}"},
params={
"limit": limit,
"offset": offset,
"include_transcript": True
}
)
conversations = response.json()
if not conversations:
break
all_conversations.extend(conversations)
offset += limit
print(f"Fetched {len(all_conversations)} conversations...")
# Save to file
with open("conversations_backup.json", "w") as f:
json.dump(all_conversations, f, indent=2)
print(f"Exported {len(all_conversations)} conversations")
```
```javascript theme={null}
const fs = require('fs');
const API_KEY = process.env.OMI_API_KEY;
async function exportConversations() {
const allConversations = [];
let offset = 0;
const limit = 100;
while (true) {
const response = await fetch(
`https://api.omi.me/v1/dev/user/conversations?limit=${limit}&offset=${offset}&include_transcript=true`,
{ headers: { Authorization: `Bearer ${API_KEY}` } }
);
const conversations = await response.json();
if (conversations.length === 0) break;
allConversations.push(...conversations);
offset += limit;
console.log(`Fetched ${allConversations.length} conversations...`);
}
fs.writeFileSync(
'conversations_backup.json',
JSON.stringify(allConversations, null, 2)
);
console.log(`Exported ${allConversations.length} conversations`);
}
exportConversations();
```
## Public shared notes for agents
For a conversation the owner has already shared, the same public URL supports
HTML, Markdown, and the public JSON payload. These representations require no
login and use the existing share visibility rules; private or revoked shares
return 404 for machine representations and show an unavailable page in HTML.
```bash theme={null}
curl -H 'Accept: text/markdown' https://h.omi.me/conversations/CONVERSATION_ID
curl -H 'Accept: application/json' https://h.omi.me/conversations/CONVERSATION_ID
```
The `.md` and `.json` suffixes select the same representations, independent of
`Accept`. Default requests receive HTML. Share responses vary on `Accept` and
advertise alternate representations in HTTP `Link` headers; the HTML page also
contains alternate links. [Share discovery](https://h.omi.me/llms.txt) documents
these URLs.
Markdown comes from the public structured payload, including overview, sections,
participants, action items, and transcript. The explicit allowlist exposes section
heading, body and kind, plus action-item description, completion, `owner_name`,
`due_at` and `context`. It excludes attendee emails, private calendar fields,
insights, task IDs, confidence scores and section evidence IDs. Existing public
note text is included as authored; this is not a content redaction service.
Times are explicitly UTC, with ISO-8601 instants in Markdown and JSON. Transcript
labels use elapsed offsets from the conversation start (`HH:MM:SS`). Speaker
names use person links or an exact match to a public participant name; roster
order cannot identify a diarized speaker. An unmatched speaker string is not a
name: those turns use `Speaker N` when a speaker id exists, otherwise
`Speaker unknown`. The account holder without a verified name stays `Owner`.
Event times on the page and in Markdown use the same UTC labels. Unknown action
owners and due times appear as Unknown. An email-shaped `owner_name` is omitted.
# Daily Summaries
Source: https://docs.omi.me/doc/developer/api/daily-summaries
Retrieve daily summaries
Use a Developer API key with `conversations:read`. See [Quick Start](/doc/developer/api/overview) for authentication and rate limits.
| Method | Endpoint |
| - | - |
| `GET` | `/v1/dev/user/daily-summaries` |
| `GET` | `/v1/dev/user/daily-summaries/{summary_id}` |
# Folders
Source: https://docs.omi.me/doc/developer/api/folders
Retrieve user-defined folders for organizing conversations via the Developer API
## Endpoints
List all folders
***
## List Folders
Retrieve all folders for the authenticated user
This endpoint is strictly read-only. If the user has not yet had folders initialized, an empty array is returned. System folders (Work, Personal, Social) are created lazily through other paths — when the mobile app opens the conversations screen or when a new conversation is created. In normal usage, any user who can issue a Developer API key has already gone through one of those paths.
```bash theme={null}
curl -H "Authorization: Bearer $API_KEY" \
"https://api.omi.me/v1/dev/user/folders"
```
```python theme={null}
import requests
response = requests.get(
"https://api.omi.me/v1/dev/user/folders",
headers={"Authorization": f"Bearer {API_KEY}"}
)
folders = response.json()
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/folders",
{ headers: { Authorization: `Bearer ${API_KEY}` } }
);
const folders = await response.json();
```
```json theme={null}
[
{
"id": "folder_uuid_work",
"name": "Work",
"description": "Work, business, professional, and career-related conversations",
"color": "#3B82F6",
"icon": "💼",
"created_at": "2025-01-15T10:00:00Z",
"updated_at": "2025-01-20T13:50:00Z",
"order": 0,
"is_default": false,
"is_system": true,
"category_mapping": "work",
"conversation_count": 12
},
{
"id": "folder_uuid_personal",
"name": "Personal",
"description": "Personal life, family, health, hobbies, and self-improvement",
"color": "#10B981",
"icon": "👤",
"created_at": "2025-01-15T10:00:00Z",
"updated_at": "2025-01-18T09:00:00Z",
"order": 1,
"is_default": false,
"is_system": true,
"category_mapping": "personal",
"conversation_count": 7
},
{
"id": "folder_uuid_social",
"name": "Social",
"description": "Friends, social gatherings, entertainment, and casual conversations",
"color": "#8B5CF6",
"icon": "👥",
"created_at": "2025-01-15T10:00:00Z",
"updated_at": "2025-01-19T11:30:00Z",
"order": 2,
"is_default": false,
"is_system": true,
"category_mapping": "social",
"conversation_count": 4
}
]
```
| Field | Type | Description |
| - | - | - |
| `id` | string | Unique folder identifier |
| `name` | string | Display name of the folder |
| `description` | string \| null | Natural language description of what belongs in this folder |
| `color` | string | Hex color code for the folder (e.g. `#3B82F6`) |
| `icon` | string | Emoji or icon identifier |
| `created_at` | datetime | When the folder was created |
| `updated_at` | datetime | When the folder was last updated |
| `order` | integer | Display order index (lower = earlier) |
| `is_default` | boolean | Whether this is the default folder for unassigned conversations |
| `is_system` | boolean | `true` for category-based system folders (Work, Personal, Social) |
| `category_mapping` | string \| null | Maps to a conversation category for backwards compatibility |
| `conversation_count` | integer | Number of non-discarded conversations in this folder |
***
## Using Folder IDs to Filter Conversations
Once you have folder IDs, you can filter conversations by folder using the `folder_id` query parameter on the conversations endpoint:
```bash theme={null}
# Step 1: Get folder list
FOLDERS=$(curl -s -H "Authorization: Bearer $API_KEY" \
"https://api.omi.me/v1/dev/user/folders")
# Step 2: Extract the Work folder ID (using jq)
WORK_FOLDER_ID=$(echo "$FOLDERS" | jq -r '.[] | select(.name=="Work") | .id')
# Step 3: Fetch conversations in Work folder
curl -H "Authorization: Bearer $API_KEY" \
"https://api.omi.me/v1/dev/user/conversations?folder_id=$WORK_FOLDER_ID"
```
```python theme={null}
import requests
headers = {"Authorization": f"Bearer {API_KEY}"}
# Step 1: Get folder list
folders = requests.get(
"https://api.omi.me/v1/dev/user/folders",
headers=headers
).json()
# Step 2: Find the Work folder
work_folder = next((f for f in folders if f["name"] == "Work"), None)
# Step 3: Fetch conversations in Work folder
if work_folder:
conversations = requests.get(
"https://api.omi.me/v1/dev/user/conversations",
headers=headers,
params={"folder_id": work_folder["id"], "limit": 50}
).json()
print(f"Found {len(conversations)} work conversations")
```
```javascript theme={null}
const headers = { Authorization: `Bearer ${API_KEY}` };
// Step 1: Get folder list
const foldersRes = await fetch(
"https://api.omi.me/v1/dev/user/folders",
{ headers }
);
const folders = await foldersRes.json();
// Step 2: Find the Work folder
const workFolder = folders.find(f => f.name === "Work");
// Step 3: Fetch conversations in Work folder
if (workFolder) {
const convsRes = await fetch(
`https://api.omi.me/v1/dev/user/conversations?folder_id=${workFolder.id}&limit=50`,
{ headers }
);
const conversations = await convsRes.json();
console.log(`Found ${conversations.length} work conversations`);
}
```
You can also filter by `starred=true` to get only starred conversations, or combine both `folder_id` and `starred` to get starred conversations within a specific folder.
# Goals
Source: https://docs.omi.me/doc/developer/api/goals
Manage goals and progress history
Use a Developer API key with `goals:read` for reads and `goals:write` for mutations. See [Quick Start](/doc/developer/api/overview) for authentication and rate limits.
| Method | Endpoint |
| - | - |
| `GET` | `/v1/dev/user/goals` |
| `POST` | `/v1/dev/user/goals` |
| `DELETE` | `/v1/dev/user/goals/{goal_id}` |
| `GET` | `/v1/dev/user/goals/{goal_id}` |
| `PATCH` | `/v1/dev/user/goals/{goal_id}` |
| `GET` | `/v1/dev/user/goals/{goal_id}/history` |
| `PATCH` | `/v1/dev/user/goals/{goal_id}/progress` |
Two parameters are not obvious from the table:
* `PATCH /v1/dev/user/goals/{goal_id}/progress` takes `current_value` as a **required query parameter** (e.g. `?current_value=42.5`) — a JSON body alone returns `422`.
* `GET /v1/dev/user/goals/{goal_id}/history` takes an optional `days` query parameter (default `30`, max `365`).
# API Keys
Source: https://docs.omi.me/doc/developer/api/keys
Manage your Developer API keys
## Endpoints
List all keys
Create new key
Revoke key
Developer API keys are self-service: while signed in to Omi, open **Developer → API Keys** to create,
list, and revoke your keys. The lifecycle endpoints below use that signed-in Omi/Firebase session; an
`omi_dev_...` key cannot create, list, or revoke keys itself.
***
## Inspect the Calling Key
`GET /v1/dev/key` accepts any valid Developer API key, regardless of data scopes or memory readiness. It returns only that credential's metadata, never user content, the secret, or its hash. Use it to validate a key without reading memories.
```bash theme={null}
curl -H "Authorization: Bearer $OMI_API_KEY" \
"https://api.omi.me/v1/dev/key"
```
```json theme={null}
{
"id": "key_123abc",
"name": "Task reader",
"key_prefix": "omi_dev_abc123",
"scopes": ["action_items:read"],
"created_at": "2025-01-15T10:30:00Z",
"last_used_at": "2025-01-20T14:22:00Z"
}
```
`scopes` reports effective permissions: legacy keys with missing scopes receive the read-only scopes; an explicit empty list stays empty. `last_used_at` is approximate: authentication updates it on cache misses, roughly hourly, so it is not an activity log and null does not prove a key has never been used. Invalid keys return `401`; the endpoint has its own rate-limit budget (see [Quick Start](/doc/developer/api/overview#rate-limits)).
## List API Keys
Retrieve all your developer API keys
This endpoint requires your signed-in Omi/Firebase session. The secret key values are not returned (only the prefix is shown).
```bash theme={null}
curl -H "Authorization: Bearer $FIREBASE_ID_TOKEN" \
"https://api.omi.me/v1/dev/keys"
```
```python theme={null}
import requests
response = requests.get(
"https://api.omi.me/v1/dev/keys",
headers={"Authorization": f"Bearer {firebase_id_token}"}
)
keys = response.json()
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/keys",
{ headers: { Authorization: `Bearer ${firebaseIdToken}` } }
);
const keys = await response.json();
```
```json theme={null}
[
{
"id": "key_123abc",
"name": "My Analytics Dashboard",
"key_prefix": "omi_dev_abc123",
"created_at": "2025-01-15T10:30:00Z",
"last_used_at": "2025-01-20T14:22:00Z",
"scopes": ["conversations:read", "memories:read"]
},
{
"id": "key_456def",
"name": "Automation Script",
"key_prefix": "omi_dev_def456",
"created_at": "2025-01-18T09:00:00Z",
"last_used_at": null,
"scopes": ["memories:read"]
}
]
```
| Field | Type | Description |
| - | - | - |
| `id` | string | Unique key identifier (used for deletion) |
| `name` | string | Descriptive name you gave the key |
| `key_prefix` | string | First part of the key (for identification) |
| `created_at` | datetime | When the key was created |
| `last_used_at` | datetime | Approximate last authentication cache miss; may lag activity by about an hour (null if not recorded) |
| `scopes` | array | Permissions assigned to the key |
***
## Create API Key
Create a new developer API key
Create keys from the signed-in Omi web app. The REST endpoint is the same self-service flow and requires a
Firebase ID token, not an existing `omi_dev_...` key. The new secret is returned only once.
| Parameter | Type | Required | Description |
| - | - | - | - |
| `name` | string | **Yes** | Descriptive name for the key |
| `scopes` | array of strings | No | Permissions for the key. If omitted, the key receives the read-only scopes. |
```bash theme={null}
curl -X POST "https://api.omi.me/v1/dev/keys" \
-H "Authorization: Bearer $FIREBASE_ID_TOKEN" \
-H "Content-Type: application/json" \
-d '{"name": "My New Integration", "scopes": ["memories:read"]}'
```
```python theme={null}
import requests
response = requests.post(
"https://api.omi.me/v1/dev/keys",
headers={
"Authorization": f"Bearer {firebase_id_token}",
"Content-Type": "application/json"
},
json={"name": "My New Integration", "scopes": ["memories:read"]}
)
new_key = response.json()
print(f"Store this key securely: {new_key['key']}")
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/keys",
{
method: "POST",
headers: {
Authorization: `Bearer ${firebaseIdToken}`,
"Content-Type": "application/json"
},
body: JSON.stringify({ name: "My New Integration", scopes: ["memories:read"] })
}
);
const newKey = await response.json();
console.log(`Store this key securely: ${newKey.key}`);
```
```json theme={null}
{
"id": "key_789ghi",
"name": "My New Integration",
"key_prefix": "omi_dev_ghi789",
"key": "omi_dev_ghi789_full_secret_key_here",
"created_at": "2025-01-20T15:00:00Z",
"last_used_at": null,
"scopes": ["memories:read"]
}
```
The full API key (`key` field) is only returned once during creation. Store it securely immediately - you won't be able to see it again!
### Scopes
Choose the least-privileged scopes needed by your integration:
* Read: `conversations:read`, `memories:read`, `action_items:read`, `goals:read`
* Write: `conversations:write`, `memories:write`, `action_items:write`, `goals:write`
Memory scopes authorize the key, but they do not bypass the account-level Developer Memory API readiness
gate. See [Memories](/doc/developer/api/memories) for that availability contract.
***
## Revoke API Key
Revoke (delete) a specific API key permanently
| Parameter | Type | Description |
| - | - | - |
| `key_id` | string | The ID of the key to revoke |
```bash theme={null}
curl -X DELETE "https://api.omi.me/v1/dev/keys/key_789ghi" \
-H "Authorization: Bearer $FIREBASE_ID_TOKEN"
```
```python theme={null}
import requests
response = requests.delete(
f"https://api.omi.me/v1/dev/keys/key_789ghi",
headers={"Authorization": f"Bearer {firebase_id_token}"}
)
if response.status_code == 204:
print("Key revoked successfully")
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/keys/key_789ghi",
{
method: "DELETE",
headers: { Authorization: `Bearer ${firebaseIdToken}` }
}
);
if (response.status === 204) {
console.log("Key revoked successfully");
}
```
```
204 No Content
```
If a key is compromised, revoke it immediately and create a new one.
***
## Best Practices
Name keys after their purpose (e.g., "Analytics Dashboard", "Zapier Integration")
Create separate keys for different apps so you can revoke them independently
Check `last_used_at` to identify unused or potentially compromised keys
Periodically create new keys and revoke old ones
***
## Use Case: Key Management Script
```python theme={null}
import requests
FIREBASE_ID_TOKEN = "your_signed_in_firebase_id_token"
headers = {
"Authorization": f"Bearer {FIREBASE_ID_TOKEN}",
"Content-Type": "application/json"
}
def list_keys():
"""List all API keys"""
response = requests.get(
"https://api.omi.me/v1/dev/keys",
headers=headers
)
return response.json()
def create_key(name):
"""Create a new API key"""
response = requests.post(
"https://api.omi.me/v1/dev/keys",
headers=headers,
json={"name": name}
)
return response.json()
def revoke_key(key_id):
"""Revoke an API key"""
response = requests.delete(
f"https://api.omi.me/v1/dev/keys/{key_id}",
headers=headers
)
return response.status_code == 204
# List existing keys
print("Current API keys:")
for key in list_keys():
last_used = key.get("last_used_at", "Never")
print(f" - {key['name']} ({key['key_prefix']}...) - Last used: {last_used}")
# Create a new key
print("\nCreating new key...")
new_key = create_key("Test Integration")
print(f"Created: {new_key['name']}")
print(f"Key: {new_key['key']}") # Store this securely!
# Revoke a key (example)
# if revoke_key("key_123abc"):
# print("Key revoked successfully")
```
```javascript theme={null}
const FIREBASE_ID_TOKEN = process.env.FIREBASE_ID_TOKEN;
const headers = {
Authorization: `Bearer ${FIREBASE_ID_TOKEN}`,
"Content-Type": "application/json"
};
async function listKeys() {
const response = await fetch("https://api.omi.me/v1/dev/keys", { headers });
return response.json();
}
async function createKey(name) {
const response = await fetch("https://api.omi.me/v1/dev/keys", {
method: "POST",
headers,
body: JSON.stringify({ name })
});
return response.json();
}
async function revokeKey(keyId) {
const response = await fetch(`https://api.omi.me/v1/dev/keys/${keyId}`, {
method: "DELETE",
headers
});
return response.status === 204;
}
// List existing keys
console.log("Current API keys:");
const keys = await listKeys();
keys.forEach(key => {
const lastUsed = key.last_used_at || "Never";
console.log(` - ${key.name} (${key.key_prefix}...) - Last used: ${lastUsed}`);
});
// Create a new key
console.log("\nCreating new key...");
const newKey = await createKey("Test Integration");
console.log(`Created: ${newKey.name}`);
console.log(`Key: ${newKey.key}`); // Store this securely!
// Revoke a key (example)
// if (await revokeKey("key_123abc")) {
// console.log("Key revoked successfully");
// }
```
***
## Managing Keys in the App
You can also manage API keys directly in the Omi app:
Launch the Omi app on your device
Go to **Settings → Developer**
Under "Developer API Keys" you can:
* View all your keys
* Create new keys
* Delete existing keys
Keys created in the app and via the API are the same - you can manage them from either place.
# Memories
Source: https://docs.omi.me/doc/developer/api/memories
Create and retrieve memories via the Developer API
## Endpoints
Retrieve memories
Create memory
Update memory
Delete memory
Create up to 25
## Access and availability
Memory endpoints require a Developer API key with the corresponding scope: `memories:read` for reads and
`memories:write` for writes. Creating a key is self-service in **Developer → API Keys**, and creation
records the matching key grant when the key includes a memory scope.
Default memory reads also have a separate, server-owned account readiness gate. A valid, correctly scoped
key can therefore receive `403` while that account is not yet provisioned for Developer Memory API access.
Do not recreate the key in that case; retry after the account is enabled or contact Omi support if it remains
unavailable.
```json theme={null}
{
"detail": {
"enabled": false,
"code": "developer_memory_access_not_ready",
"message": "Developer Memory API access is not enabled for this account.",
"reason": "missing_rollout_state"
}
}
```
`code` is the stable programmatic signal. `reason` provides a diagnostic for the current server-side gate and
must not be treated as a key or scope error.
***
## Get Memories
Retrieve your memories with optional filtering
| Parameter | Type | Default | Description |
| - | - | - | - |
| `limit` | integer | 25 | Maximum number of memories to return |
| `offset` | integer | 0 | Number of memories to skip |
| `categories` | string | - | Comma-separated list (e.g., `"interesting,system"`) |
```bash theme={null}
curl -H "Authorization: Bearer $API_KEY" \
"https://api.omi.me/v1/dev/user/memories?limit=50&categories=interesting"
```
```python theme={null}
import requests
response = requests.get(
"https://api.omi.me/v1/dev/user/memories",
headers={"Authorization": f"Bearer {API_KEY}"},
params={"limit": 50, "categories": "interesting"}
)
memories = response.json()
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/memories?limit=50&categories=interesting",
{ headers: { Authorization: `Bearer ${API_KEY}` } }
);
const memories = await response.json();
```
```json theme={null}
[
{
"id": "mem_789ghi",
"content": "User is building an AI fitness app with personal trainer",
"category": "interesting",
"visibility": "private",
"tags": [],
"created_at": "2025-01-15T10:30:00Z",
"updated_at": "2025-01-15T10:30:00Z",
"manually_added": false,
"scoring": "01_999_1705315800",
"reviewed": false,
"user_review": null,
"edited": false
}
]
```
| Field | Type | Description |
| - | - | - |
| `id` | string | Unique identifier |
| `content` | string | The memory content |
| `category` | string | `interesting`, `system`, or `manual` |
| `visibility` | string | `public` or `private` |
| `tags` | array | List of tags |
| `created_at` | datetime | When created |
| `updated_at` | datetime | When last updated |
| `manually_added` | boolean | Added via API or app |
| `reviewed` | boolean | Has been reviewed |
| `edited` | boolean | Has been edited |
***
## Create Memory
Create a new memory
| Parameter | Type | Required | Description |
| - | - | - | - |
| `content` | string | **Yes** | The memory content (1-500 characters) |
| `category` | string | No | `interesting`, `system`, or `manual` (auto-categorizes if not provided) |
| `visibility` | string | No | `public` or `private` (default: `private`) |
| `tags` | array | No | List of tags |
A memory is identified by its text. Sending text the user already has returns that memory unchanged, with the same `id`, so a create is safe to retry without any extra header.
| Request | Result |
| - | - |
| Text the user does not have yet | A new memory |
| Text the user already has | `200` with the existing memory; nothing new is stored |
| Same text with different `category`, `visibility`, or `tags` | `200` with the existing memory as stored; the new fields are **not** applied |
| Same text after the memory was deleted | A new memory with a new `id` |
Matching ignores leading and trailing whitespace and is case-sensitive. To change a memory you already created, use [Update Memory](#update-memory) with its `id`.
```bash theme={null}
curl -X POST "https://api.omi.me/v1/dev/user/memories" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"content": "User prefers dark mode in all applications",
"category": "system",
"tags": ["preferences", "ui"]
}'
```
```python theme={null}
import requests
response = requests.post(
"https://api.omi.me/v1/dev/user/memories",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
},
json={
"content": "User prefers dark mode in all applications",
"category": "system",
"tags": ["preferences", "ui"]
}
)
memory = response.json()
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/memories",
{
method: "POST",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
content: "User prefers dark mode in all applications",
category: "system",
tags: ["preferences", "ui"]
})
}
);
const memory = await response.json();
```
```json theme={null}
{
"id": "mem_xyz789",
"content": "User prefers dark mode in all applications",
"category": "system",
"visibility": "private",
"tags": ["preferences", "ui"],
"created_at": "2025-12-06T10:35:00Z",
"updated_at": "2025-12-06T10:35:00Z",
"manually_added": true,
"scoring": "01_999_1733482500"
}
```
***
## Create Memories (Batch)
Create multiple memories in a single request (max 25)
| Parameter | Type | Required | Description |
| - | - | - | - |
| `memories` | array | **Yes** | List of memory objects (max 25) |
Each memory object accepts the same fields as the single create endpoint.
Items follow the same duplicate rule: text the user already has returns the existing memory at that position, so retrying a batch never stores anything twice. `created_count` is the number of memories returned, including existing ones.
```bash theme={null}
curl -X POST "https://api.omi.me/v1/dev/user/memories/batch" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"memories": [
{"content": "User prefers async communication over meetings", "category": "system"},
{"content": "User speaks fluent Japanese and French", "category": "interesting"}
]
}'
```
```python theme={null}
import requests
response = requests.post(
"https://api.omi.me/v1/dev/user/memories/batch",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
},
json={
"memories": [
{"content": "User prefers async communication over meetings", "category": "system"},
{"content": "User speaks fluent Japanese and French", "category": "interesting"}
]
}
)
result = response.json()
print(f"Created {result['created_count']} memories")
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/memories/batch",
{
method: "POST",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
memories: [
{ content: "User prefers async communication over meetings", category: "system" },
{ content: "User speaks fluent Japanese and French", category: "interesting" }
]
})
}
);
const result = await response.json();
console.log(`Created ${result.created_count} memories`);
```
```json theme={null}
{
"memories": [
{ "id": "mem_001", "content": "User prefers async communication over meetings", "category": "system", ... },
{ "id": "mem_002", "content": "User speaks fluent Japanese and French", "category": "interesting", ... }
],
"created_count": 2
}
```
***
## Update Memory
Update a memory's content or visibility
| Parameter | Type | Description |
| - | - | - |
| `memory_id` | string | The ID of the memory to update |
| Parameter | Type | Required | Description |
| - | - | - | - |
| `content` | string | No | New content (1-500 characters) |
| `visibility` | string | No | New visibility: `public` or `private` |
At least one field must be provided.
```bash theme={null}
curl -X PATCH "https://api.omi.me/v1/dev/user/memories/mem_xyz789" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"content": "User strongly prefers dark mode in all applications",
"visibility": "private"
}'
```
```python theme={null}
import requests
response = requests.patch(
"https://api.omi.me/v1/dev/user/memories/mem_xyz789",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
},
json={
"content": "User strongly prefers dark mode in all applications",
"visibility": "private"
}
)
memory = response.json()
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/memories/mem_xyz789",
{
method: "PATCH",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
content: "User strongly prefers dark mode in all applications",
visibility: "private"
})
}
);
const memory = await response.json();
```
```json theme={null}
{
"id": "mem_xyz789",
"content": "User strongly prefers dark mode in all applications",
"category": "system",
"visibility": "private",
"tags": ["preferences", "ui"],
"created_at": "2025-12-06T10:35:00Z",
"updated_at": "2025-12-25T14:00:00Z",
"manually_added": true,
"edited": true
}
```
***
## Delete Memory
Delete a memory permanently
| Parameter | Type | Description |
| - | - | - |
| `memory_id` | string | The ID of the memory to delete |
```bash theme={null}
curl -X DELETE "https://api.omi.me/v1/dev/user/memories/mem_xyz789" \
-H "Authorization: Bearer $API_KEY"
```
```python theme={null}
import requests
response = requests.delete(
"https://api.omi.me/v1/dev/user/memories/mem_xyz789",
headers={"Authorization": f"Bearer {API_KEY}"}
)
result = response.json()
if result["success"]:
print("Memory deleted successfully")
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/memories/mem_xyz789",
{
method: "DELETE",
headers: { Authorization: `Bearer ${API_KEY}` }
}
);
const result = await response.json();
if (result.success) {
console.log("Memory deleted successfully");
}
```
```json theme={null}
{
"success": true
}
```
This action is permanent. Deleted memories cannot be recovered.
***
## What Are Memories?
**Memories are timeless facts about the user** — preferences, relationships, personal details, and notable insights. They are NOT for notes, tasks, or time-sensitive information.
* **Good memories**: "User is vegetarian", "User's sister Sarah lives in Portland", "User prefers async communication"
* **Bad memories**: "Meeting on March 15th", "Call dentist tomorrow" (these are action items or conversations)
For notes, meeting summaries, or imported text content, use the [Conversations API](/doc/developer/api/conversations) instead — it automatically extracts memories and action items.
***
## Use Case: Store User Preferences
Import known facts and preferences about the user:
```python theme={null}
import requests
API_KEY = "omi_dev_your_api_key"
# Facts about the user from another system
user_facts = [
{"content": "User is vegetarian", "category": "system"},
{"content": "User's dog is named Luna", "category": "interesting"},
{"content": "User works at Acme Corp as lead engineer", "category": "system"},
{"content": "User ran a marathon in under 4 hours", "category": "interesting"}
]
# Create batch
response = requests.post(
"https://api.omi.me/v1/dev/user/memories/batch",
headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
json={"memories": user_facts}
)
print(f"Added {response.json()['created_count']} facts about the user")
```
```javascript theme={null}
const API_KEY = process.env.OMI_API_KEY;
// Facts about the user from another system
const userFacts = [
{ content: "User is vegetarian", category: "system" },
{ content: "User's dog is named Luna", category: "interesting" },
{ content: "User works at Acme Corp as lead engineer", category: "system" },
{ content: "User ran a marathon in under 4 hours", category: "interesting" }
];
// Create batch
const response = await fetch(
"https://api.omi.me/v1/dev/user/memories/batch",
{
method: "POST",
headers: {
Authorization: `Bearer ${API_KEY}`,
"Content-Type": "application/json"
},
body: JSON.stringify({ memories: userFacts })
}
);
const result = await response.json();
console.log(`Added ${result.created_count} facts about the user`);
```
***
## Memory Categories Explained
Useful context for the AI: preferences, work details, relationships, logistical info. Example: "User prefers dark mode"
Shareable, notable facts — things the user would excitedly tell someone at dinner. Example: "User climbed Mount Kilimanjaro"
# Quick Start
Source: https://docs.omi.me/doc/developer/api/overview
Access your Omi data programmatically with the Developer API. Build custom integrations, analytics dashboards, and automation workflows using your memories, conversations, and action items.
## Overview
The Omi Developer API provides programmatic access to your personal Omi data, allowing you to build custom applications and integrations. Use it to create analytics dashboards, export data to other services, build automation workflows, or contribute data back to your Omi account.
Read & write user memories
Access full transcripts
List conversation folders
Manage tasks & to-dos
Manage API access
***
## Quick Start
While signed in, open the Omi web app and navigate to **Developer → API Keys**. Create a key and choose
only the scopes your integration needs.
Copy the key immediately - you won't be able to see it again!
```bash theme={null}
curl -H "Authorization: Bearer omi_dev_your_key_here" \
https://api.omi.me/v1/dev/user/memories?limit=5
```
```python theme={null}
import requests
response = requests.get(
"https://api.omi.me/v1/dev/user/memories",
headers={"Authorization": "Bearer omi_dev_your_key_here"},
params={"limit": 5}
)
print(response.json())
```
```javascript theme={null}
const response = await fetch(
"https://api.omi.me/v1/dev/user/memories?limit=5",
{ headers: { Authorization: "Bearer omi_dev_your_key_here" } }
);
const memories = await response.json();
console.log(memories);
```
Check out the endpoint pages for detailed documentation on each resource.
Key creation is self-service. Memory endpoints additionally require server-side account readiness, so a valid
key with `memories:read` can return `403` with code `developer_memory_access_not_ready`. That response does
not mean the key is invalid or missing its scope; see [Memories](/doc/developer/api/memories).
***
## Base URL
```
https://api.omi.me/v1/dev
```
For self-hosted instances, replace with your backend URL.
***
## Endpoints at a Glance
| Method | Endpoint | Description |
| - | - | - |
| GET | `/v1/dev/user/memories` | Retrieve memories |
| POST | `/v1/dev/user/memories` | Create a memory |
| POST | `/v1/dev/user/memories/batch` | Create up to 25 memories |
| Method | Endpoint | Description |
| - | - | - |
| GET | `/v1/dev/user/action-items` | Retrieve action items |
| POST | `/v1/dev/user/action-items` | Create an action item |
| POST | `/v1/dev/user/action-items/batch` | Create up to 50 action items |
| Method | Endpoint | Description |
| - | - | - |
| GET | `/v1/dev/user/conversations` | Retrieve conversations |
| POST | `/v1/dev/user/conversations` | Create from text |
| POST | `/v1/dev/user/conversations/from-segments` | Create from transcript segments |
| Method | Endpoint | Description |
| - | - | - |
| GET | `/v1/dev/user/folders` | List all folders |
| Method | Endpoint | Description |
| - | - | - |
| GET | `/v1/dev/keys` | List all API keys |
| POST | `/v1/dev/keys` | Create new API key |
| DELETE | `/v1/dev/keys/{key_id}` | Revoke API key |
***
## Authentication
Resource endpoints under `/v1/dev/user/...` and credential introspection at `/v1/dev/key` require a Developer API key in the `Authorization` header:
```http theme={null}
Authorization: Bearer omi_dev_your_api_key_here
```
Key management (`GET`/`POST /v1/dev/keys` and `DELETE /v1/dev/keys/{key_id}`) instead requires a signed-in Firebase ID token. A developer key cannot manage other keys. See [API Keys](/doc/developer/api/keys).
MCP keys (`omi_mcp_...`) only authenticate MCP clients against `/v1/mcp`. They do not authenticate
REST Developer API calls. For memories, conversations, folders, and action items over HTTP, create a
Developer API key (`omi_dev_...`) and call endpoints under `/v1/dev/user/...` such as
`/v1/dev/user/memories`. Never commit API keys to version control or share them publicly.
***
## Rate Limits
Budgets are operation-specific, per credential context (user/app/key), rather than a blanket per-minute or daily allowance. Default budgets are:
| Operation | Requests per hour |
| - | - |
| Calling key metadata | 120 |
| Memory reads | 120 |
| Action item reads / writes (separate buckets) | 120 each |
| Goal reads / writes (separate buckets) | 120 each |
| Conversation reads, shared across list/detail/transcript | 60 |
| Conversation list / detail (additional separate buckets) | 60 each |
| Transcript fetch (additional bucket) | 25 |
| Conversation writes, shared | 25 |
| Conversation from-segments (additional bucket) | 30 |
| Ask | 25 |
| Memory writes | 120 |
| Memory batch writes | 15 |
Single-memory creation also has a 30 requests/minute burst limit. Overlapping budgets all apply; for example, transcript fetches consume the shared conversation-read, detail-read, and transcript buckets. Deployment configuration may adjust effective limits.
Each fixed window begins with its first counted request and resets when the bucket expires, rather than on a clock-hour boundary. Denied requests do not extend the window. There is no general 10,000/day developer quota.
Quota headers are sent on rate-limit denials; do not expect them on every successful response or expect an `X-RateLimit-Reset` timestamp. Use `Retry-After` (seconds until the denied bucket resets):
```http theme={null}
HTTP/1.1 429 Too Many Requests
Content-Type: application/json
X-RateLimit-Limit: 120
X-RateLimit-Remaining: 0
Retry-After: 1800
{"detail": "Rate limit exceeded. Try again in 1800s."}
```
Wait at least the indicated time before retrying. Retry writes only when you know they were rejected before execution; avoid repeating ambiguous writes after transport failures.
***
## Error Responses
| Code | Meaning |
| - | - |
| `200 OK` | Request succeeded |
| `204 No Content` | Request succeeded with no response body |
| `400 Bad Request` | Invalid request parameters |
| `401 Unauthorized` | Invalid or missing API key |
| `403 Forbidden` | Required scope, key grant, or server-side memory availability is missing |
| `404 Not Found` | Resource not found |
| `422 Unprocessable Entity` | Validation error |
| `429 Too Many Requests` | Rate limit exceeded |
| `500 Internal Server Error` | Server error |
```json theme={null}
{
"detail": {
"code": "developer_memory_access_not_ready",
"message": "Developer Memory API access is not enabled for this account."
}
}
```
***
## Security Best Practices
Use environment variables or secret management services
Generate new keys and revoke old ones periodically
Create separate keys for different applications
Check the "last used" timestamp in your key list
If a key is compromised, revoke it immediately from **Settings → Developer** in the Omi app.
***
## Developer API vs MCP
| Feature | Developer API | MCP |
| - | - | - |
| **Purpose** | Direct HTTP API access | AI assistant integration |
| **Access** | Read & write user data | Read/write with AI context |
| **Use Case** | Custom apps, dashboards, automation | Claude Desktop, AI agents |
| **Authentication** | Bearer token | Environment variable |
| **Best For** | Web apps, integrations, batch operations | AI-powered workflows |
If you see `404 Not Found` for legacy memory paths such as `/v1/memories` or `/v2/memories`, use the
Developer API path `/v1/dev/user/memories` instead. If you see `401 Unauthorized` while using an
`omi_mcp_...` key, create and use an `omi_dev_...` key for REST API requests.
* Programmatic access for custom applications
* Batch operations (multiple memories/action items)
* Integration with external services
* Custom automation workflows
* AI assistants like Claude to interact with your data
* Natural language queries and AI-powered insights
* Context-aware AI assistance
Learn more about the Model Context Protocol in the [MCP documentation](/doc/developer/mcp/introduction).
***
## Other APIs
Looking for different API capabilities? Omi offers several APIs for different use cases:
**For App Developers** - Create conversations and memories on behalf of users who have enabled your app
**For App Developers** - Receive real-time notifications when memories are created or transcripts are processed
**For App Developers** - Add custom tools that Omi's AI can invoke during conversations
**For App Developers** - Process raw audio bytes in real-time via WebSocket
The **Developer API** (this section) is for accessing your own personal data. The APIs above are for building apps that interact with other users' data (with their permission).
# Real-Time Audio Streaming
Source: https://docs.omi.me/doc/developer/apps/AudioStreaming
Stream raw audio bytes from your Omi device to any backend for custom speech processing, VAD, or audio analysis.
## Overview
Omi allows you to stream audio bytes from your DevKit directly to your backend or any external service. This enables custom audio processing like:
Use your own ASR models instead of Omi's default transcription
Implement custom VAD logic for specialized use cases
Extract features, spectrograms, or embeddings in real-time
Store raw audio for later processing or compliance
```mermaid theme={null}
flowchart LR
subgraph Device["Omi Device"]
M[Microphone] --> A[Audio Buffer]
end
A -->|Every X seconds| B[Omi Backend]
B -->|POST octet-stream| W[Your Webhook]
W --> P[Your Processing]
```
***
## Technical Specifications
| Specification | Value |
| - | - |
| **HTTP Method** | POST |
| **Content-Type** | `application/octet-stream` |
| **Audio Format** | Raw PCM16 (16-bit signed, little-endian) |
| **Bytes per Sample** | 2 |
| **Sample Rate** | 16,000 Hz (DevKit1 v1.0.4+, DevKit2) or 8,000 Hz (DevKit1 v1.0.2) |
| **Channels** | Mono (1 channel) |
The sample rate is passed as a query parameter so your endpoint can handle different device versions.
***
## Setup Guide
Create a webhook that accepts POST requests with binary audio data.
**Request format:**
```
POST /your-endpoint?sample_rate=16000&uid=user123
Content-Type: application/octet-stream
Body: [raw PCM16 audio bytes]
```
Your endpoint should:
* Accept `application/octet-stream` content type
* Read `sample_rate` and `uid` from query parameters
* Process the raw bytes (buffer, save, or analyze)
* Return 200 OK quickly to avoid timeouts
1. Open the Omi App
2. Go to **Settings** → **Developer Mode**
3. Scroll to **Realtime audio bytes**
4. Enter your webhook URL
5. Set the **Every x seconds** field (e.g., `10` for 10-second chunks)
Start speaking while wearing your Omi device. Audio bytes should arrive at your webhook at the configured interval.
Use [webhook.site](https://webhook.site) to verify data is arriving before implementing your processing logic.
***
## Working with Audio Bytes
### Converting to WAV
The received bytes are raw PCM16 audio. To create a playable WAV file, prepend a WAV header:
```python theme={null}
import struct
import wave
import io
def create_wav(audio_bytes: bytes, sample_rate: int) -> bytes:
"""Convert raw PCM16 bytes to WAV format."""
buffer = io.BytesIO()
with wave.open(buffer, 'wb') as wav_file:
wav_file.setnchannels(1) # Mono
wav_file.setsampwidth(2) # 16-bit = 2 bytes
wav_file.setframerate(sample_rate)
wav_file.writeframes(audio_bytes)
buffer.seek(0)
return buffer.read()
```
### Accumulating Chunks
If you need continuous audio (not chunked), accumulate bytes across requests:
```python theme={null}
from collections import defaultdict
# Store audio by session
audio_buffers = defaultdict(bytes)
@app.post("/audio")
async def receive_audio(request: Request, uid: str, sample_rate: int):
audio_bytes = await request.body()
# Accumulate audio for this user
audio_buffers[uid] += audio_bytes
# Process when you have enough audio (e.g., 60 seconds)
if len(audio_buffers[uid]) >= sample_rate * 2 * 60: # 2 bytes per sample
process_audio(audio_buffers[uid], sample_rate)
audio_buffers[uid] = bytes()
return {"status": "ok"}
```
***
## Example: Save to Google Cloud Storage
A complete example that saves audio files to Google Cloud Storage.
Follow the [Saving Audio Guide](/doc/developer/savingaudio) steps 1-5 to create a bucket with proper permissions.
Fork [github.com/mdmohsin7/omi-audio-streaming](https://github.com/mdmohsin7/omi-audio-streaming)
Clone the repository and deploy to your preferred cloud provider (GCP, AWS, DigitalOcean) or run locally with ngrok.
The repository includes a Dockerfile for easy deployment.
Configure these environment variables during deployment:
| Variable | Description |
| - | - |
| `GOOGLE_APPLICATION_CREDENTIALS_JSON` | GCP service account credentials (base64 encoded) |
| `GCS_BUCKET_NAME` | Your GCS bucket name |
Set the endpoint in Developer Settings → Realtime audio bytes:
```
https://your-deployment-url.com/audio
```
Audio files should now appear in your GCS bucket every X seconds (based on your configured interval).
***
## Processing Ideas
Feed audio to your own ASR models for specialized vocabulary or languages:
```python theme={null}
import whisper
model = whisper.load_model("base")
@app.post("/audio")
async def transcribe(request: Request, sample_rate: int):
audio_bytes = await request.body()
wav_data = create_wav(audio_bytes, sample_rate)
# Save temporarily and transcribe
with tempfile.NamedTemporaryFile(suffix=".wav") as f:
f.write(wav_data)
f.flush()
result = model.transcribe(f.name)
return {"text": result["text"]}
```
Detect speech vs. silence for custom endpointing:
```python theme={null}
import webrtcvad
vad = webrtcvad.Vad(3) # Aggressiveness 0-3
def detect_speech(audio_bytes: bytes, sample_rate: int) -> bool:
# webrtcvad needs 10, 20, or 30ms frames
frame_duration = 30 # ms
frame_size = int(sample_rate * frame_duration / 1000) * 2
speech_frames = 0
total_frames = 0
for i in range(0, len(audio_bytes), frame_size):
frame = audio_bytes[i:i + frame_size]
if len(frame) == frame_size:
if vad.is_speech(frame, sample_rate):
speech_frames += 1
total_frames += 1
return speech_frames / total_frames > 0.5 if total_frames else False
```
Extract embeddings for speaker identification or audio similarity:
```python theme={null}
from speechbrain.pretrained import EncoderClassifier
classifier = EncoderClassifier.from_hparams(
source="speechbrain/spkrec-ecapa-voxceleb"
)
def get_embedding(wav_path: str):
return classifier.encode_batch(
classifier.load_audio(wav_path)
)
```
Analyze emotional tone from audio features:
```python theme={null}
import librosa
import numpy as np
def extract_features(audio_bytes: bytes, sample_rate: int):
# Convert bytes to numpy array
audio = np.frombuffer(audio_bytes, dtype=np.int16).astype(np.float32)
audio = audio / 32768.0 # Normalize
# Extract features
mfccs = librosa.feature.mfcc(y=audio, sr=sample_rate, n_mfcc=13)
energy = librosa.feature.rms(y=audio)
return {
"mfcc_mean": mfccs.mean(axis=1).tolist(),
"energy_mean": float(energy.mean()),
}
```
***
## Best Practices
Return 200 OK immediately, process async. Slow responses may cause timeouts.
Network issues may cause gaps. Design your processing to handle incomplete audio.
Choose chunk interval based on your use case. Larger chunks = fewer requests but higher latency.
Audio streaming generates significant data. Monitor storage and bandwidth costs.
Audio data is sensitive. Ensure your endpoint is secured with HTTPS and implement appropriate access controls.
***
## Related Documentation
Overview of webhook-based integrations
Guide to storing audio in cloud storage
How Omi's built-in transcription works
Overview of all Omi app types
# Chat Tools
Source: https://docs.omi.me/doc/developer/apps/ChatTools
Add custom chat tools to your Omi app that extend Omi's capabilities in user conversations.
## Overview
**Chat Tools** allow your app to provide custom functions that become available in Omi chat when users install your app. These tools enable the AI assistant to interact with your service directly through natural language conversations.
Define tools that perform actions on external services
Users interact with tools through normal conversation
Tools become available when users install your app
**Example:** A Slack integration app might provide tools to:
* Send messages to Slack channels
* List available channels
* Search messages in Slack
When a user installs your app, these tools automatically become available in their Omi chat, allowing them to say things like "Send a message to #general saying hello" or "What Slack channels do I have?"
Chat Tools require your app to have the `external_integration` capability enabled. They are not a separate capability, but rather a feature that works with external integration apps.
***
## How Chat Tools Work
```mermaid theme={null}
sequenceDiagram
participant U as User
participant O as Omi AI
participant Y as Your Server
U->>O: "Send hello to #general"
O->>O: AI decides to use your tool
O->>Y: POST /api/send_message
Note over Y: {uid, app_id, tool_name, geolocation?, ...params}
Y->>Y: Process request
Y-->>O: {result: "Message sent!"}
O-->>U: "Done! Message sent to #general"
```
User makes a request in natural language
Omi's AI determines your tool is the best way to fulfill the request
POST request with `uid`, `app_id`, `tool_name`, plus extracted parameters
Execute the action and return a result
Omi displays your response to the user
***
## Part 1: Implementing Chat Tools
### Step 1: Create Tool Endpoints
Your backend server needs to expose endpoints that Omi will call when tools are invoked.
Your endpoints should:
* Accept POST requests with JSON payload (or GET with query parameters)
* Receive these standard fields:
* `uid`: User ID
* `app_id`: Your app ID
* `tool_name`: Name of the tool being called
* `geolocation` (optional): User's cached location, see [Geolocation](#geolocation) below
* Plus any tool-specific parameters
* Return JSON with either:
* `result`: Success message (string)
* `error`: Error message (string)
#### Example: Send Message Tool
```python theme={null}
from flask import Flask, request, jsonify
import requests
app = Flask(__name__)
@app.route('/api/send_message', methods=['POST'])
def send_message():
"""
Tool endpoint: Send a message to a Slack channel
Expected payload:
{
"uid": "user_id",
"app_id": "slack_app_id",
"tool_name": "send_slack_message",
"channel": "#general",
"message": "Hello from Omi!"
}
"""
data = request.json
# Validate required parameters
if not data:
return jsonify({'error': 'Missing request body'}), 400
uid = data.get('uid')
channel = data.get('channel')
message = data.get('message')
if not uid:
return jsonify({'error': 'Missing uid parameter'}), 400
if not channel:
return jsonify({'error': 'Missing required parameter: channel'}), 400
if not message:
return jsonify({'error': 'Missing required parameter: message'}), 400
# Get user's authentication token (from your database)
slack_token = get_user_token(uid)
if not slack_token:
return jsonify({
'error': 'Slack not connected. Please connect your Slack account.'
}), 401
# Call external API (e.g., Slack API)
slack_response = requests.post(
'https://slack.com/api/chat.postMessage',
headers={'Authorization': f'Bearer {slack_token}'},
json={'channel': channel, 'text': message}
)
if slack_response.json().get('ok'):
return jsonify({
'result': f'Successfully sent message to {channel}'
})
else:
return jsonify({
'error': f"Failed to send message: {slack_response.json().get('error')}"
}), 400
```
#### Example: Search Tool
```python theme={null}
@app.route('/api/search_messages', methods=['POST'])
def search_messages():
"""
Tool endpoint: Search for messages
Expected payload:
{
"uid": "user_id",
"app_id": "slack_app_id",
"tool_name": "search_slack_messages",
"query": "meeting notes",
"channel": "#general" # optional
}
"""
data = request.json
uid = data.get('uid')
query = data.get('query')
channel = data.get('channel')
if not uid:
return jsonify({'error': 'Missing uid parameter'}), 400
if not query:
return jsonify({'error': 'Missing required parameter: query'}), 400
# Get user's authentication token
slack_token = get_user_token(uid)
if not slack_token:
return jsonify({
'error': 'Slack not connected. Please connect your Slack account.'
}), 401
# Build search query
search_query = query
if channel:
search_query = f'in:{channel} {query}'
# Call external API
slack_response = requests.get(
'https://slack.com/api/search.messages',
headers={'Authorization': f'Bearer {slack_token}'},
params={'query': search_query}
)
if slack_response.json().get('ok'):
messages = slack_response.json().get('messages', {}).get('matches', [])
if not messages:
return jsonify({
'result': f'No messages found for "{query}"'
})
results = []
for msg in messages[:5]:
results.append(f"- {msg.get('text', '')[:100]} (in #{msg.get('channel', {}).get('name', 'unknown')})")
return jsonify({
'result': f'Found {len(messages)} messages:\n' + '\n'.join(results)
})
else:
return jsonify({
'error': f"Failed to search messages: {slack_response.json().get('error')}"
}), 400
```
### Step 2: Handle Authentication
If your tools require user authentication:
Save OAuth tokens in a secure database associated with `uid`
Check if user has connected their account before processing
If auth is missing, return a clear error message
```python theme={null}
def get_user_token(uid: str) -> Optional[str]:
"""Get user's authentication token from database"""
# In production, fetch from secure database
return user_tokens.get(uid)
@app.route('/api/send_message', methods=['POST'])
def send_message():
data = request.json
uid = data.get('uid')
# Check authentication
token = get_user_token(uid)
if not token:
return jsonify({
'error': 'Account not connected. Please connect your account in app settings.'
}), 401
# Proceed with tool execution
# ...
```
### Step 3: Error Handling Best Practices
Always return helpful error messages that guide users on how to fix the issue.
```python theme={null}
# Good error messages
return jsonify({'error': 'Slack not connected. Please connect your Slack account.'}), 401
return jsonify({'error': 'Missing required parameter: channel'}), 400
return jsonify({'error': 'Channel not found. Please check the channel name.'}), 404
# Avoid exposing sensitive information
# ❌ Bad: return jsonify({'error': f'Database connection failed: {db_password}'}), 500
# ✅ Good: return jsonify({'error': 'Service temporarily unavailable. Please try again later.'}), 500
```
***
## Part 2: Adding Chat Tools in Omi App Store
Chat tools are defined via a **manifest endpoint** hosted on your server. When you create or update your app in the Omi App Store, Omi will automatically fetch the tool definitions from your manifest URL.
Tool definitions live on your server
Modify your manifest and re-save the app to refresh
Define parameters with JSON Schema
Track changes in your codebase
### Step 1: Create the Manifest File
Create a JSON file at `/.well-known/omi-tools.json` on your server:
```json theme={null}
{
"tools": [
{
"name": "send_slack_message",
"description": "Send a message to a Slack channel. Use this when the user wants to send a message, post an update, or notify a channel in Slack.",
"endpoint": "/api/send_message",
"method": "POST",
"parameters": {
"properties": {
"channel": {
"type": "string",
"description": "The Slack channel to send to (e.g., '#general')"
},
"message": {
"type": "string",
"description": "The message text to send"
}
},
"required": ["channel", "message"]
},
"auth_required": true,
"status_message": "Sending message to Slack..."
},
{
"name": "search_slack_messages",
"description": "Search for messages in Slack. Use this when the user wants to find specific messages or look up past conversations.",
"endpoint": "/api/search_messages",
"method": "POST",
"parameters": {
"properties": {
"query": {
"type": "string",
"description": "Search query"
},
"channel": {
"type": "string",
"description": "Optional channel to search in"
}
},
"required": ["query"]
},
"auth_required": true,
"status_message": "Searching Slack..."
}
]
}
```
### Step 2: Host the Manifest
Make sure your manifest is accessible via HTTPS. Common locations:
* `https://your-app.com/.well-known/omi-tools.json` (recommended)
* `https://your-app.com/omi/manifest.json`
* `https://your-app.com/api/tools-manifest`
### Step 3: Add Manifest URL in Omi App Store
Navigate to Apps → Create App (or edit an existing app)
Choose External Integration capability
In the **Chat Tools Manifest URL** field, enter:
```
https://your-app.com/.well-known/omi-tools.json
```
Omi will automatically fetch your tool definitions
**Endpoint URLs in Manifest**: You can use relative paths (e.g., `/api/send_message`) in your manifest. Omi will automatically resolve them using your `App Home URL` as the base URL.
***
## Manifest Schema Reference
| Field | Type | Required | Description |
| - | - | - | - |
| `name` | string | Yes | Unique tool identifier (e.g., `send_slack_message`) |
| `description` | string | Yes | Detailed description for the AI to understand when/how to use the tool |
| `endpoint` | string | Yes | URL endpoint (can be relative or absolute) |
| `method` | string | No | HTTP method (default: `POST`) |
| `parameters` | object | No | JSON Schema defining tool parameters |
| `auth_required` | boolean | No | Whether user auth is required (default: `true`) |
| `status_message` | string | No | Message shown to user when tool is called |
### Parameters Schema
```json theme={null}
{
"parameters": {
"properties": {
"param_name": {
"type": "string | integer | boolean | array | object",
"description": "Description of what this parameter does"
}
},
"required": ["param_name"]
}
}
```
### Example: FastAPI Manifest Endpoint
```python theme={null}
from fastapi import FastAPI
app = FastAPI()
@app.get("/.well-known/omi-tools.json")
async def get_omi_tools_manifest():
return {
"tools": [
{
"name": "search_songs",
"description": "Search for songs on Spotify",
"endpoint": "/tools/search_songs",
"method": "POST",
"parameters": {
"properties": {
"query": {"type": "string", "description": "Search query"},
"limit": {"type": "integer", "description": "Max results"}
},
"required": ["query"]
},
"auth_required": True,
"status_message": "Searching Spotify..."
},
{
"name": "add_to_playlist",
"description": "Add a song to a playlist",
"endpoint": "/tools/add_to_playlist",
"method": "POST",
"parameters": {
"properties": {
"song_name": {"type": "string", "description": "Song name"},
"artist_name": {"type": "string", "description": "Artist name"},
"playlist_name": {"type": "string", "description": "Playlist name"}
},
"required": ["song_name"]
},
"auth_required": True,
"status_message": "Adding to playlist..."
}
],
"chat_messages": {
"enabled": True,
"target": "app",
"notify": False
}
}
```
***
## Sending Chat Messages Proactively
Your app can send messages directly to a user's chat without waiting for them to invoke a tool. This is useful for:
* Sending results from long-running background tasks
* Proactive notifications based on external events
* Asynchronous updates from your service
### Enabling Chat Messages
Add the `chat_messages` configuration to your manifest:
```json theme={null}
{
"tools": [...],
"chat_messages": {
"enabled": true,
"target": "app",
"notify": false
}
}
```
| Field | Type | Default | Description |
| - | - | - | - |
| `enabled` | boolean | `false` | Whether your app can send chat messages |
| `target` | string | `"app"` | Where messages appear: `"app"` (app-specific chat) or `"main"` (main chat) |
| `notify` | boolean | `false` | Whether to send a push notification with the message |
### Sending Messages via API
Once enabled, use the notification API to send messages:
```bash theme={null}
curl -X POST https://api.omi.me/v1/integrations/notification \
-H "Authorization: Bearer YOUR_APP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"uid": "user_firebase_id",
"aid": "your_app_id",
"message": "Your task has completed! Here are the results..."
}'
```
### API Request Fields
| Field | Type | Required | Description |
| - | - | - | - |
| `uid` | string | Yes | The user's Firebase UID |
| `aid` | string | Yes | Your app's ID |
| `message` | string | Yes | The message content to send |
### Example: Background Task Completion
```python theme={null}
import httpx
async def send_task_result(uid: str, result: str):
"""Send task completion message to user's chat."""
async with httpx.AsyncClient() as client:
response = await client.post(
"https://api.omi.me/v1/integrations/notification",
headers={
"Authorization": f"Bearer {APP_API_KEY}",
"Content-Type": "application/json",
},
json={
"uid": uid,
"aid": APP_ID,
"message": result,
},
)
return response.status_code == 200
```
### Rate Limits
Chat messages are rate-limited to prevent notification fatigue:
* **Maximum**: 10 messages per hour per app per user
* **Response**: HTTP 429 with `Retry-After` header when exceeded
Abuse of chat messages (spam, excessive notifications) may result in your app being removed from the store.
***
## Tool Definition Best Practices
Use descriptive, action-oriented names:
| Good | Bad |
| - | - |
| `send_slack_message` | `slack1` |
| `list_slack_channels` | `do_stuff` |
| `search_slack_messages` | `msg` |
Write detailed descriptions that help the AI understand:
* **When to use**: "Use this when the user wants to..."
* **Required parameters**: List what's needed
* **What it does**: Clear explanation of the action
**Example:**
```
Send a message to a Slack channel. Use this when the user wants to
send a message, post an update, or notify a channel in Slack.
Required parameters: channel (e.g., '#general') and message (the text to send).
```
Choose the appropriate method:
| Method | Use For |
| - | - |
| `POST` | Creating resources or sending data (most common) |
| `GET` | Retrieving data (parameters as query params) |
| `PUT/PATCH` | Updating resources |
| `DELETE` | Deleting resources |
Provide custom messages shown during tool execution:
* "Searching Slack..."
* "Sending message..."
* "Creating calendar event..."
If not provided, Omi generates a default based on the tool name.
***
## Request and Response Format
### Request Format
**POST Request:**
```json theme={null}
{
"uid": "user_firebase_id",
"app_id": "your_app_id",
"tool_name": "send_slack_message",
"channel": "#general",
"message": "Hello from Omi!",
"geolocation": {
"latitude": 30.2672,
"longitude": -97.7431,
"address": "123 Main St, Austin, TX 78701",
"google_place_id": "ChIJLwPMoJm1RIYRetVp1EtGm10",
"location_type": "approximate"
}
}
```
**GET Request:**
```
GET /api/search?uid=user_id&app_id=app_id&tool_name=search_slack_messages&query=meeting
```
### Geolocation
Tool call payloads include an optional `geolocation` field when the user has shared their location with Omi. Use it to make your tool location-aware (e.g., tag entries with where the user was, find nearby stores, localize search results).
| Field | Type | Description |
| - | - | - |
| `latitude` | float | User's latitude in decimal degrees |
| `longitude` | float | User's longitude in decimal degrees |
| `address` | string \| null | Reverse-geocoded human-readable address |
| `google_place_id` | string \| null | Google Maps place identifier |
| `location_type` | string \| null | How the location was determined (e.g., `approximate`) |
**Backward compatible**: The `geolocation` field is **only present when location data is available**. If the user has never granted location permission or their cached location has expired, the field is **omitted entirely** from the payload — it's not set to `null`. Always treat it as optional when parsing.
```python theme={null}
@app.route('/api/add_product_entry', methods=['POST'])
def add_product_entry():
data = request.json
geolocation = data.get('geolocation') # may be None
store_location = None
if geolocation:
store_location = {
'lat': geolocation['latitude'],
'lng': geolocation['longitude'],
'address': geolocation.get('address'),
}
# ... save entry with store_location ...
```
Location data is cached server-side for 30 minutes. If the user moves significantly, your tool will receive updated coordinates on the next call.
### Response Format
```json theme={null}
{
"result": "Successfully sent message to #general"
}
```
HTTP Status: `200`
```json theme={null}
{
"error": "Slack not connected. Please connect your Slack account."
}
```
HTTP Status: `400`, `401`, or `500`
***
## Best Practices
Keep responses concise but informative
* ✅ "Successfully sent message to #general"
* ❌ "OK"
Guide users on how to fix issues
* ✅ "Slack not connected. Please connect your account."
* ❌ "Error 401"
Store tokens encrypted and implement token refresh for long sessions
Protect your endpoints from abuse
### Rate Limiting Example
```python theme={null}
from functools import wraps
from flask import request, jsonify
import time
request_counts = {}
def rate_limit(max_requests=100, window=60):
def decorator(f):
@wraps(f)
def wrapper(*args, **kwargs):
uid = request.json.get('uid')
key = f"{uid}_{f.__name__}"
now = time.time()
if key in request_counts:
requests, first_request = request_counts[key]
if now - first_request < window:
if requests >= max_requests:
return jsonify({'error': 'Rate limit exceeded'}), 429
request_counts[key] = (requests + 1, first_request)
else:
request_counts[key] = (1, now)
else:
request_counts[key] = (1, now)
return f(*args, **kwargs)
return wrapper
return decorator
@app.route('/api/send_message', methods=['POST'])
@rate_limit(max_requests=100, window=60)
def send_message():
# ...
```
***
## Testing Your Tools
Use curl or Postman before adding to your app:
```bash theme={null}
curl -X POST https://your-server.com/api/send_message \
-H "Content-Type: application/json" \
-d '{
"uid": "test_user_id",
"app_id": "slack-integration",
"tool_name": "send_slack_message",
"channel": "#general",
"message": "Test message"
}'
```
Add your tool definitions in the Omi App Store
Install it in your test account
If auth is required, click the connect button in app settings
Try using the tools naturally:
* "Send a message to #general saying hello"
* "What Slack channels do I have?"
* "Search Slack for messages about the project"
***
## Troubleshooting
* Verify the app is installed and enabled (`enabled: true`)
* Check that `chat_tools` array is properly formatted
* Ensure endpoints are accessible and return proper responses
* Verify the app has `external_integration` capability
* Check endpoint logs for errors
* Verify authentication is working
* Ensure response format matches specification
* Test endpoints directly with curl/Postman
* Check that required parameters are being sent
* Improve tool description to be more specific about when to use it
* Add examples in the description
* Ensure tool name is descriptive
* Make sure the description clearly states when the tool should be used
***
## Complete Example
See a complete working example that demonstrates OAuth authentication, multiple chat tools, handling invocations, and error handling
***
## Related Documentation
Access your own personal Omi data programmatically
Create conversations and memories via REST API
Learn about external integrations
Set up authentication for your app
Publish your app to the store
Send push notifications to users
# Data Import APIs
Source: https://docs.omi.me/doc/developer/apps/Import
Programmatically create and read conversations and memories in users' Omi accounts using the Integration Import APIs.
Omi's Data Import APIs allow your app to programmatically interact with user data in the Omi ecosystem. Unlike triggers (which respond to events), these APIs let your app proactively create or read data.
Add new conversations to a user's account from external sources
Add facts and knowledge to the user's memory bank
Access and retrieve a user's conversation history
Access facts stored in the user's knowledge base
***
## Quick Reference
| Capability | Method | Endpoint |
| - | - | - |
| Create Conversation | `POST` | `/v2/integrations/{app_id}/user/conversations?uid={user_id}` |
| Create Memories | `POST` | `/v2/integrations/{app_id}/user/memories?uid={user_id}` |
| Read Conversations | `GET` | `/v2/integrations/{app_id}/conversations?uid={user_id}` |
| Read Memories | `GET` | `/v2/integrations/{app_id}/memories?uid={user_id}` |
**Base URL:** `https://api.omi.me`
***
## Prerequisites
Familiarize yourself with [Omi app development](/doc/developer/apps/Introduction) and [integration apps](/doc/developer/apps/Integrations)
In the Omi mobile app:
1. Go to **Apps** → **Create App**
2. Under **App Capabilities**, select **External Integration**
3. Select **Imports** and choose the specific capabilities your app needs
Go to your app's management page, find the **API Keys** section, and click **Create Key**
Copy and store your API key securely - you won't be able to see it again!
The user must explicitly enable your app in their Omi account before you can access their data
***
## Authentication
All API requests require authentication using your API key in the `Authorization` header:
```
Authorization: Bearer sk_your_api_key_here
```
The API key must belong to the app specified in the URL path (`app_id`).
***
## Create Conversation API
Create new conversations in a user's Omi account from external sources like phone calls, emails, or messages.
### Endpoint
```
POST https://api.omi.me/v2/integrations/{app_id}/user/conversations?uid={user_id}
```
### Request Body
```json theme={null}
{
"text": "The full text content of the conversation",
"started_at": "2024-07-21T22:34:43.384323+00:00",
"finished_at": "2024-07-21T22:35:43.384323+00:00",
"text_source": "audio_transcript",
"text_source_spec": "phone_call",
"language": "en",
"geolocation": {
"latitude": 37.7749,
"longitude": -122.4194
}
}
```
### Field Reference
| Field | Required | Default | Description |
| - | - | - | - |
| `text` | Yes | - | The full text content of the conversation |
| `started_at` | No | Current time | When the conversation started (ISO 8601) |
| `finished_at` | No | started\_at + 5min | When the conversation ended (ISO 8601) |
| `text_source` | No | `audio_transcript` | Source type: `audio_transcript`, `message`, or `other_text` |
| `text_source_spec` | No | - | Additional source detail (e.g., `phone_call`, `sms`, `email`) |
| `language` | No | `en` | Language code |
| `geolocation` | No | - | Object with `latitude` and `longitude` coordinates |
### Response
**Success (200 OK):**
```json theme={null}
{}
```
The conversation is created asynchronously in the user's account.
```python theme={null}
import requests
from datetime import datetime, timezone
response = requests.post(
f"https://api.omi.me/v2/integrations/{APP_ID}/user/conversations?uid={USER_ID}",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"started_at": datetime.now(timezone.utc).isoformat(),
"finished_at": datetime.now(timezone.utc).isoformat(),
"text": "John: Hi Sarah, how was your weekend?\n\nSarah: It was great! I went hiking.",
"text_source": "audio_transcript",
"text_source_spec": "phone_call"
}
)
```
```python theme={null}
response = requests.post(
f"https://api.omi.me/v2/integrations/{APP_ID}/user/conversations?uid={USER_ID}",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"text": "Mom: Don't forget to pick up milk\n\nMe: Will do!",
"text_source": "message",
"text_source_spec": "sms"
}
)
```
```python theme={null}
response = requests.post(
f"https://api.omi.me/v2/integrations/{APP_ID}/user/conversations?uid={USER_ID}",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"text": "From: alex@example.com\nSubject: Dinner Plans\n\nWould you like to grab dinner Friday at 7pm?",
"text_source": "other_text",
"text_source_spec": "email"
}
)
```
```python theme={null}
response = requests.post(
f"https://api.omi.me/v2/integrations/{APP_ID}/user/conversations?uid={USER_ID}",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"text": "Just finished my first marathon! 26.2 miles in 4:15. #running",
"text_source": "other_text",
"text_source_spec": "social_post",
"geolocation": {"latitude": 40.7128, "longitude": -74.0060}
}
)
```
***
## Create Memories API
Add facts and knowledge to a user's memory bank. You can either provide raw text (Omi will extract memories automatically) or provide explicit memory objects.
### Endpoint
```
POST https://api.omi.me/v2/integrations/{app_id}/user/memories?uid={user_id}
```
### Request Options
Provide raw text and let Omi automatically extract relevant memories:
```json theme={null}
{
"text": "Your flight to Paris has been confirmed for May 15th, 2024. Departure: JFK at 9:30 PM.",
"text_source": "email",
"text_source_spec": "travel_confirmation"
}
```
| Field | Required | Default | Description |
| - | - | - | - |
| `text` | Yes\* | - | Text content from which memories will be extracted |
| `text_source` | No | `other` | Source type: `email`, `social_post`, or `other` |
| `text_source_spec` | No | - | Additional source detail |
\*Either `text` or `memories` is required
Provide structured memory objects directly:
```json theme={null}
{
"memories": [
{
"content": "User is allergic to peanuts and shellfish",
"tags": ["health", "allergies", "important"]
},
{
"content": "User's mother's birthday is on August 15th",
"tags": ["family", "dates"]
}
],
"text_source": "other",
"text_source_spec": "user_profile"
}
```
| Field | Required | Description |
| - | - | - |
| `memories` | Yes\* | Array of memory objects |
| `memories[].content` | Yes | The memory content |
| `memories[].tags` | No | Array of categorization tags |
\*Either `text` or `memories` is required
### Response
**Success (200 OK):**
```json theme={null}
{}
```
Memories are extracted/saved asynchronously in the user's account.
```python theme={null}
response = requests.post(
f"https://api.omi.me/v2/integrations/{APP_ID}/user/memories?uid={USER_ID}",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"text": "Your flight to Paris confirmed for May 15th. Departure: JFK 9:30 PM. Hotel de Paris: May 16-20.",
"text_source": "email",
"text_source_spec": "travel_confirmation"
}
)
```
```python theme={null}
response = requests.post(
f"https://api.omi.me/v2/integrations/{APP_ID}/user/memories?uid={USER_ID}",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"memories": [
{"content": "User prefers window seats when flying", "tags": ["preferences", "travel"]},
{"content": "User is vegetarian", "tags": ["preferences", "food"]}
]
}
)
```
***
## Read Conversations API
Retrieve conversations from a user's Omi account.
### Endpoint
```
GET https://api.omi.me/v2/integrations/{app_id}/conversations?uid={user_id}
```
### Query Parameters
| Parameter | Default | Description |
| - | - | - |
| `limit` | 100 | Maximum conversations to return (max: 1000) |
| `offset` | 0 | Number to skip for pagination |
| `include_discarded` | false | Include discarded conversations |
| `statuses` | - | Filter by statuses (comma-separated) |
### Response
```json theme={null}
{
"conversations": [
{
"id": "conversation_id_1",
"created_at": "2024-03-15T12:00:00Z",
"started_at": "2024-03-15T12:00:00Z",
"finished_at": "2024-03-15T12:05:00Z",
"text": "Full conversation text content...",
"structured": {
"title": "Conversation Title",
"overview": "Brief overview of the conversation"
},
"transcript_segments": [
{"text": "Segment text...", "start_time": 0, "end_time": 10}
],
"geolocation": {"latitude": 37.7749, "longitude": -122.4194}
}
]
}
```
```python theme={null}
response = requests.get(
f"https://api.omi.me/v2/integrations/{APP_ID}/conversations?uid={USER_ID}",
headers={"Authorization": f"Bearer {API_KEY}"},
params={"limit": 50, "offset": 0}
)
conversations = response.json()["conversations"]
for conv in conversations:
print(f"{conv['structured']['title']} - {conv['created_at']}")
```
***
## Read Memories API
Retrieve memories (facts) from a user's knowledge base.
### Endpoint
```
GET https://api.omi.me/v2/integrations/{app_id}/memories?uid={user_id}
```
### Query Parameters
| Parameter | Default | Description |
| - | - | - |
| `limit` | 100 | Maximum memories to return (max: 1000) |
| `offset` | 0 | Number to skip for pagination |
### Response
```json theme={null}
{
"memories": [
{
"id": "memory_id_1",
"content": "User prefers vegetarian food options",
"created_at": "2024-03-15T12:00:00Z",
"tags": ["preferences", "food"]
}
]
}
```
```python theme={null}
response = requests.get(
f"https://api.omi.me/v2/integrations/{APP_ID}/memories?uid={USER_ID}",
headers={"Authorization": f"Bearer {API_KEY}"},
params={"limit": 100}
)
memories = response.json()["memories"]
for memory in memories:
print(f"{memory['content']} (Tags: {', '.join(memory.get('tags', []))})")
```
***
## Error Handling
All endpoints return consistent error codes:
| Status | Error | Description |
| - | - | - |
| 400 | Bad Request | Invalid request format or missing required fields |
| 401 | Unauthorized | Missing or invalid `Authorization` header |
| 403 | Forbidden | Invalid API key, app not enabled by user, or app lacks required capability |
| 404 | Not Found | App not found |
| 422 | Unprocessable Entity | Valid format but invalid content (e.g., empty text and memories) |
| 429 | Too Many Requests | Rate limit exceeded - implement exponential backoff |
Implement exponential backoff when you receive a 429 error. Start with a 1-second delay and double it on each retry.
***
## Best Practices
Always ensure you have explicit user consent before creating conversations or memories on their behalf
Store API keys securely using environment variables or secret management systems - never hardcode them
Implement robust error handling with retries and logging for production reliability
Only create meaningful content that provides genuine value to users - avoid spam
***
## Example Use Cases
Automatically create conversation entries from a journaling app, syncing daily reflections to Omi
Create structured conversation summaries after calendar meetings end, including action items
Generate conversations with health insights from fitness tracker data
Extract memories from educational content or research materials to build a personal knowledge base
Extract and store contact details as memories from emails and messages
Read conversations and memories to generate insights and visualizations about user patterns
***
## Testing & Troubleshooting
Create a separate API key for testing to avoid affecting production data
Clearly mark test conversations (e.g., prefix with "\[TEST]") so they're easy to identify and clean up
After creating content, open the Omi app to verify it appears correctly
For conversations, verify that title, overview, and action items are extracted correctly
**Common Issues:**
| Problem | Solution |
| - | - |
| Authentication errors | Verify your API key is correct and matches the app\_id in the URL |
| Permission errors | Ensure the user has enabled your app and your app has the required capability |
| Empty response | Check that you're providing `text` or `memories` (for memory creation) |
| Rate limiting | Implement exponential backoff starting at 1 second |
***
## Capabilities
The Import API supports creating and updating conversations, memories, and action items. See the [API Reference](/api-reference/introduction) for the full endpoint list.
***
## Related
Access your own personal Omi data programmatically
Learn about building integration apps with triggers
Guidelines for submitting your app to the Omi store
Add custom tools to Omi's AI chat
# Integration Apps
Source: https://docs.omi.me/doc/developer/apps/Integrations
Build webhook-based apps that connect Omi to external services. Process memories, real-time transcripts, raw audio, or daily summaries.
## What Are Integration Apps?
Integration apps allow Omi to interact with external services by sending data to your webhook endpoints. Unlike prompt-based apps, these require you to host a server.
Run code when a memory is created
Process live transcripts as they happen
Receive raw audio bytes for custom processing
Receive a daily recap of a user's conversations
```mermaid theme={null}
flowchart LR
subgraph Omi["Omi Backend"]
M[Memory Created]
T[Live Transcript]
A[Audio Stream]
D[Day Summary]
end
subgraph Your["Your Server"]
W[Webhook Endpoint]
P[Process Data]
E[External Services]
end
M -->|POST| W
T -->|POST| W
A -->|POST| W
D -->|POST| W
W --> P
P --> E
```
***
## Memory Creation Triggers
These apps are activated when Omi creates a new memory, allowing you to process or store the data externally.
This trigger also covers conversations uploaded from a Limitless Pendant. If an
initial short fragment is discarded, Omi sends the event once later uploaded
speech makes the conversation visible.
1. User completes a conversation
2. Omi processes and creates a memory
3. Your webhook receives the complete memory object
4. Your server processes and responds
The webhook receives the full conversation data including transcript, structured summary, action items, and metadata.
* **Slack Integration**: Post conversation summaries to team channels
* **CRM Updates**: Log customer interactions automatically
* **Project Management**: Create tasks in Notion, Asana, or Jira
* **Knowledge Base**: Build a searchable archive of conversations
* **Analytics**: Track conversation patterns and insights
**Running FastAPI locally (no cloud deployment):**
### Webhook Payload
Your endpoint receives a POST request with the memory object:
`POST /your-endpoint?uid=user123`
```json theme={null}
{
"id": "memory_abc123",
"created_at": "2024-07-22T23:59:45.910559+00:00",
"started_at": "2024-07-21T22:34:43.384323+00:00",
"finished_at": "2024-07-21T22:35:43.384323+00:00",
"transcript_segments": [
{
"text": "Let's discuss the project timeline.",
"speaker": "SPEAKER_00",
"speakerId": 0,
"speaker_name": "John",
"is_user": false,
"start": 10.0,
"end": 15.0
}
],
"structured": {
"title": "Project Timeline Discussion",
"overview": "Brief overview of the conversation...",
"emoji": "📅",
"category": "work",
"action_items": [
{
"description": "Send project proposal by Friday",
"completed": false
}
],
"events": []
},
"apps_response": [],
"discarded": false,
"folder_id": "folder_uuid",
"folder_name": "Work"
}
```
Check out the [Notion CRM Example](https://github.com/BasedHardware/Omi/blob/main/plugins/oauth/conversation_created.py) for a complete implementation.
***
## Real-Time Transcript Processors
Process conversation transcripts as they occur, enabling real-time analysis and actions.
This trigger requires an active live-listen session. Historical or
store-and-forward uploads, including Limitless Pendant WAL uploads, do not
replay transcript segments through the real-time webhook.
1. User starts speaking
2. Omi transcribes in real-time
3. Your webhook receives transcript segments as they're created
4. Your server processes and can trigger immediate actions
Segments arrive in multiple calls as the conversation unfolds, allowing for live reactions.
* **Live Coaching**: Provide real-time feedback during presentations
* **Fact-Checking**: Verify claims as they're made
* **Smart Home**: Trigger actions based on spoken commands
* **Sentiment Analysis**: Monitor emotional tone in real-time
* **Translation**: Live translation of conversations
### Webhook Payload
Your endpoint receives transcript segments with session context:
`POST /your-endpoint?session_id=abc123&uid=user123`
```json theme={null}
[
{
"text": "I think we should prioritize the mobile app.",
"speaker": "SPEAKER_00",
"speakerId": 0,
"is_user": false,
"start": 10.0,
"end": 15.0
},
{
"text": "Agreed, let's start with iOS.",
"speaker": "SPEAKER_01",
"speakerId": 1,
"is_user": true,
"start": 16.0,
"end": 18.0
}
]
```
### Implementation Tips
Real-time processors require careful design to avoid performance issues.
Track context across multiple calls using the session\_id parameter
Implement logic to prevent processing the same segments twice
Build complete conversation context by storing segments
Fail gracefully - don't block transcription with slow processing
***
## Real-Time Audio Bytes
Stream raw audio bytes from Omi directly to your endpoint for custom audio processing.
Audio-byte delivery requires an active live-listen session. Historical or
store-and-forward uploads do not replay their audio through this webhook.
1. User speaks into Omi device
2. Raw PCM audio is streamed to your endpoint
3. Your server processes the audio bytes directly
4. Handle as needed (custom STT, VAD, feature extraction, etc.)
Unlike transcript processors, you receive the actual audio data, not text.
* **Custom ASR**: Use your own speech recognition models
* **Voice Activity Detection**: Implement custom VAD logic
* **Audio Features**: Extract spectrograms, embeddings, or other features
* **Recording**: Store raw audio for later processing
* **Real-time Translation**: Feed audio to translation services
### Technical Details
| Setting | Value |
| - | - |
| Trigger Type | `audio_bytes` |
| HTTP Method | POST |
| Content-Type | `application/octet-stream` |
| Audio Format | PCM16 (16-bit little-endian) |
| Bytes per Sample | 2 |
**Request format:**
`POST /your-endpoint?sample_rate=16000&uid=user123`
Body contains raw PCM16 audio bytes.
To produce a playable WAV file, prepend a WAV header and concatenate the received chunks.
### Configuring Delivery Frequency
You can control how often audio is sent via the Omi app Developer Settings:
```
url,seconds
```
For example: `https://your-endpoint.com/audio,5` sends audio every 5 seconds.
For a complete implementation, see the [Audio Streaming Guide](/doc/developer/apps/AudioStreaming).
***
## Day Summary
Receive a structured daily recap of a user's conversations, delivered at most once per day at the user's configured notification hour. The webhook only fires on days where the user actually had recorded, transcribed conversations — see *Delivery conditions* below.
1. An hourly cron job runs at minute 0 of every UTC hour
2. For each user whose local time matches their configured notification hour, Omi generates a comprehensive daily summary using an LLM
3. Your webhook receives the summary
4. Your server can store, display, or forward it to external services
**Day selection logic:** If the user's local time is before noon (12:00), the summary covers the *previous* day's conversations; at noon or later it covers *today's* conversations.
**Timezone requirement (scheduled delivery only):** The hourly cron job selects recipients by matching configured timezones to the current UTC hour, so users without a timezone are skipped by the schedule. The manual "Generate Summary" trigger described in the testing section falls back to UTC day boundaries and works without a configured timezone.
**Delivery conditions (when the webhook does *not* fire):**
* The user has no conversations for the selected day
* All conversations for the day are either locked or have no transcribed speech
* The user has no FCM push token registered. The cron path filters out token-less users when picking recipients, and the manual "Generate Summary" endpoint returns HTTP 400 in that case — the daily summary pipeline currently treats push delivery as a hard prerequisite for firing the webhook
* A delivery for the same `(uid, date)` has already been started — Omi acquires an atomic Redis lock *before* the LLM call (TTL 2 hours), so any subsequent cron tick within that window is a no-op even if the earlier run hasn't finished or ended up skipping for one of the reasons above
Treat days without a webhook delivery as "no recap available" rather than a failure — receivers should not assume the webhook fires every day.
* **Personal CRM**: Log daily conversation highlights to a notes app or database
* **Team Digest**: Post a Slack summary of a user's day every evening
* **Journaling**: Auto-populate a daily journal with structured reflections
* **Analytics Dashboard**: Aggregate daily stats across users
* **Goal Tracking**: Surface action items and decisions for follow-up
### Webhook Payload
Your endpoint receives a POST request with the daily summary:
`POST /your-endpoint?uid=user123`
```json theme={null}
{
"uid": "user123",
"created_at": "2024-01-15T22:00:00.123456+00:00",
"summary": "{'id': '550e8400-...', 'date': '2024-01-15', 'headline': 'Productive day with three focused work sessions', ...}",
"summary_json": {
"id": "550e8400-e29b-41d4-a716-446655440000",
"date": "2024-01-15",
"headline": "Productive day with three focused work sessions",
"overview": "...",
"day_emoji": "💼",
"stats": {
"total_conversations": 3,
"total_duration_minutes": 87,
"action_items_count": 4
},
"highlights": [],
"action_items": [],
"unresolved_questions": [],
"decisions_made": [],
"knowledge_nuggets": [],
"locations": []
}
}
```
**Field reference:**
| Field | Type | Description |
| - | - | - |
| `uid` | string | User identifier (also in query param) |
| `created_at` | string (ISO 8601 with `+00:00` offset) | Webhook send time in UTC |
| `summary_json` | object | **Recommended.** The daily summary as a real JSON object (schema below). Use this for any new integration. |
| `summary` | string | **Legacy.** The same payload serialized via Python's `str(...)`, kept for backward compatibility. See the migration note below. |
**Use `summary_json` for new integrations.** The `summary` field stays on the wire so existing receivers don't break, but it's the legacy form: Python's `str(...)` on a `dict` produces a single-quoted Python literal (e.g. `"{'headline': '...'}"`) that **cannot** be parsed by `JSON.parse` and currently requires something Python-specific like `ast.literal_eval` to read. The new `summary_json` field carries the exact same payload as a real JSON object — every standard JSON parser handles it natively. Plan to migrate existing receivers to `summary_json`; the legacy `summary` field will be deprecated in a future release.
The `summary_json` object has this shape (also reflects what `summary` represents once parsed):
```json theme={null}
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"date": "2024-01-15",
"created_at": "2024-01-15T22:00:00.000000",
"headline": "Productive day with three focused work sessions",
"overview": "You had a productive day that included a project planning meeting, a deep-work coding session, and a team retrospective.",
"day_emoji": "💼",
"stats": {
"total_conversations": 3,
"total_duration_minutes": 87,
"action_items_count": 4
},
"highlights": [
{
"topic": "Q2 Roadmap",
"emoji": "🗺️",
"summary": "Locked in the Q2 feature priorities with the product team.",
"conversation_ids": ["conv_abc123"]
}
],
"action_items": [
{
"description": "Send project proposal to design team by Friday",
"priority": "high",
"completed": false,
"source_conversation_id": "conv_abc123"
}
],
"unresolved_questions": [
{
"question": "Which deployment pipeline should we adopt?",
"conversation_id": "conv_abc123"
}
],
"decisions_made": [
{
"decision": "Migrate analytics to BigQuery",
"conversation_id": "conv_abc123"
}
],
"knowledge_nuggets": [
{
"insight": "GitHub Actions reusable workflows can take typed inputs since 2023",
"conversation_id": "conv_abc123"
}
],
"locations": [
{
"name": "Home office",
"latitude": 37.7749,
"longitude": -122.4194,
"time": "09:30"
}
]
}
```
The top-level `created_at` is the webhook send timestamp in UTC, with an explicit `+00:00` offset (e.g. `2024-01-15T22:00:00.123456+00:00`). The `created_at` *inside* `summary_json` (and inside the legacy `summary` string) is the timestamp when the summary object was built by the LLM pipeline; it is also UTC but emitted as a naive ISO 8601 string with no offset suffix. The two will be very close in time but are technically distinct timestamps.
***
## Creating an Integration App
Decide which integration type(s) you need:
* **Memory Trigger**: Process completed conversations
* **Real-Time Transcript**: React to live speech
* **Audio Bytes**: Process raw audio
* **Day Summary**: Receive a daily recap of conversations
Create a webhook endpoint that can receive POST requests. For testing, use [webhook.site](https://webhook.site) or [webhook-test.com](https://webhook-test.com/).
Your endpoint should:
* Accept POST requests
* Parse JSON body (or binary for audio)
* Read `uid` from query parameters
* Return 200 OK quickly
Process the incoming data and integrate with external services.
**Example (Python/FastAPI):**
```python theme={null}
from fastapi import FastAPI, Request
app = FastAPI()
@app.post("/webhook")
async def handle_memory(request: Request, uid: str):
memory = await request.json()
# Process memory data
await send_to_slack(memory["structured"]["title"])
return {"status": "ok"}
```
Use Developer Mode to test without creating new memories (see testing section below)
Publish through the Omi mobile app
***
## Testing Your Integration
### Locally, with no device
`omi-cli` ships the same requests Omi sends, so a receiver can be built and
debugged before any hardware is involved.
```bash theme={null}
pipx install omi-cli
omi app events # what an app can receive
omi app serve # a receiver on http://127.0.0.1:8787
omi app send memory_created --to http://127.0.0.1:8787/webhook
```
`send` posts the sample body with the same query parameters, content type and
`Idempotency-Key` header the backend uses, prints the status and your response,
and exits non-zero when the response is not 2xx. With `--to` pointing at your
own endpoint, the day summary no longer waits for the delivery hour and the
real-time transcript no longer waits for someone to speak.
For `realtime_transcript` it also reports whether your reply would reach the
user, since Omi only shows a `message` longer than 5 characters, and only when
you answer 200.
Before submitting the app, check the URL you are submitting:
```bash theme={null}
omi app verify --to https://your-app.example.com/webhook
```
`verify` posts every event and reports what Omi's delivery client would do with
your endpoint, including the parts that are invisible from the outside: a
private or loopback host is refused before any request is sent, a redirect is
never followed, and a response slower than 30 seconds is a failed delivery.
These are real deliveries, so use a staging URL if your app writes somewhere on
receipt.
### End to end, from the app
Open Omi app → Settings → Enable Developer Mode → Developer Settings
* **Memory Triggers**: Enter URL in "Memory Creation Webhook"
* **Real-Time**: Enter URL in "Real-Time Transcript Webhook"
* **Audio Bytes**: Enter URL (optionally with `,seconds` suffix) in "Audio Bytes Webhook"
* **Day Summary**: Enter URL in "Day Summary Webhook"
Go to any memory → Tap 3-dot menu → Developer Tools → Trigger webhook with existing data
Start speaking - your endpoint receives updates immediately
The Day Summary webhook only fires on the scheduled cron path. The in-app "Generate Summary" trigger (Settings → Daily Summary → ⋮ menu) regenerates the summary on demand but does *not* currently POST to the developer webhook. The fastest way to validate a receiver is to enable the webhook, set its delivery hour to the next upcoming hour, and wait for the next cron tick.
Use [webhook.site](https://webhook.site) to see exactly what data Omi sends before writing any code.
***
## App Submission Fields
When submitting your integration app:
| Field | Required | Description |
| - | - | - |
| **Webhook URL** | Yes | Your POST endpoint for receiving data |
| **Setup Completed URL** | No | GET endpoint returning `{"is_setup_completed": boolean}` |
| **Auth URL** | No | URL for user authentication (uid appended automatically) |
| **Setup Instructions** | No | Text or link explaining how to configure your app |
### Setup Instructions Best Practices
Clear numbered instructions for configuration
Visual aids for complex setup steps
If required, explain how to connect accounts
Common issues and how to resolve them
When users open your setup links, Omi automatically appends a `uid` query parameter. Use this to associate credentials with specific users.
***
## Related Documentation
Access your own personal Omi data programmatically
Create conversations and memories via REST API
Detailed guide for processing raw audio bytes
Add custom tools users can invoke in chat
Add authentication flows to your apps
Send push notifications from your app
Create apps without hosting a server
Overview of all app types
# Building Apps for Omi
Source: https://docs.omi.me/doc/developer/apps/Introduction
Create apps that extend Omi's capabilities - from custom AI personalities to real-time integrations. Publish to the app store and earn from your creations.
## What Are Omi Apps?
Omi apps are modular extensions that augment the core functionality of the Omi platform. They can modify AI behavior, analyze conversations, and connect with external services.
```mermaid theme={null}
flowchart LR
subgraph Device["📱 Omi Device"]
Audio[Audio Stream]
end
subgraph Backend["🖥️ Omi Backend"]
Trans[Transcription]
Mem[Memory Creation]
Chat[Chat System]
end
subgraph Apps["🧩 Your App"]
RT[Real-time Webhook]
MT[Memory Trigger]
CT[Chat Tools]
PP[Prompts]
end
Audio --> Trans
Trans -->|Live transcript| RT
Trans --> Mem
Mem -->|On save| MT
Chat -->|Tool calls| CT
PP -->|Customize| Chat
```
Create custom AI personalities, alter conversation styles, or add specialized knowledge
Process live transcripts as conversations happen
Connect Omi to Slack, Notion, GitHub, and any external service
Publish to the app store and monetize your creations
***
## Prompt-Based Apps
Customize how Omi thinks and responds - **no server required!**
Alter Omi's conversational style and knowledge base. Create expert personas, custom assistants, or specialized advisors.
**Example:** Make Omi communicate like a fitness coach or financial advisor
Customize how conversations are analyzed and summarized. Extract specific information based on your criteria.
**Example:** Extract action items, key decisions, or meeting notes
Prompt-based apps are the easiest to create - just define your prompt and you're done!
***
## Integration Apps
Connect Omi to external services with webhooks and APIs.
Run your code when a memory is created. Perfect for syncing to external tools.
**Example:** Post conversation summaries to Slack or update a CRM
Process live audio transcripts as they happen. Enable real-time reactions.
**Example:** Trigger smart home actions or live coaching feedback
Add custom tools that users can invoke in Omi chat.
**Example:** "Send a message to #general in Slack" or "Create a GitHub issue"
Process raw audio bytes for custom STT, VAD, or audio analysis.
**Example:** Custom speech recognition or audio feature extraction
Integration apps require a server endpoint (webhook) to receive data from Omi.
***
## Quick Start: Build Your First App in 5 Minutes
Test the integration flow without writing any server code.
Go to [webhook.site](https://webhook.site) and copy your unique URL.
Get the app from the [App Store](https://apps.apple.com/us/app/friend-ai-wearable/id6502156163) or [Google Play](https://play.google.com/store/apps/details?id=com.friend.ios).
In the Omi app: **Explore → Create an App**
If you don't see "Explore", go to Settings and select "Connect Device" first.
1. Select a capability (e.g., "Real-time Transcript")
2. Paste your webhook.site URL
3. Install the app
Start speaking - watch real-time data appear on webhook.site!
***
## Example Apps
Learn from real apps built by the community.
Ask questions and get answers via notification in real-time.
[View Source Code →](https://github.com/BasedHardware/omi/blob/main/plugins/notifications/hey_omi.py)
Proactive AI mentor that provides guidance during conversations.
[View Source Code →](https://github.com/BasedHardware/omi/blob/main/plugins/basic/mentor.py)
Post conversation summaries and memories to Slack channels.
[View Source Code →](https://github.com/BasedHardware/omi-slack-app)
Connect Omi to 5000+ apps through Zapier automations.
[View Source Code →](https://github.com/BasedHardware/omi/tree/main/plugins/zapier)
Create issues and manage PRs from your conversations.
[View Source Code →](https://github.com/aaravgarg/omi-github-app)
Integrate Omi with RingCentral communications.
[View Source Code →](https://github.com/BasedHardware/omi-ringcentral-app)
***
## Next Steps
Create apps that customize AI personality and memory processing - no server required
Build webhooks for memory triggers and real-time transcript processing
Add custom tools that users can invoke in Omi conversations
Process raw audio bytes for custom STT or audio analysis
Publish your app to the Omi app store
***
## Video Tutorials
Learn how to build apps that trigger when Omi creates a new memory.
Build apps that process live transcripts as conversations happen.
Set up a local development server without cloud deployment.
***
## Related Documentation
Add authentication flows to your integration apps
Send push notifications to users from your app
Create memories and trigger actions from external sources
Understand how the chat system works under the hood
# Sending Notifications
Source: https://docs.omi.me/doc/developer/apps/Notifications
Send push notifications to Omi users from your apps - both direct text notifications and AI-generated proactive notifications.
## Overview
Omi supports two types of notifications you can send to users:
Send immediate, exact-text messages to specific users
AI-generated contextual messages based on real-time conversations
### Direct Flow
```mermaid theme={null}
flowchart LR
A[Your Server] -->|POST /notification| O[Omi API]
O -->|Push| U[User]
```
### Proactive Flow
```mermaid theme={null}
flowchart LR
T[Real-time Transcript] -->|Webhook| W[Your Server]
W -->|Prompt Template| O[Omi API]
O -->|AI Generated| U[User]
```
***
## Direct Notifications
Send immediate messages to specific users. Perfect for alerts, updates, and responses to user actions.
* Task reminders and event notifications
* Service updates or changes
* Real-time alerts and warnings
* Responses to user queries
* New feature announcements
### API Reference
| Setting | Value |
| - | - |
| Method | `POST` |
| URL | `https://api.omi.me/v2/integrations/{app_id}/notification` |
| Auth | `Bearer ` |
**Query Parameters:**
| Parameter | Required | Description |
| - | - | - |
| `uid` | Yes | Target user's Omi ID |
| `message` | Yes | Notification text |
### Implementation
You'll need your App ID and App Secret from the Omi developer portal.
```bash theme={null}
export OMI_APP_ID="your_app_id"
export OMI_APP_SECRET="your_app_secret"
```
Store credentials securely - never commit them to version control.
Make a POST request with the user ID and message:
```bash theme={null}
curl -X POST "https://api.omi.me/v2/integrations/${OMI_APP_ID}/notification?uid=USER_ID&message=Hello!" \
-H "Authorization: Bearer ${OMI_APP_SECRET}" \
-H "Content-Type: application/json"
```
Check the response status code to confirm delivery.
### Code Examples
```javascript theme={null}
const https = require('https');
async function sendNotification(userId, message) {
const appId = process.env.OMI_APP_ID;
const appSecret = process.env.OMI_APP_SECRET;
const url = new URL(`https://api.omi.me/v2/integrations/${appId}/notification`);
url.searchParams.set('uid', userId);
url.searchParams.set('message', message);
const response = await fetch(url, {
method: 'POST',
headers: {
'Authorization': `Bearer ${appSecret}`,
'Content-Type': 'application/json'
}
});
if (!response.ok) {
throw new Error(`API Error: ${response.status}`);
}
return response.json();
}
// Usage
sendNotification('user123', 'Your task is due in 1 hour!')
.then(console.log)
.catch(console.error);
```
```python theme={null}
import os
import requests
def send_notification(user_id: str, message: str) -> dict:
app_id = os.environ['OMI_APP_ID']
app_secret = os.environ['OMI_APP_SECRET']
response = requests.post(
f'https://api.omi.me/v2/integrations/{app_id}/notification',
params={'uid': user_id, 'message': message},
headers={'Authorization': f'Bearer {app_secret}'}
)
response.raise_for_status()
return response.json()
# Usage
send_notification('user123', 'Your task is due in 1 hour!')
```
***
## Proactive Notifications
Context-aware, AI-generated notifications that leverage user data during real-time conversations. You provide a prompt template, Omi generates the personalized message.
Unlike direct notifications, proactive notifications require an app with specific capabilities and user-granted permissions.
* Real-time mentoring advice during meetings
* Context-aware suggestions based on conversation topics
* Personalized insights based on user's goals
* Intelligent responses to specific phrases
* Meeting action item reminders
1. User installs your app and grants permissions
2. During conversations, Omi sends transcripts to your webhook
3. Your webhook analyzes the transcript and returns a prompt template
4. Omi fills in user context (name, facts, etc.) and generates the notification
5. User receives a personalized, contextual notification
### App Configuration
Your app needs these capabilities:
```json theme={null}
{
"name": "Your App Name",
"capabilities": ["external_integration", "proactive_notification"],
"external_integration": {
"triggers_on": "transcript_processed",
"webhook_url": "https://yourapp.com/webhook"
},
"proactive_notification": {
"scopes": ["user_name", "user_facts", "user_context", "user_chat"]
}
}
```
### Available Scopes
| Scope | Template Variable | Description |
| - | - | - |
| `user_name` | `{{user_name}}` | User's display name |
| `user_facts` | `{{user_facts}}` | Known facts about the user |
| `user_context` | `{{user_context}}` | Relevant conversation context |
| `user_chat` | `{{user_chat}}` | Recent chat history with your app |
### Webhook Implementation
Your webhook receives real-time transcripts and returns notification prompts:
**Incoming Request:**
```
POST /your-webhook?session_id=abc123&uid=user123
```
**Request Body:**
```json theme={null}
{
"session_id": "abc123",
"segments": [
{"text": "What do you think about this approach?", "speaker": "SPEAKER_01", "is_user": false}
]
}
```
**Response (when you want to send a notification):**
```json theme={null}
{
"session_id": "abc123",
"notification": {
"prompt": "You are mentoring {{user_name}}. Based on {{user_facts}} and {{user_context}}, provide brief actionable advice in 50 words.",
"params": ["user_name", "user_facts", "user_context"]
}
}
```
**Response (when no notification needed):**
```json theme={null}
{
"session_id": "abc123"
}
```
### Complete Example
```python theme={null}
from fastapi import FastAPI
from pydantic import BaseModel
from typing import List, Optional
app = FastAPI()
class TranscriptSegment(BaseModel):
text: str
speaker: str
is_user: bool
class WebhookRequest(BaseModel):
session_id: str
segments: List[TranscriptSegment]
@app.post("/webhook")
def handle_transcript(data: WebhookRequest, uid: str, session_id: str):
# Combine all segments into transcript
transcript = " ".join([s.text for s in data.segments]).lower()
# Detect trigger phrases
if "what do you think" in transcript or "any advice" in transcript:
return {
"session_id": data.session_id,
"notification": {
"prompt": """You are a helpful mentor for {{user_name}}.
Based on their goals ({{user_facts}}) and the current discussion ({{user_context}}),
provide specific, actionable advice in under 100 words.""",
"params": ["user_name", "user_facts", "user_context"]
}
}
# No notification needed
return {"session_id": data.session_id}
```
```javascript theme={null}
const express = require('express');
const app = express();
app.use(express.json());
app.post('/webhook', (req, res) => {
const { session_id, segments } = req.body;
// Combine segments into transcript
const transcript = segments
.map(s => s.text)
.join(' ')
.toLowerCase();
// Detect trigger phrases
if (transcript.includes('what do you think') || transcript.includes('any advice')) {
return res.json({
session_id,
notification: {
prompt: `You are a helpful mentor for {{user_name}}.
Based on their goals ({{user_facts}}) and the discussion ({{user_context}}),
provide specific advice in under 100 words.`,
params: ['user_name', 'user_facts', 'user_context']
}
});
}
// No notification needed
res.json({ session_id });
});
```
### Testing
Open Omi app → Settings → Developer Mode
Install your app in Developer Mode and grant requested scopes
Configure your webhook URL in Developer Settings
Start a conversation and say trigger phrases (e.g., "what do you think?")
Notification should appear within 30 seconds
**Rate Limit:** 1 proactive notification per user per app every 30 seconds.
***
## Best Practices
* Implement delays between notifications
* Avoid duplicate messages
* Group related notifications
* Keep messages concise and clear
* Include relevant context
* Use appropriate urgency levels
* Implement retry with exponential backoff
* Log errors for debugging
* Monitor delivery status
* Store credentials securely
* Validate user IDs before sending
* Use HTTPS for all endpoints
***
## Troubleshooting
### Error Codes
| Code | Meaning | Solution |
| - | - | - |
| 401 | Unauthorized | Verify API credentials and Bearer token format |
| 404 | User not found | Check that user ID exists and has your app installed |
| 429 | Rate limited | Implement rate limiting with exponential backoff |
| 500 | Server error | Retry with backoff, check Omi status page |
**Check these:**
* API credentials are correct
* User ID is valid
* Bearer token is properly formatted
* Request includes required headers
**Check these:**
* App has `proactive_notification` capability enabled
* User has installed your app
* User has granted requested scopes
* Webhook is returning proper response format
* Prompt is at least 5 characters
**Solutions:**
* Request additional scopes in app configuration
* Include more template variables in your prompt
* Make prompts more specific about using the context
***
## Related Documentation
Build webhook-based integrations
Add authentication to your apps
Add custom tools to Omi chat
Publish your app to the store
# OAuth Authentication
Source: https://docs.omi.me/doc/developer/apps/Oauth
Integrate your application with Omi using OAuth 2.0 to securely access user data with explicit consent.
## Overview
Omi uses an OAuth 2.0-like flow to allow third-party applications to access user data with explicit consent. This ensures users maintain control over their data and what applications can access.
```mermaid theme={null}
sequenceDiagram
participant App as Your App
participant User as User Browser
participant Omi as Omi Auth Server
participant Firebase as Firebase Auth
App->>User: Redirect to /v1/oauth/authorize
User->>Omi: Visit authorization page
Omi->>Firebase: User logs in (Google/Apple)
Firebase-->>Omi: Firebase ID token
Omi->>Omi: Exchange token for uid
Omi->>User: Redirect to App Home URL
User->>App: ?uid=USER_ID&state=STATE
App->>App: Store uid for API calls
```
Users explicitly approve what data your app can access
Firebase authentication ensures secure identity verification
***
## Prerequisites
Before implementing OAuth, ensure you have:
Your app must be registered with Omi and have an **App ID**
Set your **App Home URL** in your app's settings - this is where users are redirected after authorization
Your App Home URL must use HTTPS
***
## OAuth Flow
Your app redirects the user to Omi's authorization endpoint:
```
https://api.omi.me/v1/oauth/authorize?app_id=YOUR_APP_ID&state=YOUR_STATE
```
| Parameter | Required | Description |
| - | - | - |
| `app_id` | Yes | Your application's unique ID |
| `state` | No | Opaque value for CSRF protection and maintaining state |
Always use the `state` parameter to prevent CSRF attacks. Generate a random string and verify it when the user returns.
On the Omi authorization page, the user:
1. Logs in with their Firebase credentials (Google or Apple)
2. Reviews the permissions your app is requesting
3. Grants or denies access
**Example permissions shown to users:**
* "Engage in chat conversations with Omi"
* "Access and manage your conversations"
* "Process audio data in real-time"
* "Create new conversations on your behalf"
* "Access and read your stored memories"
After user approval, Omi's authorization page handles the token exchange internally:
```http theme={null}
POST https://api.omi.me/v1/oauth/token
Content-Type: application/x-www-form-urlencoded
firebase_id_token=FIREBASE_TOKEN&app_id=YOUR_APP_ID&state=YOUR_STATE
```
**Response:**
```json theme={null}
{
"uid": "USER_UNIQUE_ID",
"redirect_url": "YOUR_APP_HOME_URL",
"state": "YOUR_STATE_IF_PROVIDED"
}
```
This step is handled automatically by Omi's authorization page - you don't need to implement this yourself.
The user's browser is redirected to your App Home URL with query parameters:
```
https://your-app.com/callback?uid=USER_UNIQUE_ID&state=YOUR_STATE
```
Your app should:
1. Validate the `state` parameter matches what you sent
2. Store the `uid` for making API calls on behalf of the user
***
## Handling the Callback
Example implementation for handling the OAuth callback:
```python theme={null}
from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import RedirectResponse
import secrets
app = FastAPI()
# Store state tokens (use Redis in production)
pending_states = {}
@app.get("/start-oauth")
async def start_oauth():
# Generate CSRF token
state = secrets.token_urlsafe(32)
pending_states[state] = True
# Redirect to Omi
return RedirectResponse(
f"https://api.omi.me/v1/oauth/authorize"
f"?app_id=YOUR_APP_ID&state={state}"
)
@app.get("/callback")
async def oauth_callback(uid: str, state: str):
# Validate state
if state not in pending_states:
raise HTTPException(400, "Invalid state parameter")
del pending_states[state]
# Store uid for this user's session
# Now you can make API calls with this uid
return {"message": f"Successfully connected! User ID: {uid}"}
```
```javascript theme={null}
const express = require('express');
const crypto = require('crypto');
const app = express();
const pendingStates = new Map();
app.get('/start-oauth', (req, res) => {
// Generate CSRF token
const state = crypto.randomBytes(32).toString('hex');
pendingStates.set(state, true);
// Redirect to Omi
res.redirect(
`https://api.omi.me/v1/oauth/authorize` +
`?app_id=YOUR_APP_ID&state=${state}`
);
});
app.get('/callback', (req, res) => {
const { uid, state } = req.query;
// Validate state
if (!pendingStates.has(state)) {
return res.status(400).send('Invalid state parameter');
}
pendingStates.delete(state);
// Store uid for this user's session
res.send(`Successfully connected! User ID: ${uid}`);
});
```
***
## Automatic App Enablement
When a user completes the OAuth flow, Omi automatically attempts to enable your app for them.
Omi performs several checks before enabling:
| Check | Description |
| - | - |
| **Privacy** | If app is private, only owner/testers can enable |
| **Setup Completion** | If `setup_completed_url` is configured, it must return `true` |
| **Payment** | If app is paid, user must have active subscription |
If any check fails, the OAuth flow halts and shows an error to the user.
If your app requires setup (e.g., connecting to external services), configure a `setup_completed_url`:
**Request from Omi:**
```
GET https://your-app.com/setup-status?uid=USER_ID
```
**Expected Response:**
```json theme={null}
{
"is_setup_completed": true
}
```
Return `false` if the user hasn't completed setup yet. Omi will show an appropriate message.
When a user successfully enables your public app through OAuth, the public install count is incremented (if applicable).
***
## App Configuration
Unique identifier for your app, provided when you register
Where users are redirected after authorization (must be HTTPS)
Configure these settings in the Omi developer portal or during app submission:
| Field | Location | Description |
| - | - | - |
| App ID | Auto-generated | Your unique application identifier |
| App Home URL | `external_integration.app_home_url` | Callback URL after OAuth (HTTPS required) |
| Setup Completed URL | `external_integration.setup_completed_url` | Optional endpoint to verify user setup |
***
## Security Best Practices
Generate a cryptographically random `state` parameter and validate it on callback to prevent CSRF attacks
Ensure the `uid` came from a legitimate OAuth flow, not a forged request
Store user `uid` values securely, treating them as sensitive credentials
Always use HTTPS for your App Home URL and all API communications
Never expose your App ID in client-side code where it could be extracted. While the App ID itself isn't secret, it should be treated as sensitive for your integration's integrity.
***
## Troubleshooting
**Possible causes:**
* App Home URL not configured or incorrect
* App Home URL doesn't use HTTPS
* User denied permissions
**Solution:** Verify your App Home URL in the developer portal
**Possible causes:**
* State expired before user completed flow
* State not properly stored server-side
* CSRF attack attempt
**Solution:** Implement proper session/state management with reasonable expiration
**Possible causes:**
* `setup_completed_url` returning `false` or error
* Endpoint not accessible from Omi servers
* JSON response malformed
**Solution:** Test your endpoint directly and check server logs
***
## Related Documentation
Build webhook-based integrations
Add custom tools to Omi chat
Send push notifications to users
Publish your app to the store
# Open Source Your App
Source: https://docs.omi.me/doc/developer/apps/OpenSource
Contribute your Omi app to the open-source repo. Share your code with the community, get visibility, and help other developers learn from your work.
## Overview
Beyond publishing to the Omi app store, you can open-source your app by contributing it to the [BasedHardware/omi](https://github.com/BasedHardware/omi) repository. This gives your app maximum visibility, lets other developers learn from and build on your work, and contributes to the Omi ecosystem.
```mermaid theme={null}
flowchart LR
B[Build Your App] --> T[Test Locally]
T --> F[Fork the Repo]
F --> A[Add Your App]
A --> PR[Open a PR]
PR --> R[Review & Merge]
R --> C[🎉 Community Access]
```
Your app is discoverable by thousands of Omi developers
Significant contributions earn free DevKit hardware
***
## Repository Structure
Open-source apps live in the `plugins/` directory:
```
omi/
├── plugins/
│ ├── your-app-name/
│ │ ├── README.md # Required: documentation
│ │ ├── requirements.txt # Python dependencies
│ │ ├── main.py # Entry point
│ │ ├── .env.example # Environment variable template
│ │ └── ... # Your app code
```
Each app should be a self-contained directory with everything needed to run it independently.
***
## Step-by-Step Guide
Go to [github.com/BasedHardware/omi](https://github.com/BasedHardware/omi) and click **Fork** to create your own copy.
```bash theme={null}
git clone https://github.com/YOUR_USERNAME/omi.git
cd omi
```
Create a new folder under `plugins/` with a descriptive name:
```bash theme={null}
mkdir plugins/your-app-name
cd plugins/your-app-name
```
Use lowercase, hyphens for spaces (e.g., `meeting-summarizer`, `crm-sync`, `daily-journal`).
Structure your app with these files at minimum:
| File | Required | Description |
| - | - | - |
| `README.md` | ✅ | Documentation (see template below) |
| `main.py` | ✅ | App entry point |
| `requirements.txt` | ✅ | Python dependencies |
| `.env.example` | ✅ | Environment variable template (no secrets!) |
| `.gitignore` | Recommended | Ignore `.env`, `__pycache__/`, etc. |
**Never commit secrets, API keys, or credentials.** Use `.env.example` with placeholder values and add `.env` to `.gitignore`.
Your README is the first thing developers see. Include:
```markdown theme={null}
# Your App Name
Brief description of what your app does.
## Features
- Feature 1
- Feature 2
## How It Works
Explain the architecture and how it integrates with Omi.
## Setup
### Prerequisites
- Python 3.8+
- Any external service accounts needed
### Installation
1. Clone the repo
2. `pip install -r requirements.txt`
3. Copy `.env.example` to `.env` and fill in your values
4. Run `python main.py` or `fastapi run`
### Environment Variables
| Variable | Description |
|----------|-------------|
| `API_KEY` | Your API key for X service |
## Registering with Omi
Explain how to register the app (webhook URLs, etc.)
## Usage
How users interact with the app once it's running.
## License
MIT
```
Before submitting, make sure:
* [ ] App runs from a clean install following your README
* [ ] All environment variables are documented in `.env.example`
* [ ] No hardcoded secrets or API keys in the code
* [ ] Webhooks handle errors gracefully
* [ ] Code is clean and reasonably commented
```bash theme={null}
git checkout -b add-your-app-name
git add plugins/your-app-name/
git commit -m "feat: add your-app-name plugin"
```
Push your branch and open a PR against `main`:
```bash theme={null}
git push origin add-your-app-name
```
Then go to GitHub and create a Pull Request with:
* **Title:** `feat: add [your-app-name] plugin`
* **Description:** What the app does, how to use it, and any relevant context
* **Screenshots/demo** (if applicable)
***
## PR Review Checklist
Our team will review your PR against these criteria:
* Clean, readable code
* No unnecessary dependencies
* Proper error handling
* No hardcoded secrets or credentials
* Complete README with setup instructions
* `.env.example` with all required variables
* Clear explanation of what the app does
* App runs independently from its directory
* All dependencies listed in `requirements.txt`
* No modifications to core Omi code
* Respects user privacy
* No harmful or inappropriate content
* Provides genuine value to users
***
## Tips for a Successful Contribution
A well-documented simple app is better than a complex, undocumented one
Include screenshots, GIFs, or a demo video in your README
Respond to review feedback quickly to get your PR merged faster
Look at existing apps in `plugins/` for style and structure inspiration
***
## Contribution Rewards
Significant open-source contributions earn rewards:
| Contributions | Reward |
| - | - |
| 1 merged PR | Free DevKit device (necklace or glasses) |
| 2+ merged PRs | Additional DevKit + community recognition |
See the full [Contribution Guide](/doc/developer/Contribution) for details on rewards and bounties.
***
## Related Documentation
Submit your app to the Omi app store (no code required)
Learn how to build Omi apps from scratch
General contribution guidelines and rewards
Build webhook-based integrations
# Prompt-Based Apps
Source: https://docs.omi.me/doc/developer/apps/PromptBased
Customize Omi's behavior with prompts - no server required. Create specialized AI personalities and memory processors.
## What Are Prompt-Based Apps?
Prompt-based apps let you customize how Omi thinks and responds without writing any backend code. Simply define your prompt and Omi's behavior changes accordingly.
Everything runs on Omi's infrastructure - just write your prompt
Create, test, and publish in minutes
***
## Chat Prompts
Chat prompts alter Omi's conversational style and knowledge base. Create specialized versions of Omi for different contexts or expertise areas.
```mermaid theme={null}
flowchart LR
U[User Message] --> O[Omi + Your Prompt]
O --> R[Customized Response]
style O fill:#e1f5fe
```
When a user chats with Omi while your app is enabled:
1. Your prompt is injected into the system context
2. Omi adopts the personality/expertise you defined
3. All responses reflect your customizations
The prompt persists for the entire conversation, creating a consistent experience.
* **Expert Personas**: Make Omi communicate like a historian, scientist, or creative writer
* **Professional Advisors**: Create a fitness coach, financial advisor, or therapist persona
* **Language Tutors**: Build a conversational partner for language learning
* **Role-Playing**: Create characters for storytelling or gaming
* **Domain Experts**: Specialized knowledge in cooking, gardening, coding, etc.
### Writing Effective Chat Prompts
A great chat prompt defines personality, expertise, and response style clearly.
**Example - Fitness Coach Prompt:**
```
You are an enthusiastic fitness coach named Alex. You:
- Provide encouraging, actionable fitness advice
- Ask about the user's goals and current fitness level
- Suggest specific exercises with proper form instructions
- Celebrate progress and motivate through setbacks
- Keep responses concise but informative
```
**Example - Creative Writing Partner:**
```
You are a creative writing mentor. You:
- Help brainstorm story ideas and plot structures
- Provide constructive feedback on writing samples
- Suggest techniques for improving prose
- Share examples from famous authors when relevant
- Encourage experimentation with different styles
```
***
## Memory Prompts
Memory prompts analyze conversations and extract specific information based on your criteria. They run after each conversation is processed.
```mermaid theme={null}
flowchart LR
C[Conversation Ends] --> M[Memory Processing]
M --> P[Your Prompt]
P --> E[Extracted Information]
style P fill:#e8f5e9
```
After a conversation is transcribed and processed:
1. The full transcript is passed to your prompt
2. Your prompt extracts/summarizes specific information
3. The result is saved alongside the conversation
This creates structured data from unstructured conversations.
* **Action Items**: Extract to-do items and deadlines from meetings
* **Key Decisions**: Identify and log important decisions made
* **Meeting Notes**: Generate structured summaries
* **Sentiment Analysis**: Track emotional tone of conversations
* **Topic Extraction**: Categorize conversations by subject matter
* **Quote Collection**: Pull notable quotes or insights
### Writing Effective Memory Prompts
Be specific about what to extract and how to format the output.
**Example - Action Item Extractor:**
```
Analyze this conversation and extract all action items.
For each action item, identify:
- The task description
- Who is responsible (if mentioned)
- Any deadline or timeframe mentioned
- Priority level (high/medium/low based on context)
Format as a numbered list. If no action items exist, respond with "No action items found."
```
**Example - Meeting Summary:**
```
Create a structured summary of this meeting:
1. **Attendees**: List people mentioned
2. **Main Topics**: Bullet points of key discussion areas
3. **Decisions Made**: Any conclusions or agreements
4. **Action Items**: Tasks assigned with owners
5. **Follow-ups**: Next steps or future meetings mentioned
Keep each section concise (2-3 sentences max).
```
***
## Combining Chat and Memory Prompts
You can create apps with both chat AND memory prompts for comprehensive functionality.
**Example - Project Manager App:**
* **Chat Prompt**: Acts as a project management assistant, helping plan tasks and track progress
* **Memory Prompt**: Extracts project updates, blockers, and timeline changes from conversations
***
## Creating Your App
Decide what your app will do:
* **Chat only**: Personality/expertise customization
* **Memory only**: Information extraction
* **Both**: Complete conversational experience with data extraction
Craft clear, specific prompts:
* Define personality traits and expertise areas
* Specify extraction criteria and output format
* Include examples if helpful
Before publishing, test your app extensively (see testing section below)
Change visibility to public or submit for review (reviewed within 24 hours)
***
## Testing Your App
### Testing Memory Prompts
Open the Omi app → Settings → Enable Developer Mode
Navigate to any existing memory in your app
Tap the 3-dot menu (top right) → Developer Tools section
Test your memory prompt against the existing conversation
### Testing Chat Prompts
Create your app with visibility set to private
Install and enable your private app
Start a conversation and verify the behavior matches expectations
Refine your prompt based on the responses
***
## Best Practices
Vague prompts lead to inconsistent results. Define exact behaviors, formats, and constraints.
Try unusual inputs to ensure your prompt handles unexpected scenarios gracefully.
Single-purpose apps perform better than apps trying to do everything.
Start simple, test, and refine. Don't try to perfect the prompt on the first attempt.
Avoid prompts that could generate harmful, misleading, or inappropriate content. Apps violating community guidelines will be removed.
***
## Related Documentation
Build webhook-based apps for external integrations
Add custom tools that users can invoke in chat
Publishing guidelines and review process
Overview of all app types
# Publish Your App
Source: https://docs.omi.me/doc/developer/apps/Submitting
Submit your app to the Omi app store. Learn the review process, guidelines, and best practices for a successful submission.
## Overview
Ready to share your app with the Omi community? Publishing is done directly through the Omi mobile app.
```mermaid theme={null}
flowchart LR
D[Develop App] --> T[Test Thoroughly]
T --> S[Submit via App]
S --> R[Review Process]
R --> P[Published!]
```
Most apps are reviewed within 24 hours
No fees to submit or list your app
***
## Before You Submit
Thoroughly test all functionality in Developer Mode:
* Verify webhooks receive data correctly
* Test with real conversations
* Check error handling
* Confirm notifications work (if applicable)
Create compelling app store content:
* **App Name**: Clear, descriptive (max 30 characters)
* **Description**: Explain what your app does and why it's useful
* **Category**: Choose the most relevant category
* **Icon**: Upload a distinctive app icon
Ensure your app follows all community guidelines (see below)
For integration apps, confirm your server is:
* Using HTTPS
* Responding quickly (\< 5 seconds)
* Handling errors gracefully
* Available 24/7
***
## How to Submit
Launch the Omi mobile app on your device
Go to **Explore** → Find your app in **My Apps** or create a new one
Fill in all required fields:
* Name and description
* Category selection
* Capabilities and permissions
* Webhook URLs (for integration apps)
Change your app's visibility from **Private** to **Public**
Tap **Submit** and wait for the review process
You can continue to update and test your app while it's under review. Changes won't affect the review process.
***
## Community Guidelines
Apps that violate these guidelines will be rejected or removed from the store.
No content promoting hate, discrimination, or harassment of any kind
Consider a general audience - avoid explicit or adult content
If providing facts or advice, strive for correctness and clarity
Respect user data, obtain clear consent, and follow data protection best practices
No facilitating illegal activities or violating terms of service
Be clear about what your app does and what data it accesses
***
## Review Process
During review, we verify:
| Check | Description |
| - | - |
| **Functionality** | App works as described |
| **Guidelines** | Follows community guidelines |
| **Security** | No obvious security issues |
| **Endpoints** | Webhooks respond correctly (integration apps) |
| **Content** | Description is accurate and appropriate |
* **Most apps**: Reviewed within 24 hours
* **Complex integrations**: May take 2-3 days
* **Weekends/holidays**: May experience delays
You'll receive a notification when your app is approved or if changes are needed.
Avoid these common issues:
1. **Broken webhooks** - Endpoint returns errors or times out
2. **Misleading description** - App doesn't do what it claims
3. **Missing permissions** - Requesting more access than needed
4. **Guideline violations** - Content issues
5. **Poor user experience** - Confusing or broken functionality
If your app is rejected:
1. Review the feedback provided
2. Make necessary changes
3. Test thoroughly
4. Resubmit for review
There's no limit on resubmissions - we want to help you succeed!
***
## Best Practices for Success
* Start with what the app does
* Explain the value to users
* Include usage examples
* List any requirements
* Productivity: Task management, notes, workflows
* Health: Fitness, wellness, medical
* Social: Communication, sharing
* Entertainment: Fun, games, creativity
* Use simple, recognizable imagery
* Ensure visibility at small sizes
* Match your app's purpose
* Avoid text-heavy designs
* Monitor your webhook uptime
* Respond to user feedback
* Update for new Omi features
* Keep your app fresh
***
## After Publishing
Track your app's install count and user engagement in the Omi app
Users may reach out with questions or suggestions - be responsive!
Keep your app current with new features and improvements
Connect with other developers in the [Omi Discord](http://discord.omi.me) for support and collaboration
Successful apps are ones that solve real problems for users. Focus on delivering value, and the installs will follow!
***
## Related Documentation
Create apps without a server
Build webhook-based integrations
Add custom tools to Omi chat
Overview of all app types
# GitHub
Source: https://docs.omi.me/doc/developer/apps/examples/Github
Create GitHub issues using voice commands through your OMI device
Install the GitHub Issues app from the Omi App Store
Check out the full source code on GitHub
## ✨ Features
* **🎤 Voice-Activated** - Say "Feedback Post" and describe your problem
* **🧠 AI-Powered** - Collects 5 segments for detailed feedback, generates title + description
* **🔐 OAuth Authentication** - Secure GitHub OAuth 2.0 integration
* **📦 Repository Selection** - Choose which repo receives issues during setup
* **⚙️ Flexible Settings** - Change target repository anytime from mobile-first homepage
* **🤖 Smart Formatting** - AI extracts key info and formats professional issues
* **🔕 Silent Collection** - Only notifies when issue is created
* **📱 Mobile-First UI** - Beautiful responsive design for all devices
## 🚀 Quick Start
Install the [GitHub Issues app](https://h.omi.me/apps/01K7BHKJR79KQQW7ZZGZX7GCGD) in your OMI mobile app
Authenticate your GitHub account (one-time setup)
Choose the repository where issues should be created
* Say: "Feedback Post, the app keeps crashing when I upload photos..."
* Keep describing the problem naturally for 15-20 seconds
* AI processes your speech and creates a formatted issue
* You get a notification with the issue link! ✅
### Trigger Phrases
* "Feedback Post"
* "Create Issue"
* "Report Issue"
* "File Issue"
* "New Issue"
### How It Works
**The app intelligently collects and processes your feedback:**
1. Detects "Feedback Post" → Starts collecting
2. Automatically collects 5 segments (15-20 seconds of detailed speech)
3. Sends all segments to AI
4. AI generates professional title and detailed description
5. Creates GitHub issue with "voice-feedback" label
6. Notifies you with issue link! 🎉
**Example Workflow:**
You: "Feedback Post, the search function isn't working"\
*\[collecting silently...]*
You: "when I type in the search bar nothing happens"\
*\[collecting silently...]*
You: "I've tried on both Chrome and Safari"\
*\[collecting silently...]*
You: "and it worked fine last week but now"\
*\[collecting silently...]*
You: "it's completely broken on all browsers"\
*→ AI processes all 5 segments*
**AI Generated Issue:**
* **Title:** "Search function not working across browsers"
* **Description:** The search function is currently not working. When typing in the search bar, no results appear. This issue has been tested on both Chrome and Safari browsers. The search functionality was working correctly last week but is now completely non-functional across all browsers.
→ Issue created! 🔔
## 🧠 AI Processing
The app uses OpenAI for intelligent issue generation:
1. **Collection** - Gathers 5 segments (\~15-20 seconds) for detailed context
2. **Title Generation** - Creates concise, descriptive title
3. **Description Formatting** - Structures problem statement professionally
4. **Cleanup** - Removes filler words, fixes grammar, adds proper formatting
**Example Transformation:**
```text Voice Input (5 segments) theme={null}
"the app keeps crashing when I um try to upload photos
it happens every single time on my iPhone 14 like
the app just freezes for a second and then
it completely closes and this started happening
after the latest update yesterday"
```
```markdown AI-Generated GitHub Issue theme={null}
Title: App crashes when uploading photos on iPhone 14
Description:
The app consistently crashes during photo uploads. When attempting to
upload a photo, the app freezes briefly and then closes completely.
This issue occurs every time on iPhone 14 and started after the latest
update.
Labels: voice-feedback
```
## 📊 How Segments Work
**OMI sends transcripts in segments** as you speak. The app:
* ✅ Detects "Feedback Post" trigger
* ✅ Collects exactly 5 segments
* ✅ Silent during collection (no spam)
* ✅ AI processes all 5 together
* ✅ One notification on completion
**Why 5 segments?**
* Allows detailed problem description
* Captures \~15-20 seconds of speech
* Gives context for better issue formatting
* AI has full information for title/description generation
### Refreshing Repository List
Click "Refresh Repos" to:
* Fetch latest list from GitHub
* Include newly created repositories
* Update repository access permissions
## 🔐 Security & Privacy
**Security Features:**
* OAuth 2.0 authentication (no password storage)
* Tokens stored securely with file persistence
* Per-user token isolation
* HTTPS enforced in production
* State parameter for CSRF protection
* Secure scope: only `repo` access
## 🐛 Troubleshooting
* Complete the GitHub OAuth flow
* Try re-authenticating if the issue persists
* Visit the app homepage
* Select a repository from the dropdown menu
* Click "Save Repository" to confirm
* Verify the repository exists and you have write access
* Ensure OAuth app has correct permissions
* Click "Refresh Repos" button on the homepage
* Ensure you have push access to the repository
* Verify GitHub OAuth app permissions are correctly set
## Support & Resources
Integration App Guide
View the full codebase
# Omi Mentor
Source: https://docs.omi.me/doc/developer/apps/examples/OmiMentor
Get personalized AI mentorship during your conversations with OMI
Install the OMI Mentor app from the OMI App Store
Check out the full source code on GitHub
## Features
* **Context-Aware Mentorship** - AI analyzes your conversations and provides timely insights
* **Smart Interruption Logic** - Only interrupts when you need meaningful guidance
* **Topic Extraction** - Automatically identifies key topics from your discussions
* **Personalized Advice** - Uses your facts, context, and chat history for tailored guidance
* **Silence Detection** - Intelligently detects conversation breaks and starts fresh
* **Real-time Processing** - Analyzes conversations every 5 minutes for relevant insights
* **Privacy-Focused** - Processes only when you're actively participating in conversations
## Quick Start
Install the OMI Mentor app in your OMI mobile app
Have natural conversations while wearing your OMI device
Get personalized mentorship when the AI detects you need guidance
Act on the specific, actionable recommendations provided
## How It Works
**The app intelligently monitors and analyzes your conversations:**
1. Listens to your conversations in real-time
2. Buffers messages and detects silence periods
3. Analyzes conversation context every 5 minutes
4. Extracts relevant topics using AI
5. Evaluates if mentorship is warranted
6. Sends personalized, direct advice when appropriate
**Example Workflow:**
You: "I'm not sure if I should take this new job offer..."\
*\[AI listens and analyzes conversation]*
Friend: "What's holding you back?"\
*\[Continues collecting context]*
You: "The salary is better but I'd lose my work-life balance..."\
*\[AI evaluates if intervention is needed]*
**AI Mentor Decision:**
* ✓ User is participating
* ✓ User expressed a specific challenge
* ✓ Has strong, clear opinion to share
* ✓ Insight is time-sensitive
**AI Sends:** "You just said work-life balance matters more than money. The answer's already clear - you're trying to convince yourself otherwise. What would you tell your best friend in this situation?"
→ Notification sent!
## Using the App
### With OMI Device
1. Install the OMI Mentor app from the OMI App Store
2. Wear your OMI device during conversations
3. The AI monitors and analyzes in the background
4. Receive timely notifications with personalized advice
5. Implement the guidance provided
## AI Processing
The app uses OpenAI for intelligent mentorship:
1. **Topic Extraction** - Identifies key topics from conversations using GPT-4
2. **Context Analysis** - Evaluates user participation and expressed challenges
3. **Interruption Logic** - Determines if advice is warranted based on 4 criteria
4. **Personalized Guidance** - Generates direct, actionable advice using user context
**Interruption Criteria:**
```text Requirements (ALL must be met) theme={null}
1. User is actively participating in the conversation
2. User expressed a specific problem, challenge, goal, or question
3. AI has a STRONG, CLEAR opinion that would significantly impact the situation
4. The insight is time-sensitive and worth interrupting for
```
```text Response Style theme={null}
- Speaks DIRECTLY to the user (no third-person analysis)
- Takes a clear stance (no "however" or "on the other hand")
- Under 300 characters
- Simple, everyday language
- References specific details from what user said
- Bold and direct advice
- Ends with a specific implementation question
```
## How the Buffer System Works
**OMI Mentor uses an intelligent message buffering system:**
* Collects conversation segments continuously
* Detects silence periods (2+ minutes of inactivity)
* Requires minimum 5 words after silence to resume
* Clears old messages after silence to start fresh
* Analyzes buffered messages every 5 minutes
* Cleans up inactive sessions after 1 hour
**Smart Collection:**
* **Analysis Interval:** Every 5 minutes
* **Silence Threshold:** 2 minutes of no activity
* **Min Words to Resume:** 5 words after silence
* **Session Cleanup:** 1 hour of inactivity
* **Message Grouping:** Combines messages from same speaker within 2 seconds
## Personalization Features
### User Context Integration
The AI uses multiple sources to personalize advice:
* **User Facts** - Information about the user's preferences and history
* **User Context** - Previous discussions and ongoing situations
* **Chat History** - Past interactions with the mentor
* **Current Topics** - Automatically extracted conversation themes
### Topic-Based Filtering
Topics are automatically extracted and used to:
* Filter relevant historical context
* Match with user's interests and expertise
* Provide domain-specific guidance
* Connect current challenges with past experiences
## Privacy & Behavior
**Privacy Features:**
* Only analyzes when you're actively speaking
* Doesn't interrupt every conversation
* Requires specific conditions before providing advice
* Clears messages after silence periods
* No persistent storage of conversation content
* Session-based processing with automatic cleanup
## Troubleshooting
* Ensure you're actively participating in conversations
* The AI only interrupts when all 4 criteria are met
* Check if 5 minutes have passed since last analysis
* Verify the app is properly installed and running
* The AI is designed to interrupt sparingly
* Only sends advice when time-sensitive and impactful
* Check if multiple problems are being discussed in short timeframes
* Ensure your user facts and context are up to date
* The AI uses your historical data for personalization
* Try having more detailed conversations about your challenges
* Check if OMI device is properly connected
* Verify app has necessary permissions
* Ensure you're speaking clearly during conversations
## Support & Resources
Integration App Guide
View the full codebase
# Slack
Source: https://docs.omi.me/doc/developer/apps/examples/Slack
Send Slack messages using voice commands through your OMI device
Install the Slack app from the OMI App Store
Check out the full source code on GitHub
## Features
* **Voice-Activated** - Say "Send message" and speak naturally
* **AI-Powered Channel Matching** - AI intelligently matches spoken channel names to your workspace
* **OAuth Authentication** - Secure Slack OAuth 2.0 integration
* **Channel Selection** - Set a default channel or specify in voice command
* **Flexible Settings** - Change channels anytime from mobile-first homepage
* **Smart Message Extraction** - AI cleans up filler words and formats professionally
* **Silent Collection** - Only notifies when message is sent
* **Mobile-First UI** - Beautiful responsive Slack-themed design
## Quick Start
Install the Slack app in your OMI mobile app
Authenticate your Slack workspace (one-time setup)
Choose a default channel (optional - you can specify in voice)
* Say: "Send message to general saying hello team!"
* Say: "Post in marketing that the campaign is live"
* Say: "Slack message to random saying great idea!"
### Trigger Phrases
* "Send Slack message" - "Send Slack message to general saying..."
* "Post Slack message" - "Post Slack message in marketing that..."
* "Post in Slack" - "Post in Slack to random saying..."
### How It Works
**The app intelligently processes your voice commands:**
1. Detects trigger phrase → Starts collecting
2. Collects up to 5 segments OR stops if 5+ second gap detected
3. AI extracts:
* Channel name (fuzzy matches to your workspace channels)
* Message content (cleaned and formatted)
4. Fetches fresh channel list automatically (new channels work immediately!)
5. Posts message to Slack
6. Notifies you with confirmation!
**Example Workflow:**
You: "Send Slack message to general saying hello team"\
*\[collecting segment 1/5...]*
You: "hope everyone is having a great day"\
*\[collecting segment 2/5...]*\
*\[5+ second pause - timeout!]*\
*→ AI processes 2 segments*
**AI Extracted:**
* **Channel:** #general
* **Message:** "Hello team, hope everyone is having a great day."
→ Message sent!
## Using the App
### With OMI Device
1. Install the Slack app from the OMI App Store
2. Authenticate your Slack workspace
3. Select a default channel (optional)
4. Say: "Send message to general saying hello team!"
5. Wait for AI processing (happens silently)
6. Get a notification with confirmation!
## AI Processing
The app uses OpenAI for intelligent processing:
1. **Channel Matching** - Fuzzy matches spoken channel names to workspace channels
2. **Message Extraction** - Extracts clean message content from voice segments
3. **Cleanup** - Removes filler words, fixes grammar, proper formatting
**Example Transformation:**
```text Voice Input (3 segments) theme={null}
"to general saying um hello team hope you're all um doing great today"
```
```markdown AI-Extracted Message theme={null}
Channel: #general (matched from "general")
Message: "Hello team, hope you're all doing great today"
```
## How Segments Work
**OMI sends transcripts in segments** as you speak. The app:
* Detects trigger phrase (Send Slack message / Post Slack message / Post in Slack)
* Collects up to 5 segments MAX
* Processes early if 5+ second gap detected (minimum 2 segments)
* Silent during collection (no spam)
* AI processes all collected segments together
* One notification on completion
**Smart Collection:**
* **Max segments:** 5 (including trigger)
* **Timeout:** 5 seconds of silence → processes immediately
* **Minimum:** 2 segments (trigger + content)
* **Duration:** \~5-20 seconds depending on speech
* **Auto-refresh:** Fetches latest channels every time (new channels work immediately!)
## Channel Management
### Specifying Channel in Voice
You can always specify the channel in your voice command:
* "Send message to **general** saying hello"
* "Post in **marketing** that campaign is live"
* "Message to **engineering** about the bug fix"
AI will fuzzy match to your workspace channels!
### Using Default Channel
Set a default channel in settings, then just say:
* "Send message saying quick update for everyone"
* Message goes to your default channel
### Refreshing Channel List
The app **automatically fetches fresh channels** every time you send a message, so new channels work immediately without manual refresh!
You can also manually refresh:
* Click "Refresh Channels" button on homepage
* Or re-authenticate to get latest channels
### Switching Workspaces
Click "Switch Workspace" to:
* Connect to a different Slack workspace
* Re-authenticate with new team
* Switch between multiple workspaces easily
## Security & Privacy
**Security Features:**
* OAuth 2.0 authentication (no password storage)
* Tokens stored securely with file persistence
* Per-user token isolation
* HTTPS enforced in production
* State parameter for CSRF protection
* Secure scopes: minimal required permissions
## Troubleshooting
* Complete the Slack OAuth flow
* Try re-authenticating if the issue persists
* Visit the app homepage
* Select a default channel OR
* Specify channel in voice command
* Verify channel exists and bot has access
* Ensure Slack app has correct scopes
* Check channel name pronunciation
* AI does fuzzy matching but might need clearer speech
* Use "Refresh Channels" to update list
* Set as default channel in settings
## Support & Resources
Integration App Guide
View the full codebase
# Backend Setup
Source: https://docs.omi.me/doc/developer/backend/Backend_Setup
Set up the Omi backend infrastructure for local development. This guide covers Google Cloud, Firebase, OAuth, and all required services.
## Overview
The Omi backend powers all AI capabilities including transcription, conversation processing, chat, and integrations. This guide will help you set up a local development environment.
```mermaid theme={null}
flowchart LR
subgraph External["External Services"]
OAI[OpenAI]
STT[Parakeet / Modulate]
Pine[Pinecone]
Redis[Redis]
end
subgraph Google["Google Cloud"]
FB[Firebase]
GCS[Cloud Storage]
end
subgraph Backend["Your Backend"]
API[FastAPI Server]
Push[Pusher Service]
end
subgraph App["Omi App"]
Flutter[Flutter Client]
end
Flutter <-->|Ngrok| API
API --> OAI
API --> DG
API --> Pine
API --> Redis
API --> FB
API --> GCS
API --> Push
```
If you just want to build apps or test features, you can use Omi's development backend instead. See the [App Setup Guide](/doc/developer/AppSetup) for the automatic setup option.
***
## Video Walkthrough
***
## Prerequisites
Before starting, gather these API keys and credentials:
| Service | Purpose | Get Key |
| - | - | - |
| OpenAI | AI language models | [Get Key](https://platform.openai.com/settings/organization/api-keys) |
| Modulate | Speech-to-text fallback | Provision `MODULATE_API_KEY` |
| Parakeet | Self-hosted speech-to-text | Configure `HOSTED_PARAKEET_API_URL` |
| Redis (Upstash) | Caching & sessions | [Get Key](https://console.upstash.com/) |
| Pinecone | Vector database | [Get Key](https://app.pinecone.io/organizations/-/projects/-/api-keys) |
| Hugging Face | Voice detection | [Get Key](https://huggingface.co/settings/tokens) |
| Service | Purpose | Get Key |
| - | - | - |
| Modal | Serverless deployment | [Get Key](https://modal.com/settings#tokens) |
| GitHub | Firmware updates | [Get Key](https://github.com/settings/tokens) |
| Google Maps | Location features | [Get Key](https://console.cloud.google.com/google/maps-apis/credentials) |
| Typesense | Search functionality | [Get Key](https://cloud.typesense.org/clusters) |
| Stripe | Payment processing | [Get Key](https://dashboard.stripe.com/apikeys) |
New to backend development? Install [Homebrew](https://docs.brew.sh/Installation) (macOS/Linux) or [Chocolatey](https://chocolatey.org/install) (Windows) first - they make installing tools much easier.
***
## 1. Google Cloud & Firebase Setup
```bash theme={null}
brew install google-cloud-sdk
```
```powershell theme={null}
choco install gcloudsdk
```
The SDK is usually pre-installed in Nix environments. Otherwise, follow the [official installation guide](https://cloud.google.com/sdk/docs/install).
Go to the [Google Cloud Console](https://console.cloud.google.com/) and enable these APIs:
Required for project management
Required for Firebase integration
Run these commands in your terminal, replacing `` with your Google Cloud project ID:
```bash theme={null}
gcloud auth login
gcloud config set project
gcloud auth application-default login --project
```
This installs **Application Default Credentials** (ADC) for the Google client
libraries. Do **not** copy a service-account JSON key into the repo — long-lived
keys are the #6800 anti-pattern. Prefer plain user ADC for local work.
Impersonation is an advanced, time-limited option (ADC tokens from
`--impersonate-service-account` expire in about an hour). Only use it when you
already have an org-provisioned lower-privilege SA and hold
`roles/iam.serviceAccountTokenCreator` on that account — replace the
placeholder below with a real SA email from your team; there is no default
`dev-readonly@…` account in this repo:
```bash theme={null}
gcloud auth application-default login \
--impersonate-service-account=@.iam.gserviceaccount.com
```
Firebase Admin and Firestore discover ADC automatically when
`SERVICE_ACCOUNT_JSON` / `GOOGLE_APPLICATION_CREDENTIALS` are unset.
***
## 2. OAuth Authentication Setup
OAuth is required for user authentication. You need to configure both Google and Apple sign-in.
1. Go to [Google Cloud Console → Credentials](https://console.cloud.google.com/apis/credentials)
2. Click **Create Credentials** → **OAuth 2.0 Client ID**
3. Select **Web application** as the application type
4. Set a name (e.g., "Omi Backend Auth")
Under **Authorized JavaScript origins**, add:
* `https://your-domain.com` (production)
* `https://your-ngrok-domain.ngrok-free.app` (local development)
Under **Authorized redirect URIs**, add:
* `https://your-domain.com/v1/auth/callback/google`
* `https://your-ngrok-domain.ngrok-free.app/v1/auth/callback/google`
Click **Create** and save your **Client ID** and **Client Secret** for the `.env` file.
Go to **APIs & Services → OAuth consent screen**:
* Fill in required app information
* Add your domain to **Authorized domains**
* Add scopes: `openid`, `email`, `profile`
Apple OAuth requires a paid Apple Developer account (\$99/year).
1. Go to [Apple Developer Console → Identifiers](https://developer.apple.com/account/resources/identifiers/list)
2. Create a new App ID with **Sign In with Apple** capability enabled
3. Note your Bundle ID
1. Create a new **Services ID** (this becomes your `APPLE_CLIENT_ID`)
2. Configure **Sign In with Apple** for this Services ID
3. Add authorized domains and return URLs:
* `https://your-domain.com/v1/auth/callback/apple`
* `https://your-ngrok-domain.ngrok-free.app/v1/auth/callback/apple`
1. Go to **Keys** in Apple Developer Console
2. Create a new key with **Sign In with Apple** enabled
3. Download the `.p8` file and note the **Key ID**
4. Note your **Team ID** from your Apple Developer account
1. Go to [Firebase Console](https://console.firebase.google.com/) → Authentication → Sign-in method
2. Enable **Apple** and add your configuration:
* Client ID: Your Services ID
* Team ID: Your Apple Developer Team ID
* Key ID: From your private key
* Private Key: Contents of your .p8 file
Your Apple environment variables will be:
* `APPLE_CLIENT_ID`: Your Services ID
* `APPLE_TEAM_ID`: Your Apple Developer Team ID
* `APPLE_KEY_ID`: The Key ID from step 3
* `APPLE_PRIVATE_KEY`: The full contents of your .p8 file (including BEGIN/END lines)
***
## 3. Backend Installation
```bash theme={null}
git clone https://github.com/BasedHardware/Omi.git
cd Omi/backend
```
The backend Docker images run Python 3.11, so use Python 3.11 locally as well. If your package manager installs a newer default Python, install or select Python 3.11 before creating the virtual environment.
```bash theme={null}
brew install python@3.11 git ffmpeg opus uv
uv --version
```
Install Python 3.11 from [python.org](https://www.python.org/downloads/windows/), then install the remaining system tools and `uv`:
```powershell theme={null}
choco install git.install ffmpeg
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
python --version # should print 3.11.x
uv --version
```
The Python `opuslib` package still needs a native `libopus` DLL on `PATH`. One common setup is MSYS2 UCRT64:
install MSYS2, run `pacman -S mingw-w64-ucrt-x86_64-opus`, add `C:\msys64\ucrt64\bin` to `PATH`,
then open a new shell and verify with `where.exe opus.dll`.
Python, Git, and FFmpeg are typically pre-installed. Install opus and `uv` via your package manager if needed, or use the standalone installer:
```bash theme={null}
curl -LsSf https://astral.sh/uv/install.sh | sh
uv --version
```
The backend uses `uv`, `.python-version`, and `pylock.toml` for reproducible local and CI installs.
```bash theme={null}
# From backend/
./scripts/sync-python-deps.sh
source .venv/bin/activate
```
```powershell theme={null}
# From backend/
bash ./scripts/sync-python-deps.sh
.venv\Scripts\Activate.ps1
```
You should see `(.venv)` at the beginning of your command prompt.
When intentionally changing backend dependencies, edit the relevant `requirements*.txt` input, then refresh the locks:
```bash theme={null}
./scripts/update-python-lock.sh
```
The refresh preserves already-locked package versions by default. Use `PYLOCK_UPGRADE=1 ./scripts/update-python-lock.sh` only when intentionally upgrading dependency versions.
```bash theme={null}
cp .env.template .env
```
Edit `.env` and fill in your API keys (see [Environment Variables](#environment-variables) below).
For a self-hosted backend used by an existing Omi mobile build, set the
Firebase token audience separately from your data project:
```dotenv theme={null}
# Stock development/community Omi mobile builds mint tokens for this project.
FIREBASE_AUTH_PROJECT_ID=based-hardware-dev
# Your own Google Cloud/Firebase project for Firestore and other backend data.
FIREBASE_PROJECT_ID=your-self-hosted-project
```
Firebase ID tokens are verified with the issuer's public signing keys, so
`FIREBASE_AUTH_PROJECT_ID` does not require Omi service-account credentials.
When it is unset, Firebase Admin uses its default project selection instead;
set the auth project explicitly when it differs from your data-project setup,
or stock mobile-build tokens will be rejected.
Do not enable `LOCAL_DEVELOPMENT` on an internet-accessible deployment: that
mode intentionally bypasses authentication for local harnesses.
***
## 4. Optional Services
Skip this if you don't need webhook functionality.
```bash theme={null}
cd pusher
cp .env.template .env
```
Prefer ADC from `gcloud auth application-default login` (leave
`SERVICE_ACCOUNT_JSON` unset). Only use a scoped secret for Modal/CI
compatibility paths that have not yet moved to Workload Identity (#6800).
Start the service:
```bash theme={null}
cd ..
uvicorn pusher.main:app --reload --env-file .env --port 8000
```
Optionally expose via Ngrok for external access.
Skip this if you don't need search functionality.
1. Create an account on [Typesense](https://typesense.org/)
2. Create a collection named `conversations` using the schema in `typesense/conversations.schema`
3. Install the [Firebase Typesense extension](https://console.firebase.google.com/project/_/extensions/install?ref=typesense/firestore-typesense-search@2.0.0-rc.1)
Configuration for the extension:
| Setting | Value |
| - | - |
| Firestore Collection Path | `users/{userId}/conversations` |
| Firestore Collection Fields | `structured,created_at,discarded,started_at,id,finished_at,geolocation,userId` |
If you have existing data, create a `typesense_sync` collection and add a document named `backfill` with `{'trigger': true}`.
***
## 5. Running the Backend
Ngrok exposes your local backend to the internet so the Omi app can connect.
1. Sign up at [ngrok.com](https://ngrok.com/) and install Ngrok
2. Authenticate with your account
3. Start the tunnel:
```bash theme={null}
ngrok http --domain=your-domain.ngrok-free.app 8000
```
Note the public URL (e.g., `https://your-domain.ngrok-free.app`).
```bash theme={null}
uvicorn main:app --reload --env-file .env --port 8000
```
| Flag | Purpose |
| - | - |
| `--reload` | Auto-restart on code changes |
| `--env-file .env` | Load environment variables |
| `--host 0.0.0.0` | Listen on all interfaces (optional) |
| `--port 8000` | Port to listen on |
In your Omi app's `.dev.env` file, set:
```
API_BASE_URL=https://your-domain.ngrok-free.app/
```
Don't forget the trailing `/` in the URL!
Make sure your OAuth redirect URIs in Google Cloud Console and Apple Developer Console include your Ngrok URL.
Deactivate the virtual environment:
```bash theme={null}
deactivate
```
To resume later, just activate the virtual environment again.
***
## Troubleshooting
Add this to `utils/stt/vad.py`:
```python theme={null}
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
```
* Double-check all API keys in your `.env` file
* Ensure there are no trailing spaces or quotes around values
* Verify keys are active and not expired
* Ensure your Ngrok tunnel is running
* Verify the URL is correctly set in the Omi app
* Check that OAuth redirect URIs match your Ngrok URL
Try reinstalling dependencies:
```bash theme={null}
./scripts/sync-python-deps.sh
```
* On Windows, you may need to enable script execution: `Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser`
* Make sure you're using Python 3.11
***
## Environment Variables
Complete reference for all `.env` variables:
| Variable | Description |
| - | - |
| `OPENAI_API_KEY` | OpenAI API key for AI models |
| `MODULATE_API_KEY` | Modulate API key for transcription fallback |
| `HOSTED_PARAKEET_API_URL` | Parakeet endpoint for transcription |
| `HUGGINGFACE_TOKEN` | Hugging Face token for voice detection models |
| `REDIS_DB_HOST` | Redis host address |
| `REDIS_DB_PORT` | Redis port number |
| `REDIS_DB_PASSWORD` | Redis password (blank if none) |
| `PINECONE_API_KEY` | Pinecone API key for vector database |
| `PINECONE_INDEX_NAME` | Name of your Pinecone index |
| `GOOGLE_APPLICATION_CREDENTIALS` | Optional. Leave unset to use ADC from `gcloud auth application-default login`. Do not point this at a long-lived service-account JSON key (#6800). |
| `ADMIN_KEY` | Temporary key for local dev (e.g., `123`) |
| `ENCRYPTION_SECRET` | At least 32 bytes for encrypting user data |
| Variable | Description |
| - | - |
| `GOOGLE_CLIENT_ID` | Google OAuth 2.0 Client ID |
| `GOOGLE_CLIENT_SECRET` | Google OAuth 2.0 Client Secret |
| `APPLE_CLIENT_ID` | Apple Services ID |
| `APPLE_TEAM_ID` | Apple Developer Team ID |
| `APPLE_KEY_ID` | Apple private key ID |
| `APPLE_PRIVATE_KEY` | Apple .p8 file contents (with BEGIN/END lines) |
| `BASE_API_URL` | Your backend URL (e.g., Ngrok URL) |
| Variable | Description |
| - | - |
| `FIREBASE_AUTH_PROJECT_ID` | Firebase project whose ID-token audience the backend accepts; set this explicitly when it differs from the data project. |
| `FIREBASE_PROJECT_ID` | Google Cloud/Firebase project used for the backend's Firestore and other data services. |
| Variable | Description |
| - | - |
| `BUCKET_SPEECH_PROFILES` | GCS bucket for speech profiles |
| `BUCKET_PLUGIN_LOGOS` | GCS bucket for app logos |
| `BUCKET_BACKUPS` | GCS bucket for backups |
| Variable | Description |
| - | - |
| `GITHUB_TOKEN` | GitHub token for firmware updates |
| `HOSTED_PUSHER_API_URL` | URL of your pusher service |
| `TYPESENSE_HOST` | Typesense server URL |
| `TYPESENSE_API_KEY` | Typesense API key |
| `NO_SOCKET_TIMEOUT` | Set `True` to disable socket timeout |
`.env.template` intentionally leaves `ENCRYPTION_SECRET` blank. Generate a unique securely managed value of at least 32 bytes for every real environment.
***
## Code Formatting
We use `black` for code formatting with a line length of 120 characters.
```bash theme={null}
source .venv/bin/activate
uv pip install black
```
To auto-format on commit, install the pre-commit hook from the repository root:
```bash theme={null}
ln -s -f ../../scripts/pre-commit .git/hooks/pre-commit
```
***
## Related Documentation
Set up the Flutter app for development
Understand the backend architecture
Learn how the chat feature works
How conversations and memories are stored
***
## Need Help?
Report bugs or request features
Get help from the community
# Storing Conversations & Memories
Source: https://docs.omi.me/doc/developer/backend/StoringConversations
This document outlines how Omi stores conversation data and extracts memories from conversations.
## Overview
Omi uses a **dual-collection architecture** for storing user data:
**Primary storage** for recorded interactions - transcripts, audio, structured summaries
**Secondary storage** for extracted facts/learnings FROM conversations
This separation allows for efficient retrieval of both full conversation context and quick access to key facts about the user.
## On-device summaries in desktop reads
The app conversation response may carry a display-only `client_processing` projection.
Binding it does not modify canonical `structured` or create authoritative tasks or memories.
The macOS `ServerConversation` adapter selects the projection once for every display consumer:
only schema v1 is understood, and a response containing transcript segments must match the
projection's `transcript_sha256` using encoding v5, including speaker identity and attribution.
A list response omitting the transcript relies on the server's transactional binding and
invalidation when the transcript changes; it cannot independently prove freshness.
An enriched canonical overview, sections, events, or non-default category wins over a retained
projection, including after an upgrade and cloud reprocessing. `processing_state` is not a
cloud-origin discriminator: it can be absent with a projection, or remain `local_pending` after
late binding. Title-only/default-field cloud results are indistinguishable from the deterministic
minimum in this wire contract, so that ambiguous shape follows the projection-first rule.
Canonical action items win exact-description collisions, preserving both completed and reopened
state; unmatched projected items are display-only additions with no canonical task identity.
Selected provenance remains available as `ServerConversation.localSummary` and is cached in
`transcription_sessions.localSummaryJson`. The producer's `clientProcessingJson` retry bytes are
separate. Revision-aware sync replaces the selected summary atomically; older or unversioned
snapshots cannot fill fields across projection/canonical sources. A cached transcript whose hash
differs from the selected projection is left unloaded until detail refresh.
Run the desktop regressions from the repository root:
```bash theme={null}
xcrun swift test --package-path desktop/macos/Desktop --filter 'ConversationProjection(Rendering|Persistence)Tests'
```
These hermetic tests exercise decode, reconciliation, and production SQLite storage. Named-bundle
end-to-end acceptance remains a separate check.
## Architecture Diagram
```mermaid theme={null}
flowchart TD
subgraph Recording["📱 Recording"]
R[User Recording] --> AS[Audio Stream]
AS --> T[Transcription
Parakeet / Modulate]
end
T --> POST[POST /v1/conversations]
POST --> PC[process_conversation]
PC --> GS[_get_structured
title, overview,
action_items, events]
PC --> EM[_extract_memories
facts → memories]
PC --> SAI[_save_action_items
standalone collection]
GS --> UC[upsert_conversation]
EM --> SM[save_memories]
UC --> FS[(Firestore:
conversations/)]
SM --> FSM[(Firestore:
memories/)]
UC --> Pine[(Pinecone:
vectors)]
```
## Firestore Structure
### Folder deletion and concurrent conversation changes
`DELETE /v1/folders/{folder_id}` rechecks each discovered conversation's current
folder membership before rehoming it. A newer move or unfile decision is preserved,
and a conversation deleted during cleanup is not recreated. Cleanup uses bounded
transactions, not one all-or-nothing transaction across a large folder. A failed
chunk leaves the source folder available for retry. This protects discovered
candidates; it does not serialize new assignments into the source folder or
concurrent deletion of the destination folder.
```
users/
├── {uid}/
│ ├── conversations/ # PRIMARY - Recorded interactions
│ │ └── {conversation_id}/
│ │ ├── id
│ │ ├── created_at
│ │ ├── started_at
│ │ ├── finished_at
│ │ ├── source
│ │ ├── language
│ │ ├── status
│ │ ├── structured
│ │ ├── transcript_segments
│ │ ├── geolocation
│ │ ├── photos/ (subcollection)
│ │ ├── audio_files
│ │ ├── apps_results
│ │ ├── discarded
│ │ ├── visibility
│ │ ├── is_locked
│ │ └── data_protection_level
│ │
│ ├── memories/ # SECONDARY - Extracted facts
│ │ └── {memory_id}/
│ │ ├── id
│ │ ├── uid
│ │ ├── conversation_id
│ │ ├── content
│ │ ├── category
│ │ ├── tags
│ │ ├── visibility
│ │ ├── created_at
│ │ ├── updated_at
│ │ ├── reviewed
│ │ ├── user_review
│ │ ├── scoring
│ │ └── data_protection_level
│ │
│ └── action_items/ # Standalone action items
│ └── {action_item_id}/
│ ├── description
│ ├── completed
│ ├── conversation_id
│ ├── created_at
│ ├── due_at
│ └── completed_at
```
***
## Part 1: Storing Conversations
### Processing Flow
The app sends a POST request to `/v1/conversations` with transcript data
`process_conversation()` in `utils/conversations/process_conversation.py` handles the logic
LLM extracts title, overview, action items, and events from the transcript
`upsert_conversation()` in `database/conversations.py` saves to Firestore
Conversation is embedded and stored in Pinecone for semantic search
### Conversation Model Fields
| Field | Type | Description |
| - | - | - |
| `id` | string | Unique conversation identifier |
| `created_at` | datetime | When the conversation record was created |
| `started_at` | datetime | When the actual conversation started |
| `finished_at` | datetime | When the conversation ended |
| `source` | enum | Source device (omi, phone, desktop, openglass, etc.) |
| `language` | string | Language code of the conversation |
| `status` | enum | Processing status: `in_progress`, `processing`, `completed`, `failed` |
| `structured` | object | Extracted structured information (see below) |
| `transcript_segments` | array | List of transcript segments |
| `geolocation` | object | Location data (latitude, longitude, address) |
| `photos` | array | Photos captured during conversation |
| `audio_files` | array | Audio file references |
| `apps_results` | array | Results from summarization apps |
| `external_data` | object | Data from external integrations |
| `discarded` | boolean | Whether conversation was marked as low-quality |
| `visibility` | enum | `private`, `shared`, or `public` |
| `is_locked` | boolean | Whether conversation is locked from editing |
| `data_protection_level` | string | `standard` or `enhanced` (encrypted) |
### Structured Information
The `structured` field contains LLM-extracted information:
| Field | Type | Description |
| - | - | - |
| `title` | string | Short descriptive title for the conversation |
| `overview` | string | Summary of key points discussed |
| `emoji` | string | Emoji representing the conversation |
| `category` | enum | Category (personal, work, health, etc.) |
| `action_items` | array | Tasks or to-dos mentioned |
| `events` | array | Calendar events to be created |
### Transcript Segments
Each segment in `transcript_segments` includes:
| Field | Type | Description |
| - | - | - |
| `text` | string | Transcribed text content |
| `speaker` | string | Speaker label (e.g., "SPEAKER\_00") |
| `start` | float | Start time in seconds |
| `end` | float | End time in seconds |
| `is_user` | boolean | Whether spoken by the device owner |
| `person_id` | string | ID of identified person (if matched) |
### Action Items
Action items are stored both inline (in `structured.action_items`) and in a standalone collection:
| Field | Type | Description |
| - | - | - |
| `description` | string | The action item text |
| `completed` | boolean | Whether the item is done |
| `created_at` | datetime | When extracted |
| `due_at` | datetime | Optional due date |
| `completed_at` | datetime | When marked complete |
| `conversation_id` | string | Source conversation |
### Events
Calendar events extracted from conversations:
| Field | Type | Description |
| - | - | - |
| `title` | string | Event title |
| `description` | string | Event description |
| `start` | datetime | Start date/time |
| `duration` | integer | Duration in minutes |
| `created` | boolean | Whether added to calendar |
***
## Part 2: Extracting & Storing Memories
Memories are **facts about the user** extracted from conversations. They represent learnings, preferences, habits, and other personal information.
### Memory Extraction Process
During `process_conversation()`, the system:
Reviews the conversation transcript for personal information
Identifies facts worth remembering about the user (\~15 words max)
Saves to `memories` collection with a link back to the source conversation
### Memory Model Fields
| Field | Type | Description |
| - | - | - |
| `id` | string | Unique memory identifier |
| `uid` | string | User ID |
| `conversation_id` | string | Source conversation (links back) |
| `content` | string | The actual fact/learning (max \~15 words) |
| `category` | enum | `interesting`, `system`, or `manual` |
| `tags` | array | Categorization tags |
| `visibility` | string | `private` or `public` |
| `created_at` | datetime | When memory was created |
| `updated_at` | datetime | Last modification time |
| `reviewed` | boolean | Whether user has reviewed |
| `user_review` | boolean | User's approval (true/false/null) |
| `edited` | boolean | Whether user edited the content |
| `scoring` | string | Ranking score for retrieval |
| `manually_added` | boolean | Whether user created manually |
| `is_locked` | boolean | Prevent automatic deletion |
| `app_id` | string | Source app (if from integration) |
| `data_protection_level` | string | Encryption level |
### Memory Categories
Notable facts about the user: hobbies, opinions, stories
Preferences and patterns: work habits, sleep schedule
User-created memories: explicitly added facts
Legacy categories (`core`, `hobbies`, `lifestyle`, `interests`, `habits`, `work`, `skills`, `learnings`, `other`) are automatically mapped to the new primary categories for backward compatibility.
### Memory Extraction Rules
The system follows these guidelines when extracting memories:
* Maximum \~15 words per memory
* Must pass the "shareability test" - would this be worth telling someone?
* Maximum 2 `interesting` + 2 `system` memories per conversation
* No duplicate or near-duplicate facts
* Skip mundane details (eating, sleeping, commuting)
***
## Part 3: Data Protection & Encryption
Both conversations and memories support encryption for sensitive data.
### Standard Protection Level
No encryption, stored as plaintext. This is the default for most users.
* Fastest read/write performance
* Data visible in Firestore console
* Suitable for general use
### Enhanced Protection Level
AES encryption for sensitive fields. Provides additional security for sensitive conversations.
**Encrypted Fields:**
* **Conversations**: `transcript_segments` (the actual transcript text)
* **Memories**: `content` (the memory text)
Enhanced encryption adds processing overhead to read/write operations.
### Implementation
```python theme={null}
# Conversations: database/conversations.py
def _prepare_conversation_for_write(conversation_data, data_protection_level):
if data_protection_level == 'enhanced':
# Encrypt transcript_segments before storage
...
def _prepare_conversation_for_read(conversation_data, data_protection_level):
if data_protection_level == 'enhanced':
# Decrypt transcript_segments after retrieval
...
```
***
## Part 4: Vector Embeddings
Conversations are also stored as vector embeddings in Pinecone for semantic search.
### What Gets Embedded
| Data | Embedded? | Stored in Metadata? |
| - | - | - |
| Title | Yes | No |
| Overview | Yes | No |
| Action Items | Yes | No |
| Full Transcript | No (too large) | No |
| People Mentioned | No | Yes |
| Topics | No | Yes |
| Entities | No | Yes |
| created\_at | No | Yes |
### Vector Creation
Vectors are created in a background thread after conversation processing:
```python theme={null}
# utils/conversations/process_conversation.py
threading.Thread(
target=save_structured_vector,
args=(uid, conversation)
).start()
```
The `save_structured_vector()` function:
1. Generates embedding from `conversation.structured` (title + overview + action\_items + events)
2. Extracts metadata via LLM (people, topics, entities, dates)
3. Upserts to Pinecone with metadata filters
Vectors are created ONCE during initial processing. Reprocessed conversations do NOT update their vectors.
***
## Key Code Locations
| Component | File Path |
| - | - |
| **Conversation Model** | `backend/models/conversation.py` |
| **Memory Model** | `backend/models/memories.py` |
| **Process Conversation** | `backend/utils/conversations/process_conversation.py` |
| **Database - Conversations** | `backend/database/conversations.py` |
| **Database - Memories** | `backend/database/memories.py` |
| **Router - Conversations** | `backend/routers/conversations.py` |
| **Router - Memories** | `backend/routers/memories.py` |
| **Vector Database** | `backend/database/vector_db.py` |
***
## API Endpoints
### Conversations
| Method | Endpoint | Description |
| - | - | - |
| POST | `/v1/conversations` | Process and store a new conversation |
| GET | `/v1/conversations` | List user's conversations |
| GET | `/v1/conversations/{id}` | Get specific conversation |
| PATCH | `/v1/conversations/{id}/title` | Update conversation title |
| DELETE | `/v1/conversations/{id}` | Delete a conversation |
### Memories
| Method | Endpoint | Description |
| - | - | - |
| POST | `/v3/memories` | Create a manual memory |
| GET | `/v3/memories` | List user's memories |
| PATCH | `/v3/memories/{id}` | Edit a memory |
| DELETE | `/v3/memories/{id}` | Delete a memory |
| PATCH | `/v3/memories/{id}/visibility` | Change memory visibility |
***
## Related Documentation
How conversations are retrieved for chat using LangGraph
WebSocket-based real-time speech-to-text
General backend architecture overview
Environment setup and configuration
### Manual speaker attribution during concurrent writes
Manual speaker decisions live in a protected sparse receipt alongside the transcript.
Live STT merging and both processing-result writers read that receipt in their
commit transaction. Live merging preserves every manually covered speech span,
including speaker-wide defaults, explicit clears, and training opt-outs. The
committed transcript and deleted-ID delta are returned together for client delivery.
Translation and inference-only writers patch current IDs without owning the segment
set. Privacy-level migration re-encodes the transcript and receipt together.
Changing attribution does not automatically regenerate a summary; refresh remains
an explicit user action. Enrollment remains best-effort.
# Backend Deep Dive
Source: https://docs.omi.me/doc/developer/backend/backend_deepdive
Welcome to the Omi backend! This document provides a comprehensive overview of Omi's architecture and code, guiding you through its key components, functionalities, and how it all works together to power a unique and intelligent AI assistant experience.
The Deepgram-specific sections in this historical architecture deep dive are retired. Serving transcription uses Parakeet and Modulate only; see [Transcription](/doc/developer/backend/transcription) for the current provider policy.
## Understanding the Omi Ecosystem
Omi is a multimodal AI assistant designed to understand and interact with users in a way that's both intelligent and human-centered. The backend plays a crucial role in this by:
* **Processing and analyzing data:** Converting audio to text, extracting meaning, and creating structured information from user interactions.
* **Storing conversations and extracting memories:** Building a rich knowledge base of user conversations and extracted facts that Omi can draw upon to provide context and insights.
* **Facilitating intelligent chat:** Understanding user requests, retrieving relevant conversations, and generating personalized responses using an agentic AI system.
* **Integrating with external services:** Extending Omi's capabilities and connecting it to other tools and platforms.
This deep dive will walk you through the **core elements** of Omi's backend, providing a clear roadmap for developers and enthusiasts alike to understand its inner workings.
Jump to the [Quick Reference](#quick-reference) table to find what you need fast.
Read through the complete documentation below to understand the architecture.
See the [System Architecture](#system-architecture) flowchart and other diagrams throughout.
## System Architecture
```mermaid theme={null}
flowchart TD
subgraph Client
App["🔵 Omi App
(Flutter)"]
end
subgraph API["API Endpoints"]
WS["/v4/listen
WebSocket"]
Conv["/v1/conversations
REST"]
Chat["/v2/messages
REST"]
end
subgraph Routers
Trans["routers/transcribe.py"]
ConvR["routers/conversations.py"]
ChatR["routers/chat.py"]
end
subgraph Processing
DG["Deepgram API
Speech-to-Text"]
PC["process_conversation()"]
LG["LangGraph
Agentic System"]
end
subgraph Storage["Data Storage"]
FS[("Firestore
Conversations & Memories")]
Pine[("Pinecone
Vector Embeddings")]
Redis[("Redis
Cache & Metadata")]
GCS[("Google Cloud Storage
Binary Files")]
end
App --> WS
App --> Conv
App --> Chat
WS --> Trans
Conv --> ConvR
Chat --> ChatR
Trans --> DG
ConvR --> PC
ChatR --> LG
PC --> FS
PC --> Pine
PC --> GCS
LG --> FS
LG --> Pine
LG --> Redis
Trans --> GCS
```
## Quick Reference
Firestore CRUD operations for storing and retrieving conversations
Pinecone embeddings for semantic similarity search
OpenAI integrations for structure extraction and chat
Google Cloud Storage for audio files
High-speed caching for profiles and preferences
Real-time speech-to-text with Deepgram
| Need to... | Go to |
| - | - |
| Store a conversation | `database/conversations.py` |
| Query similar conversations | `database/vector_db.py` |
| Process LLM calls | `utils/llm/` directory |
| Handle real-time audio | `routers/transcribe.py` |
| Manage caching | `database/redis_db.py` |
| Understand chat system | [Chat System Architecture](/doc/developer/backend/chat_system) |
| Learn data models | [Storing Conversations](/doc/developer/backend/StoringConversations) |
## The Flow of Information
Let's trace the journey of a typical interaction with Omi, focusing on how audio recordings are transformed into stored conversations:
The user starts a recording session using the Omi app, capturing a conversation or their thoughts.
The Omi app establishes a real-time connection with the backend at the `/v4/listen` endpoint in `routers/transcribe.py`.
The app streams audio data continuously through the WebSocket to the backend.
The backend forwards audio to Deepgram API for real-time speech-to-text conversion.
Transcription results stream back through the backend to the app, displaying words as the user speaks.
During the WebSocket connection, the backend creates an "in\_progress" conversation stub in Firestore. As audio streams, transcript segments are continuously added to Firestore in real-time. When recording ends, the app sends a POST request to `/v1/conversations` with an empty body (`{}`). The backend retrieves the in-progress conversation from Firestore and processes it.
The `process_conversation` function uses OpenAI to extract:
* **Title & Overview** - Summarizes the conversation
* **Action Items** - Tasks and to-dos mentioned
* **Events** - Calendar-worthy moments
* **Memories** - Facts about the user
* **Firestore**: Stores the full conversation document with transcript segments and metadata
* **Pinecone**: Stores the vector embedding for semantic search
* **Redis**: Caches frequently accessed data (speech profile durations, enabled apps, user names) for performance
* **Google Cloud Storage**: Stores binary files (speech profile audio, conversation recordings, photos)
### What Gets Extracted
| Field | Description |
| - | - |
| `title` | A short, descriptive title |
| `overview` | Concise summary of main points |
| `category` | Work, personal, etc. |
| `action_items` | Tasks or to-dos mentioned |
| `events` | Calendar-worthy events |
| `memories` | Facts about the user (stored separately) |
Use the Quick Reference table above to jump directly to the component you need. Each section below includes key functions and usage patterns.
## Core Components
Now that you understand the general flow, let's dive deeper into the key modules and services that power Omi's backend.
### 1. `database/conversations.py`: The Conversation Guardian
This module is responsible for managing the interaction with Firebase Firestore, Omi's main database for storing conversations and related data.
**Key Functions:**
* `upsert_conversation`: Creates or updates a conversation document in Firestore, ensuring efficient storage and handling of updates.
* `get_conversation`: Retrieves a specific conversation by its ID.
* `get_conversations`: Fetches a list of conversations for a user, allowing for filtering, pagination, and optional inclusion of discarded conversations.
* **Photo Functions:** Handles the storage and retrieval of photos associated with conversations.
**Firestore Structure:**
Each conversation is stored as a document in Firestore with the following fields:
```python theme={null}
class Conversation(BaseModel):
id: str # Unique ID
created_at: datetime # Creation timestamp
started_at: Optional[datetime]
finished_at: Optional[datetime]
source: Optional[ConversationSource] # omi, phone, desktop, openglass, etc.
language: Optional[str]
status: ConversationStatus # in_progress, processing, completed, failed
structured: Structured # Contains extracted title, overview, action items, etc.
transcript_segments: List[TranscriptSegment]
geolocation: Optional[Geolocation]
photos: List[ConversationPhoto]
apps_results: List[AppResult]
external_data: Optional[Dict]
discarded: bool
deleted: bool
visibility: str # private, shared, public
# See StoringConversations.mdx for complete field reference
```
#### Conversation Lifecycle
```mermaid theme={null}
stateDiagram-v2
[*] --> in_progress: Recording starts
(WebSocket connects)
in_progress --> processing: Recording ends
(POST /v1/conversations
or timeout)
processing --> completed: Successfully processed
processing --> failed: Error occurred
completed --> [*]
failed --> processing: Retry
note right of processing
LLM extracts structure
Memories extracted
Embeddings generated
end note
```
**Processing Triggers:**
* **Manual:** App sends `POST /v1/conversations` with empty body to trigger immediate processing
* **Automatic:** Backend automatically processes conversations after a timeout period (`conversation_timeout` parameter, default 120 seconds of silence)
* Both paths use the same `process_conversation()` function to extract structure, memories, and embeddings
### 2. `database/vector_db.py`: The Embedding Expert
This module manages the interaction with Pinecone, a vector database used to store and query conversation embeddings.
**Key Functions:**
* `upsert_vector`: Adds or updates a conversation embedding in Pinecone.
* `upsert_vectors`: Efficiently adds or updates multiple embeddings.
* `query_vectors`: Performs similarity search to find conversations relevant to a user query.
* `delete_vector`: Removes a conversation embedding.
**Pinecone's Role:**
Pinecone's specialized vector search capabilities are essential for:
* **Contextual Retrieval:** Finding conversations that are semantically related to a user's request, even if they don't share exact keywords.
* **Efficient Search:** Quickly retrieving relevant conversations from a large collection.
* **Scalability:** Handling the growing number of conversation embeddings as the user creates more conversations.
### 3. `utils/llm/` Directory: The AI Maestro
This directory contains modules where the power of OpenAI's LLMs is harnessed for a wide range of tasks. It's the core of Omi's intelligence!
**Key Files:**
* `clients.py`: LLM client configurations and embedding models
* `chat.py`: Chat-related prompts and processing
* `conversation_processing.py`: Conversation analysis and structuring
**Key Functionalities:**
* **Conversation Processing:**
* Determines if a conversation should be discarded.
* Extracts structured information from transcripts (title, overview, categories, etc.).
* Runs apps on conversation data.
* **External Integration Processing:**
* Creates structured summaries from photos and descriptions (OpenGlass).
* Processes data from external sources to generate conversations.
* **Chat and Retrieval:**
* Generates initial chat messages.
* Analyzes chat conversations to determine if context is needed.
* Extracts relevant topics and dates from chat history.
* Retrieves and summarizes relevant conversation content for chat responses.
* **Emotional Processing:**
* Analyzes conversation transcripts for user emotions.
* Generates emotionally aware responses based on context and user facts.
* **Fact Extraction:** Identifies and extracts new facts (memories) about the user from conversation transcripts.
**OpenAI Integration:**
* The LLM modules use a provider-agnostic model system (see `utils/llm/model_config.py`) with models from OpenAI (`gpt-5.4`, `gpt-5.4-mini`, `gpt-4.1-mini`), Anthropic (`claude-sonnet-4-6`), and others, selected per task via QoS profiles (`premium`, `max`, `fair_use`).
* `text-embedding-3-large` is used to generate vector embeddings for conversations and user queries.
**Why this is Essential:**
* **The Brain of Omi:** These modules enable Omi's core AI capabilities, including natural language understanding, content generation, and context-aware interactions.
* **Conversation Enhancement:** They enrich raw data by extracting meaning and creating structured information.
* **Personalized Responses:** They help Omi provide responses tailored to individual users, incorporating their unique facts, conversations, and emotional states.
* **Extensibility:** The app system and integration with external services make Omi highly versatile.
For detailed chat system architecture including LangGraph routing and the agentic tool system, see [Chat System Architecture](/doc/developer/backend/chat_system).
### 4. `utils/other/storage.py`: The Cloud Storage Manager
This module handles interactions with Google Cloud Storage (GCS), specifically for managing user speech profiles.
**Key Functions:**
* **`upload_profile_audio(file_path: str, uid: str)`:**
* Uploads a user's speech profile audio recording to the GCS bucket specified by the `BUCKET_SPEECH_PROFILES` environment variable.
* Organizes audio files within the bucket using the user's ID (`uid`).
* Returns the public URL of the uploaded file.
* **`get_profile_audio_if_exists(uid: str) -> str`:**
* Checks if a speech profile already exists for a given user ID in the GCS bucket.
* Downloads the speech profile audio to a local temporary file if it exists and returns the file path.
* Returns `None` if the profile does not exist.
**Usage:**
* The `upload_profile_audio` function is called when a user uploads a new speech profile recording through the `/v3/upload-audio` endpoint (defined in `routers/speech_profile.py`).
* The `get_profile_audio_if_exists` function is used to retrieve a user's speech profile when needed, for example, during speaker identification in real-time transcription or post-processing.
### 5. `database/redis_db.py`: The Data Speedster
Redis is optional for local development. The backend will work without it, but features like speech profiles and app preferences caching will be disabled.
Redis is an in-memory data store known for its speed and efficiency. The `database/redis_db.py` module handles Omi's interactions with Redis, which is primarily used for caching, managing user
settings, and storing user speech profiles.
**Data Stored and Retrieved from Redis:**
* **User Speech Profile Metadata:**
* **Storage:** When a user uploads a speech profile, the audio file is stored in Google Cloud Storage, while the duration is cached in Redis for quick access.
* **Retrieval:** During real-time transcription or post-processing, the speech profile duration is retrieved from Redis cache, while the actual audio file is loaded from Google Cloud Storage when needed for speaker identification.
* **Enabled Apps:**
* **Storage:** A set of app IDs is stored for each user, representing the apps they have enabled.
* **Retrieval:** When processing a conversation or handling a chat request, the backend checks Redis to see which apps are enabled for the user.
* **App Reviews:**
* **Storage:** Reviews for each app (score, review text, date) are stored in Redis, organized by app ID and user ID.
* **Retrieval:** When displaying app information, the backend retrieves reviews from Redis.
* **Cached User Names:**
* **Storage:** User names are cached in Redis to avoid repeated lookups from Firebase.
* **Retrieval:** The backend first checks Redis for a user's name before querying Firestore, improving performance.
**Key Functions:**
| Function | Purpose |
| - | - |
| `set_speech_profile_duration` | Cache speech profile duration (audio stored in GCS) |
| `enable_app`, `disable_app` | Manage app enable/disable states |
| `get_enabled_apps` | Get user's enabled apps |
| `get_app_reviews` | Retrieve reviews for an app |
| `cache_user_name` | Cache user names |
**Storage Separation:** Redis is used for caching metadata (durations, flags, enabled apps) to improve performance. The actual speech profile audio files are stored in Google Cloud Storage via `utils/other/storage.py`. This separation ensures Redis remains fast and lightweight while GCS handles binary file storage.
**Why Redis is Important:**
* **Performance:** Caching data in Redis significantly improves the backend's speed, as frequently accessed data can be retrieved from memory very quickly.
* **User Data Management:** Redis provides a flexible and efficient way to manage user-specific metadata, such as app preferences, speech profile durations, and enabled apps.
* **Real-time Features:** The low-latency nature of Redis makes it ideal for supporting real-time features like live transcription and instant app interactions.
* **Scalability:** As the number of users grows, Redis helps maintain performance by reducing the load on primary databases (Firestore) and storage systems (GCS).
### 6. `routers/transcribe.py`: The Real-Time Transcription Engine
This module is the powerhouse behind Omi's real-time transcription capabilities, allowing the app to convert spoken audio into text as the user is speaking. It leverages WebSockets for bidirectional
communication with the Omi app and Deepgram for accurate and efficient transcription.
#### 1. WebSocket Communication
WebSocket connections require proper authentication. The `uid` parameter must be a valid Firebase user ID, and connections without proper auth will be rejected.
* **`/v4/listen` Endpoint:** The Omi app initiates a WebSocket connection with the backend at the `/v4/listen` endpoint, which is defined in the `websocket_endpoint` function of `routers/transcribe.py`.
* **Bidirectional Communication:** WebSockets enable a two-way communication channel, allowing:
* The Omi app to stream audio data to the backend continuously.
* The backend to send back transcribed text segments as they become available from Deepgram.
* **Real-Time Feedback:** This constant back-and-forth ensures that users see their words being transcribed in real-time, creating a more interactive and engaging experience.
#### 2. STT Service Integration
Omi uses Deepgram as its Speech-to-Text (STT) service:
* **Deepgram** (Nova-3) - Primary service for all languages
The backend uses `process_audio_dg()` in `utils/stt/streaming.py` for all transcription.
**Processing Functions:**
* **`process_audio_dg()`** - Manages interaction with Deepgram API (found in `utils/stt/streaming.py`)
The audio chunks streamed from the Omi app are sent to the selected STT service API for transcription. The service's speech recognition models process the audio and return text results in real-time.
**Deepgram Options Configuration:** The `process_audio_dg` function configures various Deepgram options:
Automatically adds punctuation to transcribed text for readability. Essential for creating natural-looking transcripts.
Minimizes latency for real-time feedback. This is essential for live transcription where users expect to see words appear as they speak.
Sets the language for transcription (e.g., 'en', 'es', 'fr'). The user can specify their preferred language when starting a recording.
Controls whether to send interim (partial) transcription results or only final results. Set to `False` in production for cleaner output.
Enables speaker diarization - identifying different speakers in the audio. Critical for multi-person conversations to attribute text to the correct speaker.
Audio format settings for Deepgram compatibility. These must match the audio format being sent from the app.
#### 3. Transcription Flow
```mermaid theme={null}
sequenceDiagram
participant App as 📱 Omi App
participant Backend as 🖥️ Backend
participant STT as 🎧 STT Service
(Deepgram)
App->>Backend: Connect WebSocket /v4/listen
Backend-->>App: Connection accepted
loop Real-time Transcription
App->>Backend: Audio chunk (binary)
Backend->>STT: Forward audio
STT-->>Backend: Transcription result
Backend-->>App: Text segment + speaker
Note right of App: Display words
as user speaks
end
App->>Backend: Close connection
Backend-->>App: Session ended
```
The numbered breakdown of this flow:
1. **App Streams Audio:** The Omi app captures audio from the user's device and continuously sends chunks of audio data through the WebSocket to the backend's `/v4/listen` endpoint.
2. **Backend Receives Audio:** The backend's `websocket_endpoint` function receives the audio chunks.
3. **Backend Forwards to STT:** The backend immediately forwards audio chunks to Deepgram using `process_audio_dg`.
4. **STT Processes:** The STT service's speech recognition models transcribe the audio data in real-time.
5. **Results Sent Back:** The STT service sends the transcribed text segments back to the backend as they become available.
6. **Backend Relays to App:** The backend immediately sends these transcription results back to the Omi app over the WebSocket connection.
7. **App Displays Transcript:** The Omi app updates the user interface with the newly transcribed text, providing instant feedback.
#### 4. Key Considerations
* **Speaker Identification:** The code uses Deepgram's speaker diarization feature to identify different speakers in the audio. This information is included in the transcription results, allowing the app to display who said what.
* **User Speech Profile Integration:** If a user has uploaded a speech profile, the backend can use this information (duration cached in Redis, audio file stored in Google Cloud Storage) to improve the accuracy of speaker identification.
* **Latency Management:** Real-time transcription requires careful attention to latency to ensure a seamless user experience. The `no_delay` option in Deepgram and the efficient handling of data in the backend are essential for minimizing delays.
* **Error Handling:** The code includes error handling mechanisms to gracefully handle any issues that may occur during the WebSocket connection or STT service transcription process.
#### 5. Example Code Snippet (Simplified):
```python theme={null}
from fastapi import APIRouter, WebSocket
# ... other imports ...
router = APIRouter()
@router.websocket("/v4/listen")
async def websocket_endpoint(websocket: WebSocket, uid: str, language: str = 'en', ...):
await websocket.accept() # Accept the WebSocket connection
# Start STT transcription via Deepgram
transcript_socket = await process_audio_dg(uid, websocket, language, ...)
# Receive and process audio chunks from the app
async for data in websocket.iter_bytes():
transcript_socket.send(data)
# ... other logic for speaker identification, error handling, etc.
```
## Daily summary settings
`GET /v1/users/daily-summary-settings` returns whether summaries are enabled and
their delivery hour in the user's local timezone. Use
`PATCH /v1/users/daily-summary-settings` to change `enabled` or `hour` (0–23).
Hour `0` means midnight; it is not an unset value.
Timezone updates (`PUT /v1/users/time-zone`) and push-token registration populate
missing schedule fields with the existing defaults: enabled and 22:00. They
preserve explicit settings, including an opt-out or midnight, even when another
client changes them concurrently. The backend computes missing defaults and
persists them in one transaction. Desktop-only timezone seeding similarly fills
an absent zone without replacing a client-reported one.
## Related Documentation
For more detailed information on specific subsystems:
```mermaid theme={null}
mindmap
root((Backend Deep Dive))
Conversations
StoringConversations.mdx
Data Models
Firestore Structure
Chat System
chat_system.mdx
LangGraph
Agentic Tools
Transcription
transcription.mdx
Deepgram
WebSockets
Setup
Backend_Setup.mdx
Environment
Dependencies
```
Complete data model and storage architecture for conversations and extracted memories
LangGraph routing, agentic tools, and vector search deep dive
Detailed transcription pipeline and Deepgram configuration
Environment setup, dependencies, and local development guide
***
## Need Help?
Report bugs or request features
Get help from the community
Learn how to contribute to Omi
Browse the complete docs
# Chat System Architecture
Source: https://docs.omi.me/doc/developer/backend/chat_system
A comprehensive technical deep dive into how Omi's intelligent chat system processes questions, routes them through LangGraph, calls tools, and generates contextual responses.
## Overview
Omi's chat system is a sophisticated **agentic AI pipeline** that enables users to have intelligent conversations about their recorded memories, calendar events, health data, and more. This document provides a complete technical understanding of how questions flow through the system.
Determines if context is needed
Simple, Agentic, or Persona path
22+ integrated data sources
Vector search & metadata filters
Links to source conversations
Real-time thinking & response
## System Architecture Diagram
```mermaid theme={null}
flowchart TD
subgraph Client["📱 Flutter App"]
Q[User Question]
end
subgraph Backend["🖥️ FastAPI Backend"]
Router{LangGraph
Router}
end
Q --> Router
Router -->|Simple| NC[No Context Path]
Router -->|Context Needed| A[Agentic Path]
Router -->|Persona App| P[Persona Path]
NC --> LLM1[Direct LLM
Response]
A --> Tools[Tool Calls
22+ tools]
Tools --> LLM2[LLM with
Context]
P --> LLM3[Persona LLM
Response]
LLM1 --> Stream[Streaming Response]
LLM2 --> Stream
LLM3 --> Stream
Stream --> |with citations| Q
```
## Desktop message persistence
`POST /v2/desktop/messages` persists completed desktop chat messages; it does not generate or stream replies. For a new message, `database/chat.py::save_message` commits the message and the existing session's `message_count`, `preview` (first 100 characters), and `updated_at` together in one Firestore transaction. A missing session is not recreated. Explicit transaction contention is retried with a bounded budget and fresh transactions; unrelated or ambiguous provider errors remain errors.
Clients should keep the same `client_message_id` when retrying. A matching replay returns the stored message without incrementing the session again; changed payloads conflict, while higher journal revisions retain their existing enrichment behavior. Without a client ID, a retry after a lost acknowledgment can still create a second message. This boundary prevents new partial saves; it does not repair historical session counts or make session acquisition and quota recording part of the message transaction.
## The Three Routing Paths
### Path 1: No Context Conversation
**When triggered:** Simple greetings, general advice, brainstorming questions
**Classification criteria** (from `requires_context()` function):
* Greetings: "Hi", "Hello", "How are you?"
* General knowledge: "What's the capital of France?"
* Advice without personal context: "Tips for productivity"
**Processing:**
```python theme={null}
# Location: backend/utils/retrieval/graph.py
def no_context_conversation(state: ChatState, config: RunnableConfig):
# Direct LLM call without tool access
# Fast response, no memory retrieval
```
This path provides the fastest responses since no external data retrieval is needed.
### Path 2: Agentic Context-Dependent Conversation
**When triggered:** Questions requiring personal data, temporal queries, integration lookups
**Classification criteria:**
* References to "my", "I", personal data
* Temporal references: "yesterday", "last week", "this morning"
* Questions about conversations, memories, calendar, health
* Requests involving connected services
**Processing:**
```python theme={null}
# Location: backend/utils/retrieval/agentic.py
# LangGraph ReAct agent with full tool access
# LLM autonomously decides which tools to call
# Can make multiple tool calls to gather comprehensive context
```
This is the most powerful path - the LLM can call 22+ tools to gather comprehensive context before answering.
### Path 3: Persona Question
**When triggered:** Questions directed at persona-based apps (e.g., "Ask Einstein")
**Processing:**
* Uses the app's configured `persona_prompt`
* Character-consistent responses
* May have limited tool access based on app configuration
Persona apps can customize which tools are available, allowing for focused conversational experiences.
## Classification Logic
The `requires_context()` function determines the routing path:
```python theme={null}
# Location: backend/utils/retrieval/graph.py
def requires_context(messages: list) -> bool:
"""
Uses GPT-4-mini for fast classification.
Returns True if question needs:
- Personal memories/conversations
- Calendar/email/health data
- Temporal context
- User-specific information
"""
```
## The Agentic Tool System
### How Tool Calling Works
The LangGraph ReAct agent follows this cycle:
System prompt provides tool descriptions, user's timezone, and citation instructions
LLM autonomously decides which tool(s) to call based on question intent
Tool calls are executed and results returned to the agent
Agent synthesizes response OR makes additional tool calls if more context needed
Final answer generated with proper `[1][2]` citations linking to source conversations
### Available Tools (22+)
Tools are loaded dynamically based on user's enabled integrations and installed apps.
Core tools for retrieving user's conversations and extracted memories.
| Tool | Purpose | Key Parameters |
| - | - | - |
| `get_conversations_tool` | Retrieve by date range | `start_date`, `end_date`, `limit`, `include_transcript` |
| `search_conversations_tool` | Semantic search | `query`, `start_date`, `end_date`, `limit` |
| `get_memories_tool` | Personal facts about user | `limit`, `offset` |
Manage tasks and to-dos extracted from conversations.
| Tool | Purpose |
| - | - |
| `get_action_items_tool` | Retrieve pending tasks |
| `create_action_item_tool` | Create new task |
| `update_action_item_tool` | Mark complete/update |
Full CRUD operations on user's Google Calendar.
| Tool | Purpose |
| - | - |
| `get_calendar_events_tool` | Fetch events by date/person |
| `create_calendar_event_tool` | Create meetings with attendees |
| `update_calendar_event_tool` | Modify existing events |
| `delete_calendar_event_tool` | Cancel meetings |
Connect to external services for richer context.
| Tool | Service | Purpose |
| - | - | - |
| `get_gmail_messages_tool` | Gmail | Search emails |
| `get_whoop_sleep_tool` | Whoop | Sleep data |
| `get_whoop_recovery_tool` | Whoop | Recovery scores |
| `get_whoop_workout_tool` | Whoop | Workout history |
| `search_notion_pages_tool` | Notion | Search workspace |
| `get_twitter_tweets_tool` | Twitter/X | Recent tweets |
| `get_github_pull_requests_tool` | GitHub | Open PRs |
| `get_github_issues_tool` | GitHub | Open issues |
| `perplexity_web_search_tool` | Perplexity | Web search |
Third-party apps can define custom tools that become available when users enable them.
```python theme={null}
# Location: backend/utils/retrieval/tools/app_tools.py
def load_app_tools(uid: str) -> List[Callable]:
"""
Loads tools from user's enabled apps.
Each app can define chat_tools in its configuration.
"""
```
See [Chat Tools for Apps](/doc/developer/apps/ChatTools) to learn how to build custom tools.
### Safety Guards
```python theme={null}
# Maximum 10 tool calls per question (prevents runaway loops)
# Maximum 500K tokens in context (prevents context overflow)
# 30-second timeout per external API call
```
## Vector Search Deep Dive
```mermaid theme={null}
sequenceDiagram
participant Agent as 🤖 LLM Agent
participant Tool as 🔧 Vector Search Tool
participant Embed as 📊 OpenAI Embeddings
participant Pine as 🌲 Pinecone
participant Fire as 🔥 Firestore
Agent->>Tool: search_conversations_tool(query, dates)
Tool->>Embed: embed_query("John project discussion")
Embed-->>Tool: [0.012, -0.034, 0.056, ...] (3,072 dims)
Tool->>Pine: query(vector, uid filter, date range)
Pine-->>Tool: [conv_id_456, conv_id_789] ranked by similarity
Tool->>Fire: get_conversations_by_id(ids)
Fire-->>Tool: Full conversation data
Tool-->>Agent: Formatted context with citations
```
### Configuration
| Setting | Value |
| - | - |
| Database | Pinecone (serverless) |
| Embedding Model | `text-embedding-3-large` (OpenAI) |
| Vector Dimensions | 3,072 |
| Namespace | `"ns1"` |
| Vector ID Format | `{uid}-{conversation_id}` |
### What Gets Embedded vs Stored as Metadata
| Data | Embedded? | Metadata? |
| - | - | - |
| Title | Yes | No |
| Overview/Summary | Yes | No |
| Action Items | Yes | No |
| Full Transcript | No (too large) | No |
| People Mentioned | No | Yes |
| Topics | No | Yes |
| Entities | No | Yes |
| Dates Mentioned | No | Yes |
| `created_at` | No | Yes (Unix timestamp) |
### Vector Creation (Write Path)
```python theme={null}
# Location: backend/utils/conversations/process_conversation.py
# Triggered after conversation processing completes
def save_structured_vector(uid: str, conversation: Conversation):
"""
1. Generate embedding from conversation.structured
(title + overview + action_items + events)
2. Extract metadata via LLM (people, topics, entities, dates)
3. Upsert to Pinecone with metadata
"""
```
Vectors are created ONCE during initial processing, not on every edit. Reprocessed conversations do NOT create new vectors.
### Vector Query (Read Path)
```python theme={null}
# Location: backend/database/vector_db.py
def query_vectors(query: str, uid: str, starts_at: int, ends_at: int, k: int):
"""
1. Embed query using text-embedding-3-large
2. Query Pinecone with uid filter and optional date range
3. Return top-k conversation IDs ranked by similarity
"""
def query_vectors_by_metadata(uid, vector, dates_filter, people, topics, entities, dates, limit):
"""
Advanced query with metadata filters.
Includes fallback: if no results with filters, retries without them.
"""
```
## Memories System
Memories are distinct from Conversations. They are **structured facts** extracted about the user over time.
### Memory Categories
| Category | Examples |
| - | - |
| `interesting` | Hobbies, opinions, stories |
| `system` | Preferences, habits |
| `manual` | User-defined facts |
### Extraction Rules
```python theme={null}
# From backend/utils/llm/chat.py
# Maximum 15 words per memory
# Must pass "shareability test" - worth telling someone
# Max 2 interesting + 2 system memories per conversation
# NO duplicate/near-duplicate facts
# NO mundane details (eating, sleeping, commuting)
```
### Memory Retrieval in Chat
```python theme={null}
# Tool: get_memories_tool
# Returns formatted list of known facts about user
# Used when questions ask "What do you know about me?"
```
## Chat Sessions & Context
### Session Structure
```python theme={null}
# Location: backend/database/chat.py
# Chat sessions group related messages
# Each session tracks:
# - message_ids: List of message IDs
# - file_ids: Uploaded files for this session
# - openai_thread_id: For file-based chat
```
### Context Window
```python theme={null}
# Last 10 messages included in context
# Enables follow-up questions without re-stating context
# Older messages summarized or excluded
```
### Citation System
The LLM generates citations in `[1][2]` format:
```python theme={null}
# Citation rules:
# - No space before citation: "discussed this[1]" not "discussed this [1]"
# - Citations map to conversation IDs
# - Post-processing extracts citations → memories_id field
# - Frontend displays linked conversation cards
```
## System Prompt Structure
The main system prompt includes:
```python theme={null}
# Location: backend/utils/llm/chat.py - _get_agentic_qa_prompt()
# 1. Current datetime in user's timezone
# 2. Tool usage instructions
# 3. DateTime formatting rules for tool calls
# 4. Conversation retrieval strategies (5-step strategy)
# 5. Citation format instructions
# 6. Memory extraction guidelines
```
### DateTime Formatting Rules
Critical for correct tool behavior. All dates must use ISO format with timezone.
```python theme={null}
# Good: "2024-01-19T00:00:00-08:00"
# Bad: "yesterday", "last week" (must be converted)
# The system prompt instructs the LLM to convert relative
# references to absolute ISO timestamps before tool calls
```
### Conversation Retrieval Strategy
The system prompt guides the LLM through a 5-step strategy:
1. **Assess the question** - Determine type (temporal, topic, person, etc.)
2. **Choose primary tool** - `get_conversations` for date-based, `vector_search` for topic-based
3. **Apply filters** - Use start\_date/end\_date when temporal bounds are known
4. **Request transcripts** - Only when detailed content is needed
5. **Cite sources** - Always cite conversations used in the answer
## LLM Models Used
| Model | Use Case | Location |
| - | - | - |
| `gpt-5.4-mini` / `gpt-5.4` | Core chat, conversation processing | Premium / Max QoS profiles |
| `gpt-4.1-mini` | Fast classification, date extraction | `requires_context()`, filters |
| `claude-sonnet-4-6` | Chat agent (all QoS tiers) | `chat_agent` profile |
| `gemini-2.5-flash-lite` | Session titles, followups | Light tasks (premium) |
| `text-embedding-3-large` | Vector embeddings (3,072 dims) | Pinecone queries |
Models are selected via QoS profiles (`premium`, `max`, `fair_use`, `byok`) in `backend/utils/llm/model_config.py`. The `fair_use` profile uses `gpt-5.1` as a fallback. See the config file for the current authoritative set.
## Streaming Response Format
The backend streams responses in Server-Sent Events (SSE) format:
```
think: Searching conversations # Tool call indicator
data: Yesterday you discussed... # Response text chunks
done: {base64 encoded JSON} # Final message with metadata
```
The Flutter app parses these to show:
* Loading indicators with tool names
* Streaming response text
* Final message with linked memories
## Key File Locations
| Component | File Path |
| - | - |
| Chat Router | `backend/routers/chat.py` |
| LangGraph Router | `backend/utils/retrieval/graph.py` |
| Agentic System | `backend/utils/retrieval/agentic.py` |
| Tools Directory | `backend/utils/retrieval/tools/` |
| Conversation Tools | `backend/utils/retrieval/tools/conversation_tools.py` |
| Memory Tools | `backend/utils/retrieval/tools/memory_tools.py` |
| Calendar Tools | `backend/utils/retrieval/tools/calendar_tools.py` |
| App Tools Loader | `backend/utils/retrieval/tools/app_tools.py` |
| LLM Clients | `backend/utils/llm/clients.py` |
| Chat Prompts | `backend/utils/llm/chat.py` |
| Vector Database | `backend/database/vector_db.py` |
## Example: Question Flow
**User asks:** "What did I discuss with John yesterday about the project?"
`requires_context()` → **TRUE** (temporal + person + topic reference)
Route to: `agentic_context_dependent_conversation`
System prompt provides: current datetime, tool descriptions
Agent thinks: *"Need conversations from yesterday about project with John"*
Agent calls: `search_conversations_tool`
* `query`: "John project discussion"
* `start_date`: "2024-01-19T00:00:00-08:00"
* `end_date`: "2024-01-19T23:59:59-08:00"
1. Embed query → `[0.012, -0.034, 0.056, ...]`
2. Query Pinecone with uid filter + date range
3. Fetch full conversations from Firestore
4. Format for LLM context
LLM synthesizes answer with citations:
*"Yesterday you discussed the Q1 roadmap with John\[1]. He mentioned the frontend refactoring is ahead of schedule\[1]\[2]..."*
1. Extract citations → `memories_id: ["conv_456", "conv_789"]`
2. Save message to Firestore
3. Stream final response with linked conversation cards
## Related Documentation
Learn how to build custom chat tools for your Omi apps
How conversations and memories are stored
General backend architecture overview
WebSocket transcription and STT providers
## Messaging provider adapters
Telegram and iMessage integrations consume the same `run_chat_turn` service as app
chat, with a provider adapter for webhook verification, message normalization and
reply delivery. Each linked channel has its own chat session and uses the shared
memory and task tools. Files use the existing chat file store; voice messages use
the existing prerecorded speech-to-text service before becoming a chat turn.
Provider webhooks are acknowledged after durable inbox admission. A separate worker
runs the turn, checks the active account link before delivery, and records provider
receipts. The adapter contract tests and sanitized payload replay suite live under
`backend/testing/messaging/`. The manual dev harness is separate from CI and uses
dedicated test accounts. This source integration does not establish live availability.
# Real-time Transcription
Source: https://docs.omi.me/doc/developer/backend/transcription
A comprehensive guide to Omi's real-time audio transcription system, covering WebSocket connections, STT providers, speaker diarization, message formats, and building external custom STT services.
## Overview
Omi's transcription system provides **real-time speech-to-text** conversion with speaker identification, multiple language support, and seamless integration with the conversation processing pipeline.
```mermaid theme={null}
flowchart LR
subgraph Client["📱 Omi App"]
Audio[Audio Capture]
end
subgraph Backend["🖥️ Backend"]
WS["/v4/listen
WebSocket"]
Decode[Audio Decoder]
end
subgraph STT["🎧 STT Providers"]
Parakeet[Parakeet]
Modulate[Modulate Velma-2]
end
Audio -->|Binary stream| WS
WS --> Decode
Decode --> Parakeet
Decode --> Modulate
Parakeet -->|Transcript| WS
Modulate -->|Transcript| WS
WS -->|JSON segments| Audio
```
Connect to `/v4/listen` WebSocket with your user token and start streaming audio. Transcripts arrive in real-time as JSON.
Read through for complete endpoint details, configuration options, and message formats.
* Multiple STT providers with automatic fallback
* Speech profile for user identification
* Dual-socket architecture for speaker training
* [External Custom STT](#external-custom-stt-service) for your own transcription service
## WebSocket Endpoint
WebSocket connections require Firebase authentication. The `uid` parameter must be a valid user ID obtained through Firebase Auth.
### Endpoint URL
```
wss://api.omi.me/v4/listen?uid={uid}&language={lang}&sample_rate={rate}&codec={codec}
```
### Query Parameters
**Type:** `string`
User ID obtained from Firebase authentication. Required for all connections.
**Type:** `string` | **Default:** `'en'`
Language code for transcription. Supports:
* Standard codes: `'en'`, `'es'`, `'fr'`, `'de'`, `'ja'`, `'zh'`, etc.
* Multi-language: `'multi'` for automatic language detection
**Type:** `integer` | **Default:** `8000`
Audio sample rate in Hz. Common values: `8000`, `16000`, `44100`, `48000`
**Type:** `string` | **Default:** `'pcm8'`
Audio codec. Supported options:
* `pcm8` - 8-bit PCM (default)
* `pcm16` - 16-bit PCM
* `opus` - Opus codec (16kHz)
* `opus_fs320` - Opus with 320 frame size
* `aac` - AAC codec
* `lc3` - LC3 codec
* `lc3_fs1030` - LC3 with 1030 frame size
**Type:** `integer` | **Default:** `1`
Number of audio channels. Use `1` for mono, `2` for stereo.
**Type:** `boolean` | **Default:** `true`
Enable speaker identification using the user's stored speech profile. When enabled, the system extracts a speaker embedding from the user's speech profile and uses it to identify the user's voice via biometric matching.
**Type:** `integer` | **Default:** `120` | **Range:** `2-14400`
Seconds of silence before the conversation is automatically processed. After this timeout, the conversation is saved and LLM processing begins.
**Type:** `string` | **Optional**
Serving provider selection is controlled by the deployment policy. The supported providers are `parakeet` and `modulate`; clients cannot opt into retired providers.
**Type:** `string` | **Default:** `'disabled'`
Enable custom STT mode. When set to `'enabled'`, the backend accepts app-provided transcripts instead of using STT services. Useful for apps with their own transcription.
**Type:** `string` | **Optional**
Conversation source identifier. Examples: `'omi'`, `'openglass'`, `'phone'`
## Audio Codecs
The system supports multiple audio codecs with automatic decoding:
| Codec | Sample Rate | Description | Use Case |
| - | - | - | - |
| `pcm8` | 8kHz | 8-bit PCM | Default, low bandwidth |
| `pcm16` | 16kHz | 16-bit PCM | Better quality |
| `opus` | 16kHz | Opus encoded | Efficient compression |
| `opus_fs320` | 16kHz | Opus 320 frame | Alternative frame size |
| `aac` | Variable | AAC encoded | iOS compatibility |
| `lc3` | Variable | LC3 codec | Bluetooth audio |
| `lc3_fs1030` | Variable | LC3 1030 frame | Alternative LC3 |
All audio is internally converted to 16-bit linear PCM before being sent to STT providers.
## STT Service Selection
The canonical provider/surface matrix and default model order live in
`backend/config/stt_provider_policy.py`. The deployment validator requires
`STT_SERVICE_MODELS` and `STT_PRERECORDED_MODEL` to match that code-owned
policy; an environment edit cannot re-enable hosted Deepgram. The normal serving
defaults are `modulate-velma-2,parakeet`. Self-hosted Deepgram is a distinct,
streaming-only provider that requires `DEEPGRAM_SELF_HOSTED_ENABLED=true` and a
non-cloud `DEEPGRAM_SELF_HOSTED_URL`; it is never a fallback to
`api.deepgram.com`. If no supported provider can serve a language, the request
fails closed.
Managed live connections can use a declarative target registry to select the
cheapest capable endpoint within its rollout share and capacity. Shared health
gates remove disrupted targets and restore cheaper targets through staged
recovery. Target identity is separate from provider family, so two endpoints
using the same provider protocol can have independent routing health. Custom
STT, BYOK and multichannel connections keep their existing paths. Fleet trial
rejection requires broad failure evidence; healthy small cohorts can recover
without a distinct-user minimum. Capacity signals and short local capacity
cooldowns apply to terminal failover legs as well as primary selection. If all
candidates are capacity-blocked, one least-recently-refused candidate may dial
through normal admission. Recovery gives each trial user one majority-outcome
vote, preventing repeated client failures from blocking a healthy majority.
```mermaid theme={null}
flowchart TD
Start[Incoming Audio] --> Lang{Language?}
Lang -->|Supported by first configured model| Selected[Parakeet / Modulate]
Lang -->|Unsupported| Unavailable[Fail closed]
```
### Provider Capabilities
| Provider | Languages | Model | Best For |
| - | - | - | - |
| **Modulate** | Provider-dependent | `velma-2` | Supported configured languages |
| **Parakeet** | Deployment-dependent | `parakeet` | Self-hosted streaming and batch STT |
## Batch Transcription Proxy
For one-shot (non-streaming) transcription of an audio file, the backend exposes an authenticated proxy in front of the parakeet GPU service — clients must never call parakeet directly:
```
POST https://api.omi.me/v1/stt/transcribe
Authorization: Bearer {firebase_id_token}
Content-Type: multipart/form-data
```
| Field | Type | Default | Purpose |
| - | - | - | - |
| `file` | file upload | required | Audio file (WAV recommended; max 200MB) |
| `diarize` | bool | `true` | Attach `speaker` labels to segments |
Response (mirrors parakeet `/v2/transcribe`):
```json theme={null}
{
"text": "full transcript",
"segments": [{"start": 0.0, "end": 1.5, "text": "…", "speaker": "SPEAKER_00"}],
"detected_language": "en"
}
```
Errors: `400` empty file, `401` missing/invalid bearer token, `413` audio exceeds the duration/size limits, `429` per-user rate limit exceeded (`stt:transcribe` policy — honor `Retry-After`), `503` model loading or service busy (retry later), `502` upstream failure. Implementation: `backend/routers/stt.py`.
## Transcription outcome contract
Transport success is not transcription success. The backend uses one bounded
semantic vocabulary across voice upload, voice-message SSE, offline sync, and
live provider failures:
| Outcome | Meaning | Retryable |
| - | - | - |
| `success` | Speech-eligible audio produced non-empty normalized text | No |
| `expected_silence` | An explicit speech gate found no eligible speech | No |
| `empty_unexpected` | Eligible audio reached STT but produced no usable text | Yes |
| `timeout` | The selected provider timed out | Yes |
| `upstream_error` | The provider or response parser failed | Yes |
| `config_error` | The selected provider is not deployable on this runtime | No |
| `invalid_input` | The audio cannot be decoded or violates the route contract | No |
`POST /v2/voice-message/transcribe` returns `outcome` alongside `transcript`,
`stt_provider`, and `stt_model`. True silence remains HTTP 200 with an empty
transcript and `outcome=expected_silence`. All failure outcomes use a non-2xx
status and a fixed response body containing only `error`, `outcome`,
`provider`, `retryable`, and a public-safe message; provider response bodies,
audio identifiers, and exception text are never exposed.
The `/v2/voice-messages` stream emits the same safe failure object in a terminal
`error:` SSE frame. A normal empty stream is reserved for explicit
`expected_silence`.
For `/v4/listen`, an unusable initial or mid-session STT socket emits this event
before the client connection closes with WebSocket code 1011:
```json theme={null}
{
"type": "service_status",
"status": "stt_failed",
"outcome": "upstream_error",
"provider": "parakeet",
"retryable": true,
"reason": "connection_lost"
}
```
The backend retains any buffered audio it could not hand to the provider. It
does not keep a green client WebSocket while discarding later audio; the close
activates the client's existing reconnect or local-recovery path.
## Production transcription serving smoke
Development deployments retain the no-traffic, uniquely tagged `backend`
candidate gate. Production does not use a tagged URL, Cloud Run Job, service
account, or temporary IAM binding. Instead it validates exact no-traffic Cloud
Run revisions, snapshots traffic, promotes them, verifies the exact serving
release vector, and only then proves the real `/v2/voice-message/transcribe`
route through `https://api.omi.me`. This is post-promotion serving evidence,
not candidate evidence: any serving-vector, route-presence, or known-audio
failure restores the saved Cloud Run traffic snapshot.
`backend/testing/release_fixtures/transcription-release-probe.wav` and its
versioned JSON manifest provide the known audio, language, expected transcript,
SHA-256 digest, and CC-BY-4.0 LibriSpeech provenance. The gate requires HTTP
200, `outcome=success`, and the exact normalized transcript. It intentionally
does not assert provider or model identity: this is a semantic capability gate,
not an unreviewed routing-policy setting.
The shared `transcription-release-candidate-probe` action uses the existing
authenticated deploy identity to read the existing `FIREBASE_API_KEY` Secret
Manager secret, sign a five-minute Firebase custom token for the dedicated
non-human `omi-release-probe` UID, and exchange it immediately for an ID token.
It writes that token only to an owner-only temporary runner file, never exposes
it through workflow outputs or evidence, and deletes it at step exit. The
workflow makes no IAM changes: the existing deploy identity must be reviewed to
have only the Secret Manager read and `iam.serviceAccounts.signJwt` access this
action needs. Missing access fails closed before promotion. Reports contain only
redacted booleans and status codes.
The production smoke first makes a deliberately unauthorized, malformed-safe
reservation-route request and requires its expected validation response; it
does not create a reservation. It then runs the known-audio probe with the same
runner-local token file. The token is never a workflow output, command-line
argument, artifact, or resource metadata.
## Automatic development candidate acceptance
`gcp_backend_auto_dev.yml` keeps the development backend mutation lock while
it deploys the four Cloud Run services (`backend`, `backend-sync`,
`backend-sync-backfill`, and `backend-integration`) with the service-scoped
`candidate` tag. It resolves each exact tag URL from Cloud Run status, runs the
source-owned `backend/deploy/dev_candidate_acceptance.json` manifest, and only
then permits the one traffic-promotion step. The backend uses the authenticated
What Matters Now contract; the worker services use bounded `/v1/health` checks.
The evidence artifact records only service, bounded contract category, and
outcome — never URLs, tokens, user data, or response bodies.
Candidate requests carry an OIDC token in `X-Serverless-Authorization` while
the application keeps its own `Authorization` header. The token audience is
the canonical Cloud Run service URL as required by Cloud Run; the HTTP target
remains the exact no-traffic tagged candidate URL.
Development pusher is deliberately **GKE-only** (`pusher.omiapi.com`). A
legacy Cloud Run `pusher` service is not a deploy, candidate, or health-report
surface; do not publish an image merely to make that retired surface appear
ready.
During the direct-URL retirement window, the public dev Cloud Run `backend`
and legacy `backend-listen` services must use `http://pusher.omiapi.com`.
Other dev Cloud Run services and jobs must not define `HOSTED_PUSHER_API_URL`.
Retire those public endpoints only after they show no `/v4/listen` traffic.
## Serving STT Configuration
Serving revisions use `HOSTED_PARAKEET_API_URL` for Parakeet and
`MODULATE_API_KEY` for Modulate. Both are required because the selected
provider depends on language capability. The retained self-hosted Deepgram
deployment uses `DEEPGRAM_API_KEY`, `DEEPGRAM_SELF_HOSTED_ENABLED`, and
`DEEPGRAM_SELF_HOSTED_URL` only in its explicitly configured GKE workload;
hosted Deepgram is disabled.
## External Custom STT Service
Build your own transcription/diarization WebSocket service that integrates with Omi.
```mermaid theme={null}
flowchart LR
subgraph App["📱 Omi App"]
Capture[Audio Capture]
end
subgraph Custom["🎧 Your STT Service"]
WS[WebSocket Server]
end
subgraph Backend["🖥️ Omi Backend"]
API["/v4/listen"]
end
Capture -->|Binary audio| WS
WS -->|JSON transcripts| Capture
Capture -->|suggested_transcript| API
```
### Your Service Receives
| Message | Format | Description |
| - | - | - |
| Audio frames | Binary | Raw audio bytes (codec configured by app, typically `opus` 16kHz) |
| `{"type": "CloseStream"}` | JSON | End of audio stream |
### Your Service Sends
**Format:** JSON object with `segments` array
```json theme={null}
{
"segments": [
{
"text": "Hello, how are you?",
"speaker": "SPEAKER_00",
"start": 0.0,
"end": 1.5
},
{
"text": "I'm doing great, thanks!",
"speaker": "SPEAKER_01",
"start": 1.6,
"end": 3.2
}
]
}
```
### Segment Fields
| Field | Type | Required | Description |
| - | - | - | - |
| `text` | `string` | Yes | Transcribed text |
| `speaker` | `string` | No | Speaker label (`SPEAKER_00`, `SPEAKER_01`, etc.) |
| `start` | `float` | No | Start time in seconds |
| `end` | `float` | No | End time in seconds |
### Requirements
* Response **must be an object** with `segments` key. Raw arrays `[{...}]` will fail.
* Do **not** include a `type` field, or set it to `"Results"`. Other values are ignored.
* Connection closes after **90 seconds** of inactivity.
## Speech Profile & Speaker Embedding
When a user has a speech profile, the system uses speaker embedding comparison to identify the user's voice in real-time.
### How It Works
```mermaid theme={null}
sequenceDiagram
participant App as 📱 Omi App
participant Backend as 🖥️ Backend
participant STT as 🎧 Selected STT provider
participant Embed as 🧠 Embedding API
Note over Backend: User has speech profile
App->>Backend: Connect WebSocket
Backend->>STT: Create single socket
Backend->>Embed: Extract user embedding from profile WAV
loop Audio streaming
App->>Backend: Audio chunk
Backend->>STT: Forward decoded audio
STT-->>Backend: Transcript with speaker IDs
end
Note over Backend: New speaker detected (2s+ audio)
Backend->>Embed: Extract speaker embedding from audio
Embed-->>Backend: Compare with user embedding
Backend-->>App: Segments (is_user: true/false)
```
### Speech Profile Benefits
1. **User Identification**: Speaker embedding comparison identifies the device owner by voice biometrics
2. **No Startup Delay**: Transcription begins immediately (no profile audio prepending)
3. **Single Socket**: One selected-provider connection per session
### Teaching another person's voice
Diarization numbers such as `SPEAKER_00` are local to a conversation. Owner
enrollment writes the user's own profile; naming another speaker writes a stable
person identity under the authenticated user's people collection.
The mobile client sends selected segment IDs to
`PATCH /v1/conversations/{conversation_id}/segments/assign-bulk`. The label is
saved immediately. Background teaching requires retained audio chunks and a
ten-second window of speech belonging to that person. Adjacent segments may
extend the window, but both their identity and their text must participate in
verification. Sample transcription uses the conversation language, falling back
to the user's language preference; it is not restricted to English.
A quality-verified sample, transcript, v3 embedding, and source conversation and
segment IDs are published in one person-document transaction. Valid re-teaching
replaces the previous profile only after extraction succeeds. A person version
check rejects work that outlived a correction, replacement, or deletion. New live
sessions load stored profiles; offline sync accepts only v3 profiles with samples
and an embedding. The returned `person_id` resolves to the person's name in the
mobile people cache.
Correcting a source label invalidates the profile learned from it. Profiles from
other conversations remain intact. Historical profiles without source metadata
cannot be attributed to a corrected label: delete their sample in People settings
and teach again. Sample deletion clears its embedding, and missing-embedding
recovery is fenced against concurrent deletion. A label saved without sufficient
retained, verified speech is not proof of successful voice enrollment.
Private Cloud Sync controls retaining the audio needed to teach another person.
Live matching is admitted when cloud sync or an owner speech profile is enabled,
unless custom STT is selected. Recognition accuracy and an already-running
session's cached assignments require separate device qualification; persistence
tests do not establish acoustic accuracy or immediate cache invalidation.
## Transcription Flow
WebSocket connection accepted, user validated, STT provider selected based on language.
App sends binary audio chunks. Backend decodes based on codec parameter.
Decoded audio is sent to the selected Parakeet or Modulate provider. The provider returns word-level transcripts with speaker IDs.
Words grouped into segments. Same-speaker consecutive words merged. Timing adjusted for speech profile offset.
JSON segments streamed back to app immediately. UI updates as user speaks.
Background task monitors silence. After `conversation_timeout`, conversation is processed and saved.
## Message Formats
### Incoming Messages (App → Backend)
**Format:** Binary
Raw audio bytes encoded according to the `codec` parameter. Sent continuously during recording.
```
[Binary audio chunk - varies by codec]
```
**Keep-alive:** Messages of 2 bytes or less are treated as heartbeat pings.
**Format:** JSON
Assign a known person to detected speakers:
```json theme={null}
{
"type": "speaker_assigned",
"speaker_id": 1,
"person_id": "person-uuid-here",
"person_name": "John",
"segment_ids": ["seg-uuid-1", "seg-uuid-2"]
}
```
**Format:** JSON
When `custom_stt=enabled`, apps can provide their own transcripts:
```json theme={null}
{
"type": "suggested_transcript",
"segments": [
{
"text": "Hello there",
"speaker": "SPEAKER_00",
"speaker_id": 0,
"start": 0.0,
"end": 1.5,
"is_user": true,
"person_id": "known-person-uuid-or-null"
}
],
"stt_provider": "custom-provider-name"
}
```
See [External Custom STT Service](#external-custom-stt-service) for building your own transcription service.
**Format:** JSON
For OpenGlass and visual captures:
```json theme={null}
{
"type": "image_chunk",
"id": "temp-image-id",
"index": 0,
"total": 3,
"data": "base64-encoded-chunk"
}
```
### Outgoing Messages (Backend → App)
**Format:** JSON Array
Real-time transcript segments as they're detected:
```json theme={null}
[
{
"id": "uuid-string",
"text": "Hello there",
"speaker": "SPEAKER_00",
"speaker_id": 0,
"is_user": true,
"person_id": null,
"start": 0.0,
"end": 1.5,
"speech_profile_processed": true,
"stt_provider": "parakeet"
}
]
```
**Format:** JSON
Connection and service status updates:
```json theme={null}
{
"type": "service_status",
"status": "ready",
"provider": "deepgram",
"reason": "fallback_from_modulate"
}
```
`provider` names the STT provider actually serving the session, resolved after the
fallback chain has settled — a session may be served by a provider other than the one
the serving policy selected. `reason` appears only in that case, as
`fallback_from_`, so a client can tell a fallback session from a healthy one.
Both fields are omitted when absent, and `provider` is omitted entirely for custom-STT
sessions, where the client produces its own transcripts.
**Format:** JSON
System suggests a known person for a detected speaker:
```json theme={null}
{
"type": "speaker_label_suggestion",
"speaker_id": 1,
"person_id": "person-uuid",
"person_name": "John",
"segment_id": "segment-uuid"
}
```
**Format:** JSON
Sent when conversation timeout triggers processing:
```json theme={null}
{
"type": "memory_created",
"memory": {
"id": "conversation-uuid",
"structured": {
"title": "Meeting Discussion",
"overview": "..."
}
},
"messages": []
}
```
**Format:** JSON
When translation is enabled:
```json theme={null}
{
"type": "translation",
"segments": [
{
"id": "segment-uuid",
"translations": [
{"lang": "es", "text": "Hola ahí"}
]
}
]
}
```
## Transcript Segment Model
Each transcript segment contains:
| Field | Type | Description |
| - | - | - |
| `id` | `string` | Unique UUID for the segment |
| `text` | `string` | Transcribed text content |
| `speaker` | `string` | Speaker label (`"SPEAKER_00"`, `"SPEAKER_01"`, etc.) |
| `speaker_id` | `integer` | Numeric speaker ID (0, 1, 2...) |
| `is_user` | `boolean` | `true` if spoken by device owner |
| `person_id` | `string?` | UUID of identified person (if matched) |
| `start` | `float` | Start time in seconds |
| `end` | `float` | End time in seconds |
| `speech_profile_processed` | `boolean` | Whether speech profile was used for identification |
| `stt_provider` | `string?` | Name of STT provider used |
## Connection Lifecycle
```mermaid theme={null}
stateDiagram-v2
[*] --> Connecting: WebSocket request
Connecting --> Authenticating: Connection accepted
Authenticating --> Ready: User validated
Authenticating --> Closed: Auth failed
Ready --> Streaming: Audio received
Streaming --> Streaming: More audio
Streaming --> Processing: Silence timeout
Processing --> Streaming: New audio
Processing --> Closed: Session complete
Ready --> Closed: Client disconnect
Streaming --> Closed: Client disconnect
note right of Processing
Conversation saved
LLM extracts structure
Memories extracted
end note
```
### Lifecycle Events
1. WebSocket accepted
2. User authentication verified
3. Language/STT service selected
4. STT connections initialized (with retry logic)
5. Speech profile loaded in background
6. Heartbeat task started (10s interval)
1. Audio received and decoded
2. Sent to STT provider(s)
3. Results collected in buffers
4. Processed every 600ms
5. Segments sent to client
6. Speaker suggestions generated
1. Usage statistics recorded
2. All STT sockets closed
3. Client WebSocket closed (code 1000/1001)
4. Buffers and collections cleared
## Error Handling & Retry Logic
The system includes robust error handling:
| Error Type | Handling |
| - | - |
| **STT Connection Failed** | Before live audio is accepted, bounded connection retry may select a configured provider. After a live provider becomes unusable, emit `service_status(stt_failed)` and close the client WebSocket with `1011`; do not discard audio as successful. |
| **Provider Error** | Pre-recorded/sync work classifies the provider failure and remains retryable. Live `/v4/listen` fails the session visibly; the desktop/mobile client retains local retry material and reconnects. |
| **Decode Error** | Log and skip corrupted audio chunk |
| **WebSocket Error** | Clean close with appropriate code |
Live STT never silently downgrades a terminal provider failure to a successful
transcription. The app receives the bounded failure status before the `1011`
close and must retain unconfirmed local audio until a later successful sync.
## Key File Locations
| Component | Path |
| - | - |
| WebSocket Handler | `backend/routers/transcribe.py` |
| Streaming STT Integration | `backend/utils/stt/streaming.py` |
| Audio Decoding | `backend/routers/transcribe.py` |
| Speaker Match Policy | `backend/utils/stt/speaker_match.py` |
| VAD (Voice Activity) | `backend/utils/stt/vad.py` |
| Transcript Model | `backend/models/transcript_segment.py` |
## Related Documentation
Complete backend architecture overview
How conversations and memories are stored
How the AI chat system uses transcriptions
Environment setup and configuration
## Ramped windowed TDT live transcription
Managed single-channel listen sessions can opt into `parakeet-window,soniox` with
`STT_CONNECT_ORDER_FROM_CONFIG=true` and a stable UID allocation configured by
`PARAKEET_WINDOW_ALLOCATION_PERCENT` (default zero). The TDT leg uses speech-gated,
six-second batch POSTs, a process session cap, bounded buffering and per-leg health.
PTT, multi-channel, custom STT and BYOK stay outside this rollout. Provider changes
retain session-relative audio timestamps and separate speaker epochs. Detailed
controls, capacity arithmetic and rollback are in
[`backend/docs/operational/windowed-live-stt.md`](https://github.com/BasedHardware/omi/blob/main/backend/docs/operational/windowed-live-stt.md).
# For agents
Source: https://docs.omi.me/doc/developer/cli/agents
Stable agent contract — JSON output, exit codes, retry semantics.
`omi-cli` is built so an LLM-driven harness can drive it without a wrapper
layer. This page documents the stable contract.
## The JSON contract
Pass `--json` (a **global** flag, before the subcommand) and the CLI behaves
like a strict tool:
* **stdout** receives a single JSON document and *only* a JSON document.
No spinners, no progress messages, no decorative output.
* **stderr** receives any error as a JSON object: `{"error": "...", "detail": "..."}`.
* **exit code** signals what happened.
```bash theme={null}
omi --json memory list --limit 25 | jq '.[] | {id, content, category}'
```
## Exit codes
| Code | Meaning |
| - | - |
| `0` | Success. |
| `1` | Usage/validation error, or the specific Desktop SQL failure described below. |
| `2` | Auth error — no creds, expired token, insufficient scope. |
| `3` | Server error — 5xx after retries exhausted, or connection failure. |
| `4` | Rate limited — 429 after retries exhausted. `detail` includes the wait. |
| `5` | Not found — 404, or client-side scan came up empty. |
These codes are **part of the contract** — they won't shift between minor
versions. Branch on them in your harness without parsing English errors.
When Desktop returns the exact `execute_sql` result
`SQL Error: The local database could not complete that query.`, both
`omi local sql` and `omi local call execute_sql` exit with code `1`, including
when Desktop wraps that result in an HTTP 200 response. With `--json`, this
error goes to stderr and stdout stays empty. This message match does not
classify other SQL result strings as failures. A successful query with no
rows still returns exit code `0`.
## Authentication: API key vs browser OAuth
The CLI supports both — but for **agents** specifically, the API-key path is
almost always the right call:
| Need | Use this |
| - | - |
| Headless / CI / container runtime | `OMI_API_KEY` env var |
| Long-running unattended automation | API key |
| Scoped permissions | API key (granular scopes) |
| Interactive use by a human | `omi auth login --browser` |
API keys are long-lived, scoped, and don't need a browser — perfect for
agents. The browser OAuth flow exists for humans on a laptop, uses a
loopback callback with PKCE, and stores short-lived Firebase ID tokens that
auto-refresh between calls.
```bash theme={null}
export OMI_API_KEY=omi_dev_...
```
The on-disk config file is great for humans but a footgun in shared CI
runners. The env var injection runs the same prefix validation as
`omi auth login` so a malformed value still fails fast with exit `1`.
## Retry behavior
Transient failures are retried automatically before the CLI surfaces an error:
* **5xx** — exponential backoff with jitter (initial 0.5s, cap 8s).
* **429** — honors the server's `Retry-After` header when present (capped at
60s so a misconfigured upstream can't pin your agent forever); falls back to
jitter otherwise. The `Retry-After` cap means a 429 with a one-hour hint
becomes a one-minute wait — your agent gets exit `4` quickly enough to
decide whether to back off itself.
* **Transport errors** (DNS failures, dropped TCP) — retried up to 4 attempts.
After retries are exhausted, the CLI surfaces a structured error and the
appropriate exit code. The `detail` field for rate-limit errors looks like:
```json theme={null}
{"error": "Rate limited: dev:conversations (25/hr)", "detail": "Retry in 12s. ..."}
```
The policy name (`dev:conversations`, `dev:memories`, `dev:memories_batch`)
matches the backend's rate-limit policy IDs so you can map them to your own
backoff strategies.
## Worked example: Python harness
```python theme={null}
import json
import subprocess
from datetime import datetime, timedelta, timezone
from typing import Any
class OmiCliError(RuntimeError):
pass
def omi(*args: str) -> Any:
"""Invoke the omi CLI in JSON mode, raising on non-success exit codes."""
result = subprocess.run(
["omi", "--json", *args],
capture_output=True,
text=True,
check=False,
)
if result.returncode == 0:
return json.loads(result.stdout) if result.stdout.strip() else None
# Errors come back as JSON on stderr in JSON mode.
try:
err = json.loads(result.stderr)
except json.JSONDecodeError:
err = {"error": result.stderr.strip()}
err["exit_code"] = result.returncode
raise OmiCliError(err)
# Read all open action items and mark anything older than 30 days complete.
cutoff = datetime.now(timezone.utc) - timedelta(days=30)
items = omi("action-item", "list", "--open") or []
for item in items:
created = datetime.fromisoformat(item["created_at"].replace("Z", "+00:00"))
if created < cutoff:
omi("action-item", "complete", item["id"])
```
## Handling rate limits in your harness
```python theme={null}
import json
import re
import subprocess
import time
def omi_with_backoff(*args: str, max_attempts: int = 3) -> object:
for _ in range(max_attempts):
result = subprocess.run(
["omi", "--json", *args],
capture_output=True,
text=True,
)
if result.returncode == 0:
return json.loads(result.stdout) if result.stdout.strip() else None
if result.returncode != 4: # not a rate-limit error
raise RuntimeError(result.stderr)
# Pull "Retry in Ns" out of the detail message.
match = re.search(r"Retry in (\d+)s", result.stderr)
wait_s = int(match.group(1)) if match else 60
time.sleep(wait_s)
raise RuntimeError("rate-limited after retries")
```
## Tips
* **One JSON document per invocation.** Don't try to stream — the CLI doesn't
emit incremental output. Run it again for the next page.
* **Use `--profile` to isolate environments.** A staging profile + a prod
profile saves you from accidentally writing to prod with a test script.
* **Use `--api-base http://localhost:8080`** for local backend testing.
* **`--verbose` is safe in JSON mode.** Debug output goes to stderr, stdout
stays valid JSON.
* **Pipe stdin with `--text -`** for `omi conversation create` — handy when
the content is generated by another tool and you don't want to shell-escape it.
# Command reference
Source: https://docs.omi.me/doc/developer/cli/commands
Every verb, every flag.
The full command tree (run `omi --help` for the live version):
```text theme={null}
omi
├── auth login [--browser] [--api-key KEY] | logout | status | whoami | refresh
├── config show | set | path | profile {list, use , delete }
├── memory list | get | create | update | delete
├── conversation list | get | create | from-segments | update | delete
├── action-item list | get | create | update | complete | delete
└── goal list | get | create | update | progress | history | delete
```
## Global flags
```text theme={null}
--json Emit JSON to stdout (machine-readable, agent-friendly).
--profile, -p NAME Use a specific profile from ~/.omi/config.toml.
--api-base URL Override the API base URL.
--verbose, -v Log HTTP traffic to stderr.
--no-color Disable colored output (also honors $NO_COLOR).
--version Print the version.
--help Show contextual help.
```
Global flags must come **before** the subcommand
(`omi --json memory list`, not `omi memory list --json`). This is a Click
framework constraint; you'll get a friendly usage message if you put them
in the wrong place.
***
## `omi auth`
| Command | Notes |
| - | - |
| `omi auth login` | Interactive prompt: pick browser OAuth or paste an API key. |
| `omi auth login --browser` | Force the Firebase OAuth browser flow (default provider: Google). |
| `omi auth login --browser --provider apple` | Same flow, Apple Sign-In. |
| `omi auth login --api-key K` | Force the API-key path. Exposes the key in `ps`/history — prefer the prompt. |
| `omi auth logout` | Clear the active profile's credentials. |
| `omi auth status` | Show profile + masked credential + API base. OAuth profiles also show expiry. |
| `omi auth whoami` | Round-trip to the API to verify the credential is valid. |
| `omi auth refresh` | Force-refresh the OAuth ID token. No-op for API-key profiles. |
OAuth ID tokens are auto-refreshed before each request when the cached token
is past (or near) its expiry — you should rarely need to call `refresh` by
hand. The refresh token rotates if Firebase decides to.
## `omi config`
| Command | Notes |
| - | - |
| `omi config show` | Print all profiles + masked credentials. |
| `omi config path` | Print the on-disk config file path. |
| `omi config set api_base URL` | Override the API base for the active profile. |
| `omi config profile list` | Tabular list of profiles. |
| `omi config profile use NAME` | Switch the active profile. Creates it if missing. |
| `omi config profile delete NAME` | Wipe a profile (prompts unless `--yes`). |
## `omi memory`
| Command | Endpoint |
| - | - |
| `omi memory list [--limit N --offset N --categories CSV]` | `GET /v1/dev/user/memories` |
| `omi memory get ID` | client-side scan (no direct GET in the dev API) |
| `omi memory create CONTENT [--category C --visibility V --tag T...]` | `POST /v1/dev/user/memories` |
| `omi memory update ID [--content ... --category ... --visibility ... --tag ...]` | `PATCH /v1/dev/user/memories/{id}` |
| `omi memory delete ID [-y]` | `DELETE /v1/dev/user/memories/{id}` |
Categories: `core`, `hobbies`, `lifestyle`, `interests`, `habits`, `work`,
`skills`, `learnings`, `other`, `system`.
Visibility: `public` or `private`.
## `omi conversation`
| Command | Endpoint |
| - | - |
| `omi conversation list [...filters]` | `GET /v1/dev/user/conversations` |
| `omi conversation get ID [--include-transcript]` | `GET /v1/dev/user/conversations/{id}` |
| `omi conversation create --text ... [...]` | `POST /v1/dev/user/conversations` |
| `omi conversation from-segments FILE.json [--source ...]` | `POST /v1/dev/user/conversations/from-segments` |
| `omi conversation update ID [--title ... --discarded/--no-discarded]` | `PATCH /v1/dev/user/conversations/{id}` |
| `omi conversation delete ID [-y]` | `DELETE /v1/dev/user/conversations/{id}` |
The `--text` flag accepts `-` to read from stdin:
```bash theme={null}
cat meeting_notes.md | omi conversation create --text - --text-source message
```
## `omi action-item`
| Command | Endpoint |
| - | - |
| `omi action-item list [--completed/--open ...]` | `GET /v1/dev/user/action-items` |
| `omi action-item get ID` | client-side scan |
| `omi action-item create DESC [--due-at ISO]` | `POST /v1/dev/user/action-items` |
| `omi action-item update ID [--description ... --completed/--open --due-at ... --clear-due-at]` | `PATCH /v1/dev/user/action-items/{id}` |
| `omi action-item complete ID` | shortcut for `update --completed` |
| `omi action-item delete ID [-y]` | `DELETE /v1/dev/user/action-items/{id}` |
## `omi goal`
| Command | Endpoint |
| - | - |
| `omi goal list [--limit N --include-inactive]` | `GET /v1/dev/user/goals` |
| `omi goal get ID` | `GET /v1/dev/user/goals/{id}` |
| `omi goal create TITLE --target N [--type ...] [...]` | `POST /v1/dev/user/goals` |
| `omi goal update ID [...]` | `PATCH /v1/dev/user/goals/{id}` |
| `omi goal progress ID VALUE` | `PATCH /v1/dev/user/goals/{id}/progress` |
| `omi goal history ID [--days N]` | `GET /v1/dev/user/goals/{id}/history` |
| `omi goal delete ID [-y]` | `DELETE /v1/dev/user/goals/{id}` |
Use `omi goal update ID --unit liters` to set a unit label or
`omi goal update ID --clear-unit` to remove it. These options are mutually
exclusive. Omitting both leaves the unit unchanged.
Goal types: `boolean`, `scale`, `numeric`. Up to **3 active goals per user**;
the oldest is deactivated automatically when you create a fourth.
## `omi app`
Build and test an Omi app without a device. Every event is a POST to the URL
the user configures in Developer Settings; these commands replay the same
request locally and receive it locally.
| Command | What it does |
| - | - |
| `omi app events` | List the webhook events an app can receive, with their body shapes. |
| `omi app payload EVENT [--uid UID]` | Print the sample body for one event. |
| `omi app send EVENT --to URL [--uid U] [--payload-file F] [--idempotency-key K] [--timeout S]` | POST one sample event the way Omi posts it. |
| `omi app verify --to URL [--event E] [--uid U]` | Report whether a URL would work in production. |
| `omi app serve [--port N] [--host H] [--status N] [--reply JSON] [--requests N]` | Receive deliveries locally and print each one. |
Events: `memory_created`, `realtime_transcript`, `day_summary`, `audio_bytes`,
`button_event`.
```bash theme={null}
omi app serve --reply '{"message": "Noted, I will remind you"}' # one terminal
omi app send realtime_transcript --to http://127.0.0.1:8787/webhook # another
```
`send` exits non-zero when the receiver does not answer 2xx, so it works as a
check in CI. It also reports what Omi would do next: whether the delivery would
be retried or dropped, and for `realtime_transcript` whether your reply was
long enough to reach the user as a notification.
`verify` posts every event to one URL and prints a report card: whether Omi's
delivery client would accept the URL at all, the status and latency per event,
whether a failure would be retried or dropped, and whether a
`realtime_transcript` reply would reach the user. A private or loopback host, a
redirect, or a response slower than 30 seconds all fail, because the backend
refuses the first, never follows the second, and gives up on the third. It
exits non-zero when any check fails. The events it posts are real deliveries,
so point it at staging if your app writes somewhere on receipt.
***
## Environment variables
| Variable | Effect |
| - | - |
| `OMI_API_KEY` | Use this dev API key without persisting anything to disk. Validated against the prefix. |
| `OMI_API_BASE` | Override the API base URL (handy for staging or local backends). |
| `OMI_PROFILE` | Default profile name when `--profile` is not passed. |
| `OMI_CONFIG` | Path to the config file (default `~/.omi/config.toml`). |
| `NO_COLOR` | Disable color output (standard, see [no-color.org](https://no-color.org)). |
# Install
Source: https://docs.omi.me/doc/developer/cli/install
Install omi-cli and confirm the binary is on your PATH.
`omi-cli` is published on PyPI: [pypi.org/project/omi-cli](https://pypi.org/project/omi-cli/).
Source lives in the [Omi monorepo](https://github.com/BasedHardware/omi/tree/main/sdks/python-cli).
## Install
```bash theme={null}
pipx install omi-cli
```
`pipx` keeps `omi-cli` in its own isolated virtualenv so it can't conflict
with anything else on your system. If you don't have pipx yet:
```bash theme={null}
brew install pipx # macOS
apt install pipx # Debian/Ubuntu
pipx ensurepath
```
```bash theme={null}
pip install omi-cli
```
Suitable if you're already inside a virtualenv or you want the package
available alongside the rest of your project's deps.
```bash theme={null}
git clone https://github.com/BasedHardware/omi
cd omi/sdks/python-cli
pip install -e ".[dev]"
```
Useful if you want to hack on the CLI itself. The `[dev]` extra brings in
`pytest`, `respx`, `black`, `mypy`, `isort`, and the build tooling.
The PyPI distribution name is `omi-cli` because the bare `omi` slot belongs
to an unrelated package. The console command is **`omi`** regardless.
## Verify
```bash theme={null}
omi --version
# omi-cli 0.3.0
omi --help
```
You should see the full command tree (`auth`, `config`, `memory`,
`conversation`, `action-item`, `goal`).
## Requirements
* **Python 3.10+** — the package is tested on 3.10, 3.11, and 3.12.
* **A network path to `https://api.omi.me`** — overridable via `--api-base` or
`$OMI_API_BASE` for staging or local environments.
* **A way to authenticate** — either a Google/Apple account (browser OAuth)
or a developer API key from the Omi web app. The
[Quickstart](/doc/developer/cli/quickstart) walks through both.
## Where state lives
`omi-cli` stores its config and credentials at `~/.omi/config.toml` (file
permissions `0600`, parent dir `0700`). You can override the location with the
`OMI_CONFIG` environment variable.
# Introduction
Source: https://docs.omi.me/doc/developer/cli/introduction
Talk to Omi from your terminal. Designed for humans and agents alike.
`omi-cli` is the command-line interface to the Omi developer API. It exposes
scoped, JSON-friendly verbs for the four primary nouns Omi maintains about you,
so you can drive Omi from shell pipelines, CI jobs, AI agent harnesses, or your
own personal scripts.
* **PyPI:** [pypi.org/project/omi-cli](https://pypi.org/project/omi-cli/)
* **Source:** [github.com/BasedHardware/omi/tree/main/sdks/python-cli](https://github.com/BasedHardware/omi/tree/main/sdks/python-cli)
* **Install:** `pipx install omi-cli`
Facts and learnings the system knows about you.
Captured & processed audio/text exchanges.
Tasks and follow-ups extracted from your captures.
Tracked progress metrics over time.
***
## Why a CLI?
Omi already ships:
* a **Python SDK** ([`omi-sdk`](/doc/developer/sdk/python)) for Bluetooth and audio capture,
* a **Flutter mobile app**, **macOS desktop app**, and
* an **MCP server** ([`MCP`](/doc/developer/mcp/introduction)) for Claude Desktop and similar tools.
What was missing was a **terminal-native, scriptable, JSON-emitting** interface
to the data plane — the thing every modern dev tool ships (`gh`, `stripe`,
`vercel`, `linear`) and the thing AI agents reach for first.
`omi-cli` fills that gap. The same binary works equally well for a human
reading their inbox of action items at the terminal and for an LLM agent
piping `--json` output through `jq`.
## What you can do with it
```bash theme={null}
# Read your memories with a category filter, just the IDs and content as JSON:
omi --json memory list --categories work,skills | jq '.[] | {id, content}'
# Pipe the transcript of a meeting into a new conversation:
cat meeting_notes.md | omi conversation create --text - --text-source message
# Mark every action item older than 30 days as complete:
omi --json action-item list --open \
| jq -r '.[] | select(.created_at < (now - 30*86400 | todate)) | .id' \
| xargs -I{} omi action-item complete {}
# Track progress on a daily goal in a cron job:
omi goal progress g_xyz "$(get_today_progress)"
```
## How it relates to MCP
The two are complementary, not competing:
* **MCP** is the right surface for AI assistants embedded in apps that speak
the Model Context Protocol (Claude Desktop, Cursor). It's invisible to you —
the LLM calls tools directly.
* **CLI** is the right surface when you want a stable command line your
scripts, agents, and shell pipelines can drive — anywhere a process can run.
Both hit the same Omi API and respect the same scopes.
## What's next
`pipx install omi-cli` and you're done.
Your first five commands, end to end.
Every verb, every flag.
Agent contract, exit codes, JSON shape.
# Quickstart
Source: https://docs.omi.me/doc/developer/cli/quickstart
Get from zero to your data — pick browser OAuth or paste a dev API key.
This guide assumes you've already [installed `omi-cli`](/doc/developer/cli/install).
## 1. Log in
`omi auth login` (with no flags) asks how you'd like to authenticate:
```text theme={null}
$ omi auth login
How would you like to log in?
1) Browser — sign in with Google or Apple via OAuth (recommended for humans)
2) API key — paste a developer key from app.omi.me (recommended for agents/CI)
Choose 1 or 2 [1]: _
```
```bash theme={null}
omi auth login # pick option 1
# or skip the prompt:
omi auth login --browser
```
The CLI starts a localhost callback server on an ephemeral port, opens
your default browser to Omi's auth page, and waits for the callback.
Sign in with Google (or `--provider apple`) and you're back at the
terminal — no copying tokens around.
Tokens are stored at `~/.omi/config.toml` (file mode `0600`). The
short-lived Firebase ID token is auto-refreshed before each request
using the long-lived refresh token.
```bash theme={null}
omi auth login --api-key omi_dev_...
```
Or interactively (input is hidden — your token never lands in shell
history):
```bash theme={null}
omi auth login # pick option 2
```
Generate a dev API key in the Omi web app under **Developer → API Keys**.
Choose the scopes you want — `memories:read`, `memories:write`,
`conversations:read`, etc. The CLI validates the `omi_dev_` prefix
client-side so you fail fast on a typo.
Memory scopes are necessary but not sufficient for the Memory API: Omi also
applies a server-side account readiness gate. A `403` with code
`developer_memory_access_not_ready` means the key can still be valid and
correctly scoped. Do not rotate it; retry after the account is enabled or
contact Omi support if the state persists.
```bash theme={null}
export OMI_API_KEY=omi_dev_...
```
The env var takes effect immediately; no on-disk config needed. The same
prefix validation runs, so a malformed value fails with a clear error
rather than a cryptic 401. Pair with `--api-base` if you're targeting a
staging backend.
```bash theme={null}
omi auth login < /path/to/key.txt
```
Useful when the key lives in a secret manager and you can pipe it in
without ever displaying it. (Browser flow doesn't work over stdin —
the CLI assumes piped input is an API key.)
Confirm:
```bash theme={null}
omi auth status
```
```text theme={null}
omi auth status
profile default
authenticated ✓
auth_method oauth
api_base https://api.omi.me
credential eyJhbG…1234
id_token_expires_at 1714142400.0
```
## 2. Read your data
```bash theme={null}
omi memory list
omi conversation list --limit 5
omi action-item list --open
omi goal list
```
Add `--json` (as a global flag, before the verb) for machine-readable output:
```bash theme={null}
omi --json memory list --limit 25 | jq '.[] | {id, content, category}'
```
## 3. Make a change
```bash theme={null}
# Create a memory:
omi memory create "Prefers async over sync meetings" --category work --tag preferences
# Mark an action item complete:
omi action-item complete a1b2c3d4
# Update goal progress:
omi goal progress g_xyz 7
```
## Profiles
Got more than one Omi account (e.g. personal + work)? Use named profiles:
```bash theme={null}
omi config profile use work
omi auth login # logs in the active profile
omi --profile personal memory list # one-off override
```
Each profile has its own auth method, credential, and API base. You can mix
browser-OAuth in your personal profile with an API-key in a CI profile.
## Where to go next
Every verb, every flag.
Stable JSON contract + exit codes for LLM use.
# Troubleshooting
Source: https://docs.omi.me/doc/developer/cli/troubleshooting
Common failure modes and how to fix them.
## "Not authenticated" (exit 2)
```text theme={null}
✗ Not authenticated
Run `omi auth login` and paste your dev API key. (See `omi auth login --help`.)
```
The active profile has no credentials. Either:
* Run `omi auth login` and paste a dev key, or
* `export OMI_API_KEY=omi_dev_...` in the current shell.
Confirm with `omi auth status`.
## "That doesn't look like an Omi developer key" (exit 1)
The CLI validates the key's prefix client-side before ever calling the API.
Real dev keys start with `omi_dev_` and are at least \~24 characters long.
If you see this from `OMI_API_KEY`, double-check that you pasted the whole
token and didn't accidentally include `Bearer ` or wrap it in quotes that
made it into the env value.
## "Insufficient permissions" (exit 2)
```text theme={null}
✗ Insufficient permissions
Your API key does not have the required scope for this operation.
```
The dev key was minted without the scope you're trying to exercise. For
example, `omi memory create` requires `memories:write` but a read-only key
only has `memories:read`.
Generate a new key in the Omi web app under **Developer → API Keys** with the
scope you need, then `omi auth login` to swap it in.
## "Developer Memory API access is not ready" (exit 2)
```text theme={null}
✗ Developer Memory API access is not enabled for this account
```
If a memory command returns `403` with code `developer_memory_access_not_ready`, the key has passed
authentication and can already have the required memory scope. Omi also checks a server-side account
readiness gate before exposing the canonical memory layer.
Do not rotate or recreate the key to resolve this response. Retry after the account is enabled, or contact Omi
support if it persists.
## "Rate limited" (exit 4)
```text theme={null}
✗ Rate limited: dev:conversations (25/hr)
Retry in 12s. Slow down and retry shortly.
```
The dev API enforces hourly limits per policy:
| Policy | Limit |
| - | - |
| `dev:conversations` | 25 / hour |
| `dev:memories` | 120 / hour |
| `dev:memories_batch` | 15 / hour |
The CLI retries automatically up to 4 attempts with backoff before raising. If
you're seeing the error code from a script, sleep for the `Retry in Ns` value
and try again — see [For agents](/doc/developer/cli/agents) for a worked
backoff example.
## "Server error" (exit 3)
```text theme={null}
✗ Server error (502)
The Omi API returned an error. Try again, then check status.omi.me.
```
A `5xx` response from the API after retries exhausted. The CLI already retried
4 times with backoff, so the issue is server-side. Try again in a minute; if
the problem persists, check the Omi status page or report it.
## "No such option: --json" with the flag after the verb
```bash theme={null}
omi memory list --json # ❌ wrong
omi --json memory list # ✅ right
```
Global flags must come **before** the subcommand. This is a Click framework
constraint.
## Token rolled back after a failed login verification
If `omi auth login` succeeds but the verification round-trip returns 401, the
CLI clears the just-stored credential — better to be unauthenticated than
silently broken. The fix: regenerate the key in the Omi web app and try
again. Network errors don't trigger the rollback (the credential is kept and
the CLI just warns).
## OAuth: "OAuth flow timed out"
```text theme={null}
✗ OAuth flow timed out
No callback received within 5 minutes. Try again, or use `omi auth login --api-key` instead.
```
The CLI started a localhost server, opened your browser, and waited 5
minutes for the callback. Possible causes:
* Browser didn't open. Visit the URL the CLI printed manually.
* A firewall blocked the loopback redirect. Try `--api-key` instead.
* You closed the browser tab before the redirect fired.
The CLI cleans up after itself; just rerun `omi auth login --browser`.
## OAuth: "OAuth state mismatch"
```text theme={null}
✗ OAuth state mismatch
The browser callback did not match the original session token. Possible CSRF — aborting.
```
The CSRF protection caught a mismatch — the `state` token in the callback
URL didn't match what the CLI generated. This shouldn't happen in practice;
if it does, run `omi auth login --browser` again. Don't paste random URLs.
## OAuth: "Firebase refresh failed"
```text theme={null}
✗ Firebase refresh failed (401)
Re-run `omi auth login --browser` to get fresh credentials.
```
The stored refresh token is no longer valid (revoked, signed out elsewhere,
or rotated and the rotation didn't persist). Run `omi auth login --browser`
to mint a fresh pair. Your data is safe; only the local credentials are
stale.
## OAuth: which provider?
By default, `omi auth login --browser` uses Google. To use Apple Sign-In:
```bash theme={null}
omi auth login --browser --provider apple
```
## Where state lives
```bash theme={null}
omi config path
# /Users/you/.omi/config.toml
```
Permissions are owner-only (`0600` for the file, `0700` for the parent dir).
If you ever see permissions other than these, something modified the file
outside of `omi-cli` — fix with `chmod 600 ~/.omi/config.toml`.
To wipe everything:
```bash theme={null}
omi auth logout
# or, nuclear option:
rm -rf ~/.omi/
```
## Verbose mode for debugging
```bash theme={null}
omi -v memory list
# [debug] GET /v1/dev/user/memories → 200 (0.42s)
```
Debug output goes to stderr; stdout stays clean for piping. Combine with
`--json` if your script expects JSON.
## Still stuck?
* Check the [Command reference](/doc/developer/cli/commands) for the exact
expected flags.
* File an issue at
[github.com/BasedHardware/omi](https://github.com/BasedHardware/omi/issues)
with the failing command, the full stderr, and the `omi --version` output.
# Compile Firmware
Source: https://docs.omi.me/doc/developer/firmware/Compile_firmware
Build and flash custom firmware for your Omi device using nRF Connect.
## Overview
Build firmware from source to customize your Omi device. If you just want to install the latest release, use the [Flash Device](/doc/get_started/Flash_device) page instead.
`omi/firmware/omi` — SDK v2.9.0
`omi/firmware/devkit` — SDK v2.7.0
***
## Build Steps
Install the [nRF Connect for VS Code](https://www.nordicsemi.com/Products/Development-tools/nRF-Connect-for-VS-Code) extension.
Follow this walkthrough: [How to Set Up nRF Connect](https://youtu.be/EAJdOqsL9m8)
In the nRF Connect extension, install the correct SDK + toolchain version:
| Firmware | SDK Version | Folder |
| - | - | - |
| Consumer (Omi) | **2.9.0** | `omi/firmware/omi` |
| DevKit | **2.7.0** | `omi/firmware/devkit` |
Using the wrong SDK version will cause build failures. Double-check the version before proceeding.
1. Open the **nRF Connect Extension** panel in VS Code
2. Click **Open an existing application**
3. Navigate to `omi/firmware/omi` (consumer) or `omi/firmware/devkit` (DevKit)
1. In the application panel, click **Add Build Configuration**
2. Set the board target:
| Firmware | Board target | Build output |
| - | - | - |
| Consumer (Omi) | `omi/nrf5340/cpuapp` | `dfu_application.zip` + `merged.hex` + `merged_CPUNET.hex` (sysbuild) |
| DevKit | `xiao_ble_sense` (XIAO nRF52840 Sense) | `zephyr.uf2` |
These targets are not interchangeable. The consumer CV1 is an nRF5340 image
built from `omi/firmware/omi` with **sysbuild** (MCUboot-signed
`dfu_application.zip` plus `merged.hex` and `merged_CPUNET.hex`, not a UF2).
The DevKit UF2 is built only from `omi/firmware/devkit`.
In nRF Connect, use the `OMI` preset in `omi/firmware/omi/CMakePresets.json`
and turn **sysbuild** on. The preset sets the board and `CONF_FILE=omi.conf`;
it does not enable sysbuild.
Or, from the west workspace `omi/firmware/v2.9.0`, after
`cp ../omi/omi.conf ../omi/prj.conf`:
```bash theme={null}
west build -b omi/nrf5340/cpuapp ../omi --sysbuild -d build --pristine always -- -DBOARD_ROOT=..
```
The DevKit presets use the legacy spelling `xiao_ble_sense`. NCS 2.7.0 maps it to `xiao_ble/nrf52840/sense`, the spelling used by `omi/firmware/scripts/build-firmware-in-docker.sh`; both select the same board.
3. Click **Build Configuration** to start the build
4. A VS Code notification will show build progress
After a successful build, find the outputs:
```text theme={null}
DevKit (UF2, drag-and-drop):
omi/firmware/devkit/build//zephyr/zephyr.uf2
Consumer CV1, nRF Connect preset OMI with sysbuild on:
omi/firmware/omi/build/omi/dfu_application.zip
omi/firmware/omi/build/omi/merged.hex
omi/firmware/omi/build/omi/merged_CPUNET.hex
Consumer CV1, west command above (-d build inside omi/firmware/v2.9.0):
omi/firmware/v2.9.0/build/dfu_application.zip
omi/firmware/v2.9.0/build/merged.hex
omi/firmware/v2.9.0/build/merged_CPUNET.hex
```
Follow the [Flash Device](/doc/get_started/Flash_device) guide to install the compiled firmware on your device.
***
## Related
Install pre-built or custom firmware
Omi consumer hardware specs
# Examples
Source: https://docs.omi.me/doc/developer/mcp/examples
Example prompts and integrations with the Omi MCP server
## Example Prompts
Once connected, try these prompts with your AI assistant:
```
"What do you know about my work projects?"
"Find conversations where I discussed travel plans"
"Search my memories for anything about fitness goals"
"What did I talk about last Tuesday?"
```
```
"Remember that I prefer morning meetings before 10am"
"Update my memory about my favorite restaurant — it's now Sushi Nakazawa"
"Delete the memory about my old phone number"
"What are all my memories in the 'work' category?"
```
```
"Summarize my conversations from this week"
"Find the conversation where I brainstormed app ideas"
"What topics come up most in my recent conversations?"
"Show me the full transcript of my last meeting"
```
***
## Integration Examples
Build chains with Omi data
Create AI agents using Omi
Programmatic LLM pipelines
***
## Python SDK Example
This programmatic example uses the manual MCP-key fallback because the adapter receives
static headers. Create an `omi_mcp_...` key as described in the [Setup guide](/doc/developer/mcp/setup)
and load it from a secret store or environment variable in production.
```python theme={null}
import asyncio
import os
from langchain_mcp_adapters.client import MultiServerMCPClient
from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI
async def main():
# Current langchain-mcp-adapters API: the client is not a context manager —
# construct it and await get_tools().
client = MultiServerMCPClient({
"omi": {
"url": "https://api.omi.me/v1/mcp",
"transport": "streamable_http",
"headers": {"Authorization": f"Bearer {os.environ['OMI_MCP_API_KEY']}"},
}
})
tools = await client.get_tools()
agent = create_react_agent(ChatOpenAI(model="gpt-4o-mini"), tools)
response = await agent.ainvoke({
"messages": "Search my memories for anything about my morning routine"
})
print(response["messages"][-1].content)
if __name__ == "__main__":
asyncio.run(main())
```
# Introduction
Source: https://docs.omi.me/doc/developer/mcp/introduction
Connect AI assistants to your Omi data using the Model Context Protocol
## What is MCP?
The [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) is an open standard that lets AI assistants like Claude, Cursor, and other tools interact with external data sources. Omi's MCP server gives these assistants direct access to your memories and conversations.
Semantic search across your memories and conversations
Create, edit, and delete memories
Browse and search full conversation transcripts
***
## How It Works
```mermaid theme={null}
sequenceDiagram
participant A as AI Assistant (Claude, Cursor, etc.)
participant M as Omi MCP Server
participant D as Your Omi Data
A->>M: tools/list (discover available tools)
M-->>A: Tools allowed by the user's OAuth grant or MCP key
A->>M: tools/call search_memories {query: "morning routine"}
M->>D: Semantic search via Pinecone
D-->>M: Ranked results
M-->>A: Memories with relevance scores
```
Your AI assistant connects to the Omi MCP server, discovers the tools allowed by its
authorization, and calls them as needed during your conversations. The hosted server
supports OAuth access tokens (the default for supported ChatGPT and Claude cloud flows)
and manually provisioned `omi_mcp_...` keys for compatible clients and local servers.
The same hosted MCP connection works across Claude, Claude Code, ChatGPT, Codex, and any compatible client. It exposes the user's Omi context as scoped tools: durable memories, conversations, people, action items, goals, chat history, daily summaries, and synced screen activity. The server's initialization instructions tell clients to retrieve task-specific evidence, confirm important claims against source records, and avoid writes unless the user clearly requested them.
When the Omi macOS app connects Claude Code or Codex, it also installs a small `omi` skill in that client's personal skills directory. The skill contains routing guidance only—never credentials or user data—and Omi preserves an existing user-authored skill with the same name.
***
## Available Tools
| Tool | Description |
| - | - |
| `get_memories` | List memories with optional category filtering |
| `search_memories` | Semantic search across memories |
| `create_memory`, `create_memories` | Create one memory, or up to 25 in a bounded batch |
| `edit_memory` | Edit an existing memory |
| `delete_memory` | Delete a memory |
| `get_conversations` | List conversations with date/category filters |
| `search_conversations` | Semantic search across conversations |
| `get_conversation_by_id`, `get_conversations_by_ids` | Get full conversations with transcripts — one, or up to 20 in a bounded batch |
| `get_user_profile` | Get Omi's cached high-level user summary |
| `get_x_posts`, `search_x_posts` | Browse or search imported X posts and bookmarks |
| `get_action_items`, `search_action_items` | Browse or search tasks |
| `create_action_item`, `update_action_item`, `complete_action_item`, `delete_action_item` | Manage tasks |
| `get_goals` | Retrieve active or historical goals |
| `get_chat_messages` | Retrieve recent Omi chat history |
| `get_people` | Retrieve recognized people and speaker samples |
| `rename_person` | Correct a recognized person's display name while retaining the old name as an alias |
| `dismiss_person` | Soft-dismiss a false-positive person from normal People reads |
| `get_screen_activity` | Retrieve or summarize synced screen activity |
| `get_daily_summaries` | Retrieve Omi's daily summaries |
See the [Tools Reference](/doc/developer/mcp/tools) for full parameter documentation.
***
## Comparison with Developer API
| Feature | MCP | Developer API |
| - | - | - |
| **Purpose** | AI assistant integration | Direct HTTP API access |
| **Access** | Read/write with AI context | Read & write user data |
| **Search** | Semantic search built-in | Filter-based queries |
| **Use Case** | Claude Desktop, Cursor, AI agents | Custom apps, dashboards |
| **Best For** | AI-powered workflows | Batch operations, integrations |
For programmatic access without AI assistants, use the [Developer API](/doc/developer/api/overview) instead.
# Setup
Source: https://docs.omi.me/doc/developer/mcp/setup
Connect your AI assistant to the Omi MCP server
## Choose a setup method
Omi supports four setup paths:
| Method | Authentication | Best for |
| - | - | - |
| Omi's guided connection | OAuth for ChatGPT and Claude; a generated MCP key for local clients | The current Omi macOS app (recommended) |
| Hosted server with OAuth | Browser sign-in and consent; no key to copy | Supported cloud clients |
| Hosted server with an MCP key | `Authorization: Bearer omi_mcp_...` | Clients that accept custom headers |
| Local stdio server | An MCP key passed to `mcp-server-omi` | Clients that require a local process or a self-hosted backend |
OAuth and MCP keys grant access to the same hosted endpoint. OAuth is the default for the
new cloud-connector UI; manual MCP keys remain available as a fallback. Cleanup capabilities
that hide retained data, such as `people.cleanup`, are never added to a default or legacy MCP
key. They must be requested explicitly when the OAuth grant or key is created.
## Connect from the Omi macOS app (Recommended)
On the Omi home screen, find **Use omi memory anywhere**, choose a destination, and follow
the connection card:
* **ChatGPT:** choose **ChatGPT / Codex**, then add Omi from its approved ChatGPT listing.
ChatGPT opens Omi's OAuth consent flow; you do not need to create or paste a key.
* **Claude:** choose **Claude / Claude Code**, then **Claude (cloud)**. Omi opens Claude's
custom-connector flow with the registered public OAuth client.
* **Claude Code, Codex, OpenClaw, and Hermes:** Omi generates an MCP connection key and
offers guided local setup. Expand **Manual installation** to copy the server URL, key,
command, or configuration yourself.
* **ChatGPT custom app:** expand **Developer-mode fallback** only if your workspace cannot
use the approved directory listing.
OAuth grants can be reviewed or revoked from the connected client. Revoking an MCP key is
separate; use the Developer settings described below.
***
## Hosted Server with OAuth
Use the hosted server URL as the remote MCP URL:
```text theme={null}
https://api.omi.me/v1/mcp
```
The endpoint speaks **Streamable HTTP** (MCP `2026-07-28`; older protocol
versions remain supported) and advertises OAuth metadata at:
```text theme={null}
https://api.omi.me/.well-known/oauth-protected-resource/v1/mcp
https://api.omi.me/.well-known/oauth-authorization-server
```
`https://api.omi.me/v1/mcp/sse` is a permanent compatibility alias of the same
endpoint (older configs and OAuth metadata keep working). New setups should
always use the canonical `/v1/mcp` URL above.
OAuth uses browser sign-in, explicit consent, authorization code + PKCE, and refresh
tokens. Omi provides registered public clients for its ChatGPT and Claude setup
flows. A generic MCP client that supports **Client ID Metadata Documents** can
also identify itself with an HTTPS `client_id` URL (PKCE); dynamic client
registration is not offered. Omitting the `scope` parameter grants all
available read scopes — writes are never included by default.
The preferred path is **Omi → Use omi memory anywhere → ChatGPT / Codex → ChatGPT
(cloud)**. Add Omi from the directory page and approve the OAuth consent screen.
For the advanced developer-mode fallback, use:
* **Connection / server URL:** `https://api.omi.me/v1/mcp`
* **Authentication:** OAuth
* **OAuth Client ID:** `omi-chatgpt-prod`
* **OAuth Client Secret:** leave blank
* **Token auth method:** `none`
* **Authorization URL:** `https://api.omi.me/authorize`
* **Token URL:** `https://api.omi.me/token`
In Claude, open **Customize → Connectors → Add custom connector**, then use:
* **Name:** Omi Memory
* **Remote MCP server URL:** `https://api.omi.me/v1/mcp`
* **Advanced settings** (only if asked): OAuth Client ID `omi-claude-prod`,
OAuth Client Secret **leave blank**
Click **Add**, then **Connect**, and approve Omi's OAuth consent screen. The Omi macOS
app exposes the same flow under **Use omi memory anywhere → Claude / Claude Code →
Claude (cloud)**.
***
## Manual MCP-key fallback
Use a manual key for clients that support a bearer header but cannot use Omi's registered
OAuth flows.
In the current macOS UI, open **Use omi memory anywhere**, choose Claude Code, Codex,
OpenClaw, or Hermes, and expand **Manual installation**. Omi generates the connection
key and shows a masked **Your key** row with a **Copy** button.
You can also use the cross-platform Omi app: open **Settings → Developer Settings**,
scroll to **MCP Server**, and create a key in its **API Keys** list. The complete
`omi_mcp_...` value is shown only when it is created, so copy and store it then.
Your key will look like: `omi_mcp_...`
MCP keys are distinct from Developer API keys. An `omi_mcp_...` key authenticates the
hosted MCP endpoint and `/v1/mcp/...` REST routes. An `omi_dev_...` key authenticates
`/v1/dev/...` routes and will not authenticate MCP.
Use the following connection details:
* **Server URL:** `https://api.omi.me/v1/mcp`
* **Authorization:** `Bearer omi_mcp_...` (your generated key)
* **Transport:** Streamable HTTP (MCP `2026-07-28`; older protocol versions supported)
***
## Manual hosted-client configuration
Claude Desktop adds remote MCP servers through the Connectors UI — not a
config file. Open **Settings → Connectors → Add custom connector**, then use:
* **Name:** Omi Memory
* **Remote MCP server URL:** `https://api.omi.me/v1/mcp`
* **Advanced settings** (only if asked): OAuth Client ID `omi-claude-prod`,
OAuth Client Secret **leave blank**
Click **Add**, then **Connect**, and approve Omi's OAuth consent screen.
`claude_desktop_config.json` only launches local stdio `command` servers — it
cannot hold a hosted HTTP entry. For a manual-key local path use the
[deprecated local stdio fallback](#local-stdio-server-with-uvx-or-python-deprecated).
Add the hosted server at user scope:
```bash theme={null}
claude mcp add --scope user --transport http omi-memory \
https://api.omi.me/v1/mcp \
--header "Authorization: Bearer omi_mcp_YOUR_KEY_HERE"
```
Equivalently, add the entry to `~/.claude.json` directly:
```json theme={null}
{
"mcpServers": {
"omi": {
"type": "http",
"url": "https://api.omi.me/v1/mcp",
"headers": {
"Authorization": "Bearer omi_mcp_YOUR_KEY_HERE"
}
}
}
}
```
Add a native remote entry to your `mcp.json` (project `.cursor/mcp.json` or
global `~/.cursor/mcp.json`) — Streamable HTTP, no SSE transport selection:
```json theme={null}
{
"mcpServers": {
"omi": {
"url": "https://api.omi.me/v1/mcp",
"headers": {
"Authorization": "Bearer omi_mcp_YOUR_KEY_HERE"
}
}
}
}
```
Add a native HTTP entry to `~/.codex/config.toml` — no `mcp-remote` bridge:
```toml theme={null}
[mcp_servers.omi]
url = "https://api.omi.me/v1/mcp"
http_headers = { Authorization = "Bearer omi_mcp_YOUR_KEY_HERE" }
```
To keep the key out of the file, Codex also supports
`bearer_token_env_var = "OMI_MCP_API_KEY"` (requires that env var to be set
wherever Codex runs). Restart Codex after saving the file.
Add to your project's `opencode.json` (or `~/.config/opencode/opencode.json` globally):
```json theme={null}
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"omi": {
"type": "remote",
"url": "https://api.omi.me/v1/mcp",
"enabled": true,
"headers": {
"Authorization": "Bearer {env:OMI_MCP_API_KEY}"
}
}
}
}
```
Store your key in a local `.env` file (or export `OMI_MCP_API_KEY` in your environment):
```bash theme={null}
OMI_MCP_API_KEY=omi_mcp_YOUR_KEY_HERE
```
Be sure to use an MCP key from one of the locations in
[Manual MCP-key fallback](#manual-mcp-key-fallback). It starts with `omi_mcp_`.
Developer API keys (`omi_dev_...`) only authenticate Developer API endpoints and will
return `401 Unauthorized` here.
Enter the server URL and API key in Poke's MCP connection settings.
The Omi macOS app provides the current command or configuration under each client's
**Manual installation** disclosure. Copy it there so the generated key is inserted
without retyping it.
Any MCP client that supports **Streamable HTTP** and custom authorization headers can
use the manual-key path. With the `2026-07-28` protocol there is no
initialize handshake or session ID — each request carries the protocol
version and client identity in `_meta`, plus the `MCP-Protocol-Version` /
`Mcp-Method` headers. Discovery starts with `server/discover`:
```bash theme={null}
POST https://api.omi.me/v1/mcp
Authorization: Bearer omi_mcp_YOUR_KEY_HERE
Content-Type: application/json
MCP-Protocol-Version: 2026-07-28
Mcp-Method: server/discover
{
"jsonrpc": "2.0",
"id": 1,
"method": "server/discover",
"params": {},
"_meta": {
"io.modelcontextprotocol/protocolVersion": "2026-07-28",
"io.modelcontextprotocol/clientCapabilities": {},
"io.modelcontextprotocol/clientInfo": {"name": "my-client", "version": "1.0"}
}
}
```
Older handshake clients still work: send an `initialize` request with your
`protocolVersion` (e.g. `2025-03-26`) and the server answers it in that
protocol revision — no `Mcp-Session-Id` management or GET stream is
required.
***
## Local stdio server with uvx or Python (deprecated)
The local `mcp-server-omi` package is **deprecated** — the hosted endpoint above
is the supported path for every client that accepts a remote URL. Keep this
section only for clients that genuinely require a local stdio process or a
self-hosted backend.
Run the MCP server locally over the standard input/output transport. `uvx` (which
ships with [uv](https://docs.astral.sh/uv/)) downloads and runs the published
package without a separate installation. Python 3.11.6 or newer is required.
Add this to your MCP client's local-server configuration:
```json theme={null}
{
"mcpServers": {
"omi": {
"command": "uvx",
"args": ["mcp-server-omi"],
"env": {
"OMI_API_KEY": "omi_mcp_YOUR_KEY_HERE"
}
}
}
}
```
Install [uv](https://docs.astral.sh/uv/getting-started/installation/) first. The
`OMI_API_KEY` value can also be supplied with each tool call.
Install the package in a Python 3.11.6+ environment:
```bash theme={null}
pip install mcp-server-omi
```
Then configure your MCP client to launch the local server:
```json theme={null}
{
"mcpServers": {
"omi": {
"command": "mcp-server-omi",
"env": {
"OMI_API_KEY": "omi_mcp_YOUR_KEY_HERE"
}
}
}
}
```
Generate the MCP key using either location in [Manual MCP-key fallback](#manual-mcp-key-fallback).
Developer API keys that start with `omi_dev_` will not authenticate MCP requests.
***
## Local stdio server with Docker (deprecated)
Migration-only, like the uvx path above — prefer the hosted endpoint. If you
must run the MCP server locally:
Generate an MCP key using either location in
[Manual MCP-key fallback](#manual-mcp-key-fallback).
Install Docker. We recommend [OrbStack](https://orbstack.dev/) for macOS.
Add to your `claude_desktop_config.json`:
```json theme={null}
{
"mcpServers": {
"omi": {
"command": "docker",
"args": ["run", "--rm", "-i", "-e", "OMI_API_KEY=omi_mcp_YOUR_KEY_HERE", "omiai/mcp-server"]
}
}
}
```
The same Docker command can be used by any MCP client that supports local stdio servers.
If you prefer to keep the key out of the configuration file, pass it through the
`OMI_API_KEY` environment variable:
```bash theme={null}
docker run --rm -i \
-e OMI_API_KEY="omi_mcp_YOUR_KEY_HERE" \
omiai/mcp-server
```
The API key can also be provided with each tool call. If not provided, the
server uses the `OMI_API_KEY` environment variable as a fallback.
***
## Custom Backend URL
Only applies to the **local `mcp-server-omi` package** above when self-hosting
the Omi backend. The package appends REST segments (`memories`,
`conversations/...`) directly, so the value **must end with `/v1/mcp/`**:
```bash theme={null}
export OMI_API_BASE_URL="https://your-backend-url.com/v1/mcp/"
```
This is the *REST base* for the self-hosted package — distinct from the
hosted MCP endpoint remote clients connect to (`https://api.omi.me/v1/mcp`,
no trailing slash). Only needed for self-hosted Omi instances.
# Tools Reference
Source: https://docs.omi.me/doc/developer/mcp/tools
Complete reference for all Omi MCP tools
## Memory Tools
The hosted server returns only the tools allowed by the OAuth grant or MCP key. A local
`mcp-server-omi` stdio process currently exposes the eight memory and conversation tools
in the first two sections below; the additional tools are hosted-server tools.
Retrieve a list of user memories with optional filtering.
**Parameters:**
| Name | Type | Required | Description |
| - | - | - | - |
| `categories` | array | No | Categories to filter by |
| `limit` | number | No | Maximum number of memories (default: 20, max: 100) |
| `offset`/`cursor` | number/string | No | Offset for pagination, or opaque `next_cursor` from a previous page (mutually exclusive) |
| `sort` | string | No | `scoring_desc`, `created_desc`, `updated_desc`, or `manual_first` |
| `reviewed` | boolean | No | Filter by reviewed state |
| `manually_added` | boolean | No | Filter by manually-added state |
| `updated_after` | string | No | Only return memories updated after this ISO 8601 timestamp |
| `include_activity` | boolean | No | Include focus/screen/activity memories, which are excluded by default |
| `include_sensitive` | boolean | No | Include memories above standard data protection, default `true` for compatibility |
**Returns:** `{ "memories": [...], "returned_count": 25, "has_more": true, "offset": 0, "limit": 25, "sort": "created_desc" }`
The response may also include scan diagnostics such as `scanned_count` and `scan_truncated`.
**Example:**
```
"Search my memories" → get_memories with no filters
"What do you know about my hobbies?" → get_memories with categories: ["hobbies"]
```
**Memory categories:** `interesting`, `core`, `hobbies`, `lifestyle`, `interests`, `habits`, `work`, `skills`, `learnings`, `other`
Semantic search across memories. Returns results ranked by relevance using vector similarity.
**Parameters:**
| Name | Type | Required | Description |
| - | - | - | - |
| `query` | string | Yes | Natural language search query |
| `limit` | number | No | Maximum number of results (default: 10) |
**Returns:** `{ "memories": [{ ..., "relevance_score": 0.92 }, ...] }`
Each result includes a `relevance_score` (0.0 to 1.0) indicating how well it matches the query.
**Example:**
```
"What do I know about machine learning?" → search_memories with query: "machine learning"
"Find memories about my morning routine" → search_memories with query: "morning routine"
```
Create a new memory. Category is auto-detected if not provided.
**Parameters:**
| Name | Type | Required | Description |
| - | - | - | - |
| `content` | string | Yes | Content of the memory |
| `category` | string | No | Category (auto-detected if omitted) |
**Returns:** `{ "success": true, "memory": { ... } }`
Create up to **25** memories in one call — prefer this over repeated
`create_memory` calls when saving several facts.
**Parameters:**
| Name | Type | Required | Description |
| - | - | - | - |
| `items` | array | Yes | 1–25 objects of `{ "content": string, "category"?: string }` |
**Returns:** `{ "results": [...] }` — every item is rate-limited
individually and returns its own status: `created`, `duplicate` (exact
content+category already created earlier in the same batch), or `error`.
Edit an existing memory's content.
**Parameters:**
| Name | Type | Required | Description |
| - | - | - | - |
| `memory_id` | string | Yes | ID of the memory to edit |
| `content` | string | Yes | New content for the memory |
**Returns:** `{ "success": true }`
Delete a memory by ID.
**Parameters:**
| Name | Type | Required | Description |
| - | - | - | - |
| `memory_id` | string | Yes | ID of the memory to delete |
**Returns:** `{ "success": true }`
***
## Conversation Tools
Retrieve a list of conversations with optional date and category filtering.
**Parameters:**
| Name | Type | Required | Description |
| - | - | - | - |
| `start_date` | string | No | Filter after this date (YYYY-MM-DD) |
| `end_date` | string | No | Filter before this date (YYYY-MM-DD) |
| `categories` | array | No | Categories to filter by |
| `limit` | number | No | Maximum number of conversations (default: 20) |
| `offset`/`cursor` | number/string | No | Offset for pagination, or opaque `next_cursor` from a previous page (mutually exclusive) |
**Returns:** `{ "conversations": [...] }` — metadata only. Use `get_conversation_by_id` for full transcripts.
**Example:**
```
"What did I talk about last week?" → get_conversations with date range
"Show my work conversations" → get_conversations with categories: ["work"]
```
**Conversation categories:** `personal`, `education`, `health`, `finance`, `technology`, `business`, `work`, `social`, `travel`, `entertainment`, `sports`, `family`, and more.
Semantic search across conversations. Returns results ranked by relevance using vector similarity.
**Parameters:**
| Name | Type | Required | Description |
| - | - | - | - |
| `query` | string | Yes | Natural language search query |
| `start_date` | string | No | Filter after this date (YYYY-MM-DD) |
| `end_date` | string | No | Filter before this date (YYYY-MM-DD) |
| `limit` | number | No | Maximum number of results (default: 10) |
**Returns:** `{ "conversations": [...] }` — ranked by relevance to the query.
**Example:**
```
"When did I discuss the product launch?" → search_conversations with query: "product launch"
"Find conversations about hiring from January" → search_conversations with query: "hiring", start_date: "2026-01-01", end_date: "2026-01-31"
```
Retrieve a single conversation by ID, including the full transcript with speaker segments.
**Parameters:**
| Name | Type | Required | Description |
| - | - | - | - |
| `conversation_id` | string | Yes | The ID of the conversation |
| `max_segments` | number | No | Max transcript segments (default 120, max 500) |
| `max_chars` | number | No | Max transcript characters (default 24000, max 100000) |
**Returns:** Full conversation object with transcript segments, timestamps, structured summary, and metadata.
Deep-read up to **20** conversations in one call — the preferred follow-up
after `get_conversations` or `search_conversations` returns several
relevant ids.
**Parameters:**
| Name | Type | Required | Description |
| - | - | - | - |
| `conversation_ids` | array | Yes | 1–20 conversation IDs (duplicates fetched once) |
| `max_segments` | number | No | Max transcript segments per conversation (same bounds as `get_conversation_by_id`) |
| `max_chars` | number | No | Max transcript characters per conversation (same bounds) |
**Returns:** `{ "conversations": [...], "not_found": [...], "truncated": false }`
— each item carries the same bounded card and transcript as
`get_conversation_by_id` with its own `truncated` flag; ids that resolve to
nothing are listed in `not_found`. When the response budget is hit, later
items are omitted and top-level `truncated` is `true` — retry them with
smaller `max_segments`/`max_chars`.
***
## Profile, Imported Data, and Activity Tools
| Tool | Parameters | Description |
| - | - | - |
| `get_user_profile` | None | Return Omi's cached high-level user summary, if generated |
| `get_x_posts` | `kind?` (`tweet` or `bookmark`), `limit?` (default 50) | List imported X posts and bookmarks, newest first |
| `search_x_posts` | `query`, `limit?` (default 10) | Semantic search over imported X posts and bookmarks |
| `get_goals` | `include_inactive?` (default false) | List goals, active by default |
| `get_chat_messages` | `limit?` (default 50), `offset?`/`cursor?` | List recent Omi chat messages, newest first |
| `get_people` | None | List recognized people with speaker samples |
| `rename_person` | `person_id`, `name` | Correct a display name without exposing speaker samples in the response |
| `dismiss_person` | `person_id` | Soft-dismiss a false-positive person; requires the opt-in `people.cleanup` scope |
| `get_screen_activity` | `start_date?`, `end_date?`, `app?`, `group_by?` (`none`\|`app`\|`hour`\|`day`, default `none`), `summary?`, `limit?`, `cursor?` | List synced screen activity rows, aggregate them into buckets, or return the legacy per-app summary |
| `get_daily_summaries` | `start_date?`, `end_date?`, `limit?` (default 30), `offset?` | List Omi's daily summaries, newest first |
Dates use `YYYY-MM-DD`. `get_screen_activity` defaults to 200 raw rows per page
(max 1000); `limit` is ignored when `summary` is true. `group_by` buckets rows
by app, hour, or day with counts, estimated observation seconds (bounded
capture gaps — never actual usage duration), and top window titles.
Screen summaries count **synced observations**, not elapsed app usage. OCR gating,
frame deduplication, privacy exclusions, and sync compaction mean a row cannot be
converted to a fixed number of seconds or treated as proof of the user's intent.
The same summary contract serves hosted MCP, REST MCP, and
`GET /v1/screen-activity/summary`:
* `total_screenshots` and per-app `count` count only the summarized rows.
* `coverage.source` is `synced_screen_activity`; `coverage.row_limit` is 5000.
* `coverage.truncated` is true only when a lookahead row proves more matching
rows exist. Only the earliest 5000 rows contribute to the summary. Narrow the
date range to inspect later observations.
* `coverage.first_observed_at` and `last_observed_at` bound the summarized rows,
in UTC (`YYYY-MM-DD HH:MM:SS.mmm`), and are null for an empty result. They do
not establish continuous activity or the device's latest capture time.
* `coverage.capture_completeness` is `unknown`, even when `truncated` is false:
the cloud query cannot attest to device capture status, excluded activity,
retention, or pending sync. An empty result does not prove inactivity.
Window titles are samples, not a ranking by dwell time. Summaries remain derived
views of screen evidence; they do not create or promote semantic memories.
`rename_person` requires `people.rename`, which is part of the compatible default MCP key grant.
`dismiss_person` requires the explicitly requested `people.cleanup` scope. Dismissal is soft: it
hides the person from default app and MCP People reads while preserving the record for data export.
Existing and newly created default keys do not receive `people.cleanup` automatically.
***
## Pagination and cursors
List tools that can return more results emit an opaque `next_cursor` in the
response. Pass it back as the `cursor` argument — together with the **same**
filters — to fetch the next page (`cursor` and a non-zero `offset` are mutually
exclusive). The field is absent when no further page exists.
The same key authenticates the REST `/v1/mcp/*` list endpoints; they carry the
cursor in the **`X-Next-Cursor`** response header instead of the body.
## Incremental sync (`updated_since`)
`GET /v1/mcp/action-items` accepts `updated_since`, a strict ISO-8601 timestamp
with an explicit timezone offset. It switches the feed to an incremental
`(updated_at ASC, id ASC)` keyset — combine it with the `X-Next-Cursor` cursor
to page through updates. On each sync pass, re-read starting from
`watermark − 60 seconds` and deduplicate by `id`: `updated_at` is a timestamp,
not a transaction sequence, and the overlap covers delayed-commit visibility.
Note that action-item deletes are **hard deletes** — a deleted item leaves no
row and is not emitted by the feed.
`GET /v1/mcp/conversations` and `GET /v1/mcp/memories` do **not** support
`updated_since`: a valid timestamp returns HTTP `400` with detail
`incremental_sync_unsupported` (permanent capability gap — no `Retry-After`,
do not retry). For conversations, page instead with the
`(created_at DESC, id)` keyset cursor.
***
## Action Item Tools
| Tool | Required parameters | Optional parameters | Description |
| - | - | - | - |
| `get_action_items` | None | `completed`, `due_start_date`, `due_end_date`, `limit` (default 100), `offset`/`cursor` | List tasks and to-dos |
| `search_action_items` | `query` | `limit` (default 10) | Semantic search over tasks |
| `create_action_item` | `description` | `due_at`, `completed` (default false) | Create a task; identical retries do not duplicate it |
| `complete_action_item` | `action_item_id` | `completed` (default true) | Complete or reopen a task |
| `update_action_item` | `action_item_id` | `description`, `due_at` | Update only the supplied fields |
| `delete_action_item` | `action_item_id` | None | Delete a task |
Due dates accept `YYYY-MM-DD` or an ISO 8601 date-time where noted by the client schema.
***
## Error Handling
A `tools/call` result always carries `structuredContent` (a JSON value) plus a
matching text block for compatibility. **Tool execution failures** come back as
a tool result — visible to the model — with `isError: true` and
`structuredContent.error` of `{ "code": string, "message": string }`; `code` is
one of:
| `structuredContent.error.code` | Meaning |
| - | - |
| `invalid_arguments` | Bad parameter (bad date format, unknown category, malformed ID) |
| `not_found` | Resource not found (memory or conversation doesn't exist) |
| `paid_plan_required` | Locked content (paid plan required) |
| `authorization_denied` | The OAuth grant is missing the tool's required scope, or policy denied the operation |
| `rate_limited` | Per-user rate limit hit — retry after the current window |
| `unavailable` | Temporarily unavailable (e.g. a search index still building) |
| `internal` | Unexpected server-side failure |
**Protocol failures** stay JSON-RPC errors: unknown method/tool (`-32601`),
malformed params (`-32602`), unsupported protocol version (`-32022`), and
header/body mismatches (`-32020`) never reach the tool result.
**Example tool failure** — the text block mirrors the serialized
`structuredContent`, so both forms carry the same error:
```json theme={null}
{
"isError": true,
"content": [{"type": "text", "text": "{\"error\":{\"code\":\"not_found\",\"message\":\"Memory not found\"}}"}],
"structuredContent": {"error": {"code": "not_found", "message": "Memory not found"}}
}
```
***
## Locked Content
Memories and conversations behind the paid plan are handled gracefully:
* **Memories:** Content is truncated to 70 characters with `...`
* **Conversations:** Action items and events are hidden from the structured data
* **Direct access:** Returns a tool result with `isError: true` and
`structuredContent.error.code` of `paid_plan_required`, with a clear message
# Troubleshooting
Source: https://docs.omi.me/doc/developer/mcp/troubleshooting
Debug and fix common MCP connection issues
## Common Issues
* Confirm the remote server URL is exactly `https://api.omi.me/v1/mcp`
* Use Omi's registered ChatGPT or Claude connection flow, or a client that identifies itself with an HTTPS **Client ID Metadata Document** URL (`client_id`) + PKCE — dynamic client registration is not offered, and the client must send the user through Omi's consent flow on one of its registered redirect URIs
* For Claude, use client ID `omi-claude-prod` and leave the client secret blank
* For ChatGPT's developer-mode fallback, use `omi-chatgpt-prod`, leave the secret blank, and set token auth method to `none`
* If the client cannot complete OAuth, use the manual MCP-key fallback in the [Setup guide](/doc/developer/mcp/setup)
* OAuth clients: reconnect so the client can refresh or replace its OAuth access token
* Manual-key clients: verify the key starts with `omi_mcp_`
* Check that the header uses the `Bearer` prefix: `Bearer omi_mcp_...`
* Generate a new key from the macOS connection card's **Manual installation** section, or from **Settings → Developer Settings → MCP Server → API Keys** in the cross-platform app
* Ensure the OAuth grant or MCP key has not been revoked
An `omi_mcp_...` key authenticates the MCP endpoints only. Two things follow:
* **Use the `/v1/mcp/` REST endpoints** — e.g. `GET /v1/mcp/memories`, `GET /v1/mcp/memories/search?query=...`, `GET /v1/mcp/conversations`, `GET /v1/mcp/action-items`. They accept the same key as the MCP server and return plain JSON.
* **`/v1/memories` and `/v2/memories` do not exist.** Requests to them return `404 Not Found` regardless of the key.
```bash theme={null}
curl -H "Authorization: Bearer omi_mcp_YOUR_KEY" \
"https://api.omi.me/v1/mcp/memories?limit=5"
```
To call the [Developer API](/doc/developer/api/overview) (`/v1/dev/...`) instead, create a separate
key in **Settings → Developer Settings → Developer API Keys**; it starts with `omi_dev_`. Using either key on the
other's endpoints returns a `401` that names the endpoints that key does authenticate.
* With protocol `2026-07-28` there is no initialize handshake — call
`server/discover` then `tools/list` directly; older clients still need
their `initialize` handshake first
* Check that your client supports the Streamable HTTP transport (the server
advertises `2026-07-28` and answers older protocol revisions)
* Try the `/v1/mcp/sse/info` endpoint to verify the server is reachable:
```bash theme={null}
curl https://api.omi.me/v1/mcp/sse/info
```
* Semantic search requires conversations/memories to be indexed in the vector database
* New data may take a few minutes to be indexed after creation
* Try broader queries — very specific queries may not match if phrased differently than the original
* Use `get_memories` or `get_conversations` with filters as a fallback
* The MCP server has per-user rate limits to prevent abuse
* Wait a moment and retry
* Reduce the frequency of tool calls in automated workflows
* Some memories and conversations are behind the paid plan
* Upgrade your plan to access all content
* Locked memories still appear in list/search results with truncated content
***
## Debugging Tools
The official inspector CLI can exercise the hosted endpoint directly:
```bash theme={null}
npx @modelcontextprotocol/inspector --cli https://api.omi.me/v1/mcp \
--transport http --method tools/list \
--header "Authorization: Bearer $OMI_MCP_API_KEY"
```
For local development with the deprecated stdio package:
```bash theme={null}
npx @modelcontextprotocol/inspector uvx mcp-server-omi
```
Test the server directly. With protocol `2026-07-28` there is no handshake
or session header — every request carries `_meta` plus the
`MCP-Protocol-Version`/`Mcp-Method` headers:
```bash theme={null}
# Check server info
curl https://api.omi.me/v1/mcp/sse/info
# Discover (2026-07-28)
curl -X POST https://api.omi.me/v1/mcp \
-H "Authorization: Bearer omi_mcp_YOUR_KEY" \
-H "Content-Type: application/json" \
-H "MCP-Protocol-Version: 2026-07-28" \
-H "Mcp-Method: server/discover" \
-d '{"jsonrpc":"2.0","id":1,"method":"server/discover","params":{},
"_meta":{"io.modelcontextprotocol/protocolVersion":"2026-07-28",
"io.modelcontextprotocol/clientCapabilities":{}}}'
# List tools (2026-07-28)
curl -X POST https://api.omi.me/v1/mcp \
-H "Authorization: Bearer omi_mcp_YOUR_KEY" \
-H "Content-Type: application/json" \
-H "MCP-Protocol-Version: 2026-07-28" \
-H "Mcp-Method: tools/list" \
-d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{},
"_meta":{"io.modelcontextprotocol/protocolVersion":"2026-07-28",
"io.modelcontextprotocol/clientCapabilities":{}}}'
```
Older handshake clients send `initialize` with their `protocolVersion`
(e.g. `2025-03-26`) instead; the server answers in that revision and no
`Mcp-Session-Id` tracking is required.
View Claude Desktop MCP logs:
```bash theme={null}
# macOS
tail -n 20 -f ~/Library/Logs/Claude/mcp-server-omi.log
# Windows PowerShell
Get-Content "$env:APPDATA\Claude\logs\mcp-server-omi.log" -Tail 20 -Wait
```
***
## Need Help?
Get help from the community and team
Report bugs or request features
# First-Party Memory Clients
Source: https://docs.omi.me/doc/developer/memory-clients
The shared memory currency and history contract for Omi's first-party clients
This page describes the shared contract used by Omi's macOS, Flutter, web, and
Windows clients. It is an internal first-party client contract, not the
Developer API. The Developer API reference remains the source for
`/v1/dev/user/memories`.
## Beta capability
When the memory belief feature is available, the backend returns the exact
header `X-Omi-Memory-Belief-Enabled: true`. It returns `false` when the
capability is unavailable. The header is present on both `/v3/memories` and
`/v3/memories/ledger-history`, including empty responses, and is exposed to
browser clients through CORS.
Clients scope their cached capability state to the signed-in owner and backend
environment. A missing, malformed, or non-`true` response clears Beta-only
state and restores the released behavior.
## List and history views
The optional `view` query parameter accepts:
| Value | Meaning |
| - | - |
| `useful_now` | Current, fading, and unknown memories suitable for the default view |
| `history` | Dated, superseded, or suppressed history, with the owner record retained |
| `all` | Both useful-now and history records |
If `view` is omitted, the backend preserves the released list semantics for
older callers. Beta first-party clients request `useful_now` for their default
list and provide a discoverable history path. This presentation choice does
not change the existing `short_term`, `long_term`, or `archive` storage tiers;
the ledger-history route is not blanket Archive access.
Unknown rows remain inspectable in both `useful_now` and `history` until their
classification is known; `all` includes both views.
The optional `as_of` query parameter fixes the evaluation clock for the whole
bounded traversal. It is separate from the per-record `as_of` response field:
the query parameter is the traversal clock, the response field is the original
evidence clock, and `belief_computed_at` is the assessment clock.
Historical semantic search supplements current-only indexes with one authorized
newest-first canonical scan of at most 60 raw rows. It keyword-matches and
interleaves provider IDs, then rehydrates at most 60 unique memories. This is a
bounded supplement, so it does not search every retained version or guarantee
complete semantic recall of old records. Use explicit paged history for deeper
inspection.
## Currency and evidence
Currency is a read-side assessment. It does not change a memory's truth,
status, tier, or deletion state.
| Field | Meaning |
| - | - |
| `currency` | A value from `0` to `1`, or `null` when no usable horizon is known |
| `currency_band` | `current`, `fading`, `history`, or `null` when currency is unknown |
| `as_of` | The original evidence timestamp used for the assessment |
| `belief_computed_at` | When the current policy computed the returned assessment |
An unknown row may therefore have both `currency: null` and
`currency_band: null` while still carrying `belief_computed_at`. Clients must
not treat display, retrieval, citations, or the passage of time as new
evidence.
Evidence keeps its original capture lineage and source family. A screen/OCR
capture cannot become independent corroboration simply because it was seen
again. Manual saves preserve owner authority and the record, but a temporary
claim can still become dated when its class or explicit horizon says it
should. A standing instruction or timeless identity claim can remain current.
Evidence authority follows the original source: direct owner input, the
owner's writing, the owner's speech, screen/OCR, then third-party or inferred
material. Unknown or assistant-generated material cannot overpower an owner
fact. Display, retrieval, and citation do not raise authority.
Legacy evidence with missing attribution remains unknown even when transport
metadata says `transcription` or the subject is the owner. Transport metadata
alone does not establish source authority.
When supplied, evidence `captured_at` remains the original capture time. It is
separate from item creation time and does not rewrite creation timestamps or
create a new TTL or status transition. Currency assessment never makes a new
status claim.
## Owner use
The owner can mark a memory as not useful or useful without deleting it. The
shared endpoint is:
```text theme={null}
POST /v3/memories/{memory_id}/use
```
```json theme={null}
{
"action": "suppress",
"feedback_id": "stable-client-retry-id"
}
```
`action` is one of `suppress`, `allow`, or `useful`. `suppress` removes the
memory from default reliance while leaving the owner record inspectable;
`allow` restores default reliance; `useful` records a positive signal and does
not undo an existing suppression. None of these actions changes truth,
currency, evidence, or storage tier.
`feedback_id` is the retry-stable receipt identity. The canonical transaction
writes the owner-use state and the unified feedback receipt atomically. A
replayed request with the same owner, memory, and action is idempotent; using
the same receipt for a different action returns a conflict.
Evidence-event audits use a typed patch with rationale JSON in
`journal.mutation_metadata`. Event details do not go into opaque logical
metadata, so the operation digest remains stable and replay-safe.
## Operational stop
`MEMORY_BELIEF_AUTOMATION_PAUSED=true` pauses automated evidence, admission,
enrichment, and backfill writers while leaving read-side currency and explicit
owner corrections available. This is a deployment-wide incident control, not a
per-user rollout switch. The belief capability header and the automation pause
are separate controls.
## Daily sweep recovery
A failed daily summary does not attest an empty day or advance the sweep
cursor. When the backend proves that preparation failed before the first
provider dispatch, it releases that claim for a later run, at most three
times. The fourth such failure records `pre_dispatch_exhausted` and requires
explicit operator repair. Failures after dispatch or with an unknown origin
remain fenced against automatic retry. Account deletion and generation
boundaries continue to block writes throughout recovery.
Missing gateway accounting is never proof of no dispatch: the ledger is
best-effort and written after provider execution. QA repair without recorded
attempts requires an explicit operator attestation that the exact claim never
dispatched and its worker has terminated, bound to independently reviewed
evidence and an expired lease plus margin. The receipt labels this
`operator_attested_no_dispatch`, not a reservation-absence or no-spend proof.
Without that evidence the claim remains indeterminate, including legacy
claims with no persisted provider identifiers. The operator procedure and
reservation-boundary analysis live in
`backend/docs/runbooks/jit-qa-cloud-run.md`.
# Cloud Audio Storage
Source: https://docs.omi.me/doc/developer/savingaudio
Set up Google Cloud Storage for saving audio files from your Omi device.
## Overview
Store audio files from your Omi device in Google Cloud Storage (GCS). This guide walks you through creating a bucket, setting up credentials, and connecting it to the [Audio Streaming](/doc/developer/AudioStreaming) feature.
You'll need an active Google Cloud Platform account. If you don't have one, [create one here](https://console.cloud.google.com/).
***
## Setup
1. Go to the [GCS console](https://console.cloud.google.com/storage/browser)
2. Click **Create Bucket**
3. Configure the bucket:
| Setting | Recommended Value |
| - | - |
| Name | Something unique, e.g. `omi-audio-files` |
| Location | Multi-region for reliability, or single region for lower cost |
| Storage class | **Standard** (for frequently accessed files) |
| Public access | **Off** (keep audio private) |
| Access control | **Uniform** |
4. Click **Create**
1. Go to **IAM & Admin → Service Accounts** in the GCP console
2. Click **Create Service Account**
3. Set the name (e.g. `omi-audio-uploader`)
4. Grant the **Storage Object User** role
5. Click **Done**
1. Click on the service account you just created
2. Go to the **Keys** tab
3. Click **Add Key → Create new key → JSON**
4. The key file downloads automatically
Keep this key file secure. It grants access to your storage bucket.
Convert the key to Base64 for use as an environment variable:
```bash theme={null}
base64 -i your-key-file.json
```
```powershell theme={null}
[Convert]::ToBase64String([IO.File]::ReadAllBytes("your-key-file.json"))
```
Copy the output — this is your `GOOGLE_APPLICATION_CREDENTIALS_JSON` value.
Go back to the [GCS console](https://console.cloud.google.com/storage/browser) and copy your bucket name from the list. This is your `GCS_BUCKET_NAME` value.
***
## Environment Variables
You'll need these two values when deploying the audio streaming service:
| Variable | Description |
| - | - |
| `GOOGLE_APPLICATION_CREDENTIALS_JSON` | Base64-encoded service account key |
| `GCS_BUCKET_NAME` | Your storage bucket name |
***
## Next Steps
Stream real-time audio from your device to cloud storage
Create apps that use audio data
# React Native SDK
Source: https://docs.omi.me/doc/developer/sdk/ReactNative
Cross-platform mobile SDK for connecting to Omi devices via Bluetooth Low Energy
## Overview
A React Native SDK for connecting to and interacting with Omi devices via Bluetooth Low Energy (BLE). Build cross-platform mobile apps for iOS and Android.
iOS and Android support
Bluetooth Low Energy
Stream and transcribe
***
## Features
* Scan for nearby Omi devices
* Connect to Omi devices
* Get device audio codec information
* Stream audio data from the device
* Monitor battery levels
* Handle connection state changes
* Real-time audio transcription using Deepgram
***
## Installation
### In Your Project
```bash theme={null}
npm install @omiai/omi-react-native
# or
yarn add @omiai/omi-react-native
```
This SDK relies on [react-native-ble-plx](https://github.com/Polidea/react-native-ble-plx) for BLE communication:
```bash theme={null}
npm install react-native-ble-plx
```
For iOS projects, you **must** run `pod install` after installing dependencies:
```bash theme={null}
cd ios && pod install
```
### Platform-Specific Setup
Add to your `Info.plist`:
```xml theme={null}
NSBluetoothAlwaysUsageDescription
This app uses Bluetooth to connect to Omi devices
NSBluetoothPeripheralUsageDescription
This app uses Bluetooth to connect to Omi devices
```
Add to your `AndroidManifest.xml`:
```xml theme={null}
```
***
## Development Setup
```bash theme={null}
git clone https://github.com/BasedHardware/omi.git
cd sdks/react-native
npm install
```
```bash theme={null}
cd example
npm install
```
```bash theme={null}
cd ios
pod install
```
This step is crucial. Without it, the app will fail to build with native module errors.
```bash theme={null}
cd example/ios
open OmiSDKExample.xcworkspace
```
Open the `.xcworkspace` file, NOT the `.xcodeproj` file.
Select your target device and build. Bluetooth doesn't work in simulators - use a physical device.
```bash theme={null}
cd example
npm run android
```
Make sure USB debugging is enabled on your connected Android device.
***
## Quick Start
```javascript theme={null}
import { OmiConnection, DeviceConnectionState, BleAudioCodec } from '@omiai/omi-react-native';
// Create an instance of OmiConnection
const omiConnection = new OmiConnection();
// Scan for devices
const stopScan = omiConnection.scanForDevices((device) => {
console.log('Found device:', device.name, device.id);
}, 10000); // Scan for 10 seconds
// Connect to a device
async function connectToDevice(deviceId) {
const success = await omiConnection.connect(deviceId, (id, state) => {
console.log(`Device ${id} connection state changed to: ${state}`);
});
if (success) {
console.log('Connected successfully!');
// Get the audio codec
const codec = await omiConnection.getAudioCodec();
console.log('Device audio codec:', codec);
// Start listening for audio data
const subscription = await omiConnection.startAudioBytesListener((bytes) => {
console.log('Received audio bytes:', bytes.length);
});
// Get battery level
const batteryLevel = await omiConnection.getBatteryLevel();
console.log('Battery level:', batteryLevel);
// Later, stop listening for audio
await omiConnection.stopAudioBytesListener(subscription);
// Disconnect when done
await omiConnection.disconnect();
}
}
```
***
## API Reference
### OmiConnection Methods
| Method | Description | Returns |
| - | - | - |
| `scanForDevices(onDeviceFound, timeoutMs)` | Scan for nearby Omi devices | Function to stop scanning |
| `connect(deviceId, onConnectionStateChanged)` | Connect to an Omi device | `Promise` |
| `disconnect()` | Disconnect from current device | `Promise` |
| `isConnected()` | Check connection status | `boolean` |
| `getAudioCodec()` | Get device audio codec | `Promise` |
| `startAudioBytesListener(callback)` | Start receiving audio bytes | `Promise` |
| `stopAudioBytesListener(subscription)` | Stop receiving audio bytes | `Promise` |
| `getBatteryLevel()` | Get battery percentage | `Promise` |
### Types
```typescript theme={null}
interface OmiDevice {
id: string;
name: string;
rssi: number;
}
```
```typescript theme={null}
enum DeviceConnectionState {
CONNECTED = 'connected',
DISCONNECTED = 'disconnected'
}
```
```typescript theme={null}
enum BleAudioCodec {
PCM16 = 'pcm16',
PCM8 = 'pcm8',
OPUS = 'opus',
UNKNOWN = 'unknown'
}
```
***
## Stopping Deepgram transcription
`stop()` sends Deepgram's `CloseStream` once and stops accepting audio. By
default, it closes the socket immediately, preserving existing teardown behavior.
To receive final transcripts after stopping, set a positive `drainTimeoutMs`
on `createDeepgramTranscriber` or the shared
`createTranscriber('deepgram', options)` factory. For example, `drainTimeoutMs: 5000`
keeps the socket open until Deepgram closes it or five seconds elapse. `stop()`
returns immediately; transcript callbacks and the socket's `onclose` can occur
afterward, so keep those handlers available during the drain. Omitting the option
or setting it to `0` closes immediately. Failed terminal sends also close immediately.
## Troubleshooting
* Ensure you've run `pod install` in the ios directory
* Try cleaning the build folder: **Product → Clean Build Folder**
* Make sure you're opening the `.xcworkspace` file, not `.xcodeproj`
* Ensure Bluetooth is enabled on your device
* Check that you have the necessary permissions
* Make sure the Omi device is powered on and in range
* Bluetooth scanning doesn't work in iOS simulators - use a physical device
* Try restarting the Omi device
* Ensure the device is not connected to another application
* Check the battery level of the Omi device
* Verify that the device supports the audio service
* Check that you're properly handling the audio bytes in your callback
* Ensure you have a valid Deepgram API key
* Check that the audio listener is started before enabling transcription
* Verify your internet connection is stable
The example app includes padding at the bottom of the ScrollView to ensure input fields remain visible when the keyboard is open.
***
## License
MIT License
***
## Related
Compare all available SDKs
View source code and contribute
# Python SDK
Source: https://docs.omi.me/doc/developer/sdk/python
Connect to Omi devices over Bluetooth, decode Opus audio, and transcribe in real-time using Deepgram
## Overview
A pip-installable Python SDK for connecting to Omi wearable devices over Bluetooth, decoding Opus-encoded audio, and transcribing it in real time using Deepgram.
Deepgram transcription requires `websockets` 14.0 or newer. The SDK's dependency declarations enforce this minimum for its authenticated WebSocket handshake.
Connect to any Omi device
Decode Opus audio to PCM
Deepgram-powered STT
***
## Installation
### Prerequisites
The SDK requires the Opus audio codec library:
```bash theme={null}
brew install opus
```
```bash theme={null}
sudo apt-get install libopus0 libopus-dev
```
```bash theme={null}
sudo yum install opus opus-devel
```
```bash theme={null}
sudo dnf install opus opus-devel
```
### Install the SDK
```bash theme={null}
pip install omi-sdk
```
```bash theme={null}
git clone https://github.com/BasedHardware/omi.git
cd omi/sdks/python
pip install -e .
```
***
## Quick Start
Get a free API key from [Deepgram](https://deepgram.com):
```bash theme={null}
export DEEPGRAM_API_KEY=your_actual_deepgram_key
```
Scan for nearby Bluetooth devices:
```bash theme={null}
omi-scan
```
Look for a device named "Omi" and copy its MAC address:
```
0. Omi [7F52EC55-50C9-D1B9-E8D7-19B83217C97D]
```
```python theme={null}
import asyncio
import os
from omi import listen_to_omi, OmiOpusDecoder, transcribe
from asyncio import Queue
# Configuration
OMI_MAC = "YOUR_OMI_MAC_ADDRESS_HERE" # From omi-scan
OMI_CHAR_UUID = "19B10001-E8F2-537E-4F6C-D104768A1214" # Standard Omi audio UUID
DEEPGRAM_API_KEY = os.getenv("DEEPGRAM_API_KEY")
async def main():
audio_queue = Queue()
decoder = OmiOpusDecoder()
def handle_audio(sender, data):
pcm_data = decoder.decode_packet(data)
if pcm_data:
audio_queue.put_nowait(pcm_data)
def handle_transcript(transcript):
print(f"Transcription: {transcript}")
# Save to file, send to API, etc.
# Start transcription and device connection
await asyncio.gather(
listen_to_omi(OMI_MAC, OMI_CHAR_UUID, handle_audio),
transcribe(audio_queue, DEEPGRAM_API_KEY, on_transcript=handle_transcript)
)
if __name__ == "__main__":
asyncio.run(main())
```
```bash theme={null}
python examples/main.py
```
The example will:
* Connect to your Omi device via Bluetooth
* Decode incoming Opus audio packets to PCM
* Transcribe audio in real-time using Deepgram
* Print transcriptions to the console
***
## API Reference
### Core Functions
| Function | Description |
| - | - |
| `omi.print_devices()` | Scan for Bluetooth devices |
| `omi.listen_to_omi(mac, uuid, handler)` | Connect to Omi device and receive audio |
| `omi.OmiOpusDecoder()` | Decode Opus audio to PCM |
| `omi.transcribe(queue, api_key)` | Real-time transcription via Deepgram |
### Command Line Tools
| Command | Description |
| - | - |
| `omi-scan` | Scan for nearby Bluetooth devices |
***
## Deepgram language selection
The Deepgram engine defaults to `en-US`. Set `language` to a language supported
by your chosen Deepgram model when transcribing other languages:
```python theme={null}
from omi.stt import create_transcriber
transcriber = create_transcriber("deepgram", api_key=DEEPGRAM_API_KEY, language="es")
await transcriber.run(audio_queue, on_transcript=handle_transcript)
```
The `omi.transcribe.transcribe` function also forwards `language="es"` through
its engine options. This selects a transcription language; it does not translate
audio or enable automatic language detection. See Deepgram's
[model and language support](https://developers.deepgram.com/docs/models-languages-overview).
## Development
### Stopping Deepgram transcription
Cancel and await the task running the transcriber to stop it. The SDK stops
sending PCM, sends Deepgram's `CloseStream`, and keeps receiving final transcripts
until the server closes or the receive timeout expires. `drain_timeout` bounds
both the terminal send and the receive wait (default: five seconds per phase).
Callbacks can therefore run while cancellation is being awaited. Set
`drain_timeout=0` on `create_transcriber("deepgram", ...)` to skip waiting for final
results; the terminal send still has a five-second timeout. A server EOF or
connection error follows the reconnect path; it does not send `CloseStream`.
### Local Development Setup
```bash theme={null}
git clone https://github.com/BasedHardware/omi.git
cd omi/sdks/python
# Create virtual environment
python3 -m venv venv
source venv/bin/activate
# Install in editable mode
pip install -e .
# Install dev dependencies
pip install -e ".[dev]"
```
Run the SDK's hardware-free regression suite with `bash test.sh`. It uses the
repository's `uv` environment manager to test the minimum supported `websockets`
version and is also selected by the shared local/CI check manifest.
***
## Troubleshooting
Make sure the Opus audio codec is installed on your system. See the Prerequisites section above for installation commands.
Go to **System Preferences → Privacy & Security → Bluetooth** and grant access to Terminal and Python.
The SDK requires Python 3.10 or higher. Check your version with `python --version`.
Make sure your Omi device is powered on and nearby. Try running `omi-scan` to verify it's discoverable.
The SDK requires `websockets>=14.0`. Try upgrading: `pip install --upgrade "websockets>=14.0"`
***
## License
MIT License — built by the community.
***
## Related
Compare all available SDKs
View source code and contribute
# SDK Overview
Source: https://docs.omi.me/doc/developer/sdk/sdk
Build on top of Omi with official SDKs for Python, Swift, and React Native
## Omi SDKs
The Omi SDKs make it easy to build applications that connect directly to Omi devices. Connect via Bluetooth, stream audio, and get real-time transcription in your preferred language.
***
## Available SDKs
pip-installable SDK with Opus decoding and Deepgram transcription
Native iOS/macOS package with local Whisper transcription
Cross-platform mobile SDK with BLE support
***
## Key Features
Connect to any Omi device over Bluetooth Low Energy
Run transcription locally (Whisper) or via cloud (Deepgram)
No dependency on Omi infrastructure - build completely standalone apps
MIT licensed - use and modify freely
***
## Quick Comparison
| Feature | Python | Swift | React Native |
| - | - | - | - |
| Installation | `pip install omi-sdk` | Swift Package Manager | `npm install @omiai/omi-react-native` |
| Transcription | Deepgram (cloud) | Whisper (local) | Deepgram (cloud) |
| Platforms | macOS, Linux, Windows | iOS, macOS | iOS, Android |
| Audio Codec | Opus | Opus | Opus, PCM |
***
## Getting Started
Pick the SDK that matches your development environment and use case.
Follow the installation guide for your chosen SDK.
Use the SDK's scan function to find nearby Omi devices.
Connect to a device and start receiving audio or transcription data.
***
## Need Help?
Browse source code and report issues
Get help from the community
# Swift SDK
Source: https://docs.omi.me/doc/developer/sdk/swift
Native iOS/macOS SDK for connecting to Omi devices with local Whisper transcription
## Overview
An easy-to-install Swift package for connecting to Omi devices. Get started in seconds with local Whisper-based transcription - no cloud API required.
Native iOS/macOS support
Whisper runs on-device
Connect in minutes
***
## Installation
Open Xcode → File → New Project → iOS → App
Make sure to select **Storyboard** as the Interface option.
1. Navigate to **File → Swift Packages → Add Package Dependency...**
2. Select your project
3. Paste the repository URL: `https://github.com/BasedHardware/omi`
4. Click **Next**
If you aren't prompted to add the package to your target, click "Add Package" again, then "Add to Target" and choose your project.
Go to **Targets → Your Project → Info** and add this permission:
```xml theme={null}
NSBluetoothAlwaysUsageDescription
This app needs Bluetooth access to connect to BLE devices.
```
***
## Quick Start
Get transcription working in 2 minutes:
Replace your `ViewController.swift` with:
```swift theme={null}
import UIKit
import omi_lib
class ViewController: UIViewController {
override func viewDidLoad() {
super.viewDidLoad()
self.lookForDevice()
}
func lookForDevice() {
OmiManager.startScan { device, error in
print("starting scan")
if let device = device {
print("got device ", device)
self.connectToOmiDevice(device: device)
OmiManager.endScan()
}
}
}
func connectToOmiDevice(device: Device) {
OmiManager.connectToDevice(device: device)
self.listenToLiveTranscript(device: device)
self.reconnectIfDisconnects()
}
func reconnectIfDisconnects() {
OmiManager.connectionUpdated { connected in
if connected == false {
self.lookForDevice()
}
}
}
func listenToLiveTranscript(device: Device) {
OmiManager.getLiveTranscription(device: device) { transcription in
print("transcription:", transcription ?? "no transcription")
}
}
}
```
1. Select your development team
2. Connect your iPhone via cable (simulators don't support Bluetooth)
3. Run the project
1. Turn on your Omi device
2. The app should connect automatically
3. Speak - you'll see transcription in the Xcode console
There's no UI in this example - transcription appears in the Xcode logs.
***
## API Reference
The `OmiManager` class provides all device interaction methods:
### Device Scanning
```swift theme={null}
import omi_lib
// Scan for any Omi device
func lookForDevice() {
OmiManager.startScan { device, error in
if let device = device {
print("got device ", device)
self.connectToOmiDevice(device: device)
OmiManager.endScan()
}
}
}
// Scan for a specific device
func lookForSpecificDevice(device_id: String) {
OmiManager.startScan { device, error in
if let device = device, device.id == "some_device_id" {
print("got device ", device)
self.connectToOmiDevice(device: device)
OmiManager.endScan()
}
}
}
```
### Connection Management
```swift theme={null}
func connectToOmiDevice(device: Device) {
OmiManager.connectToDevice(device: device)
self.reconnectIfDisconnects()
}
func reconnectIfDisconnects() {
OmiManager.connectionUpdated { connected in
if connected == false {
self.lookForDevice()
}
}
}
```
### Live Data
```swift theme={null}
func listenToLiveTranscript(device: Device) {
OmiManager.getLiveTranscription(device: device) { transcription in
print("transcription:", transcription ?? "no transcription")
}
}
```
```swift theme={null}
func listenToLiveAudio(device: Device) {
OmiManager.getLiveAudio(device: device) { file_url in
print("file_url: ", file_url?.absoluteString ?? "no url")
}
}
```
***
## OmiManager Methods
| Method | Description |
| - | - |
| `startScan(callback)` | Start scanning for Omi devices |
| `endScan()` | Stop scanning |
| `connectToDevice(device)` | Connect to a discovered device |
| `connectionUpdated(callback)` | Monitor connection state changes |
| `getLiveTranscription(device, callback)` | Receive real-time transcription |
| `getLiveAudio(device, callback)` | Receive audio file URLs |
## Streaming STT (Deepgram)
Transcribe streaming audio with customizable Deepgram model and language options:
```swift theme={null}
import omi_lib
// Via OmiSttFactory
let transcriber = try OmiSttFactory.makeStreaming(
engine: .deepgram,
deepgramAPIKey: "YOUR_DEEPGRAM_API_KEY",
deepgramModel: "nova-2",
deepgramLanguage: "es", // e.g. "en-US", "es", "fr", "ja"
onTranscript: { text in
print("Transcript: \(text)")
}
)
// Or directly with OmiDeepgramTranscriber
let deepgram = OmiDeepgramTranscriber(
apiKey: "YOUR_DEEPGRAM_API_KEY",
sampleRate: 16000,
model: "nova-2",
language: "es",
onTranscript: { text in
print("Transcript: \(text)")
}
)
// Send PCM audio data
deepgram.appendPcm(pcmData)
// Disconnect when done
deepgram.stop()
```
***
## License
MIT License - Omi's Swift SDK is open source.
***
## Related
Compare all available SDKs
View source code and contribute
# Update Omi Firmware
Source: https://docs.omi.me/doc/get_started/Flash_device
Keep your Omi device up to date with the latest firmware. Use the app for easy updates or follow the manual process.
## Overview
There are two ways to update your Omi device firmware:
**Recommended** - The easiest way to update
Go to **Settings → Device Settings → Product Update**
For advanced users or troubleshooting
Follow the instructions below for your device
## Manual Update for DevKit
**Bootloader** (required for first-time setup or if corrupted):
* Download [bootloader0.9.0.uf2](https://github.com/BasedHardware/omi/releases/download/v2.0.1-Omi/bootloader0.9.0.uf2)
* Compatible with Seeed XIAO nRF52840 Sense (SoftDevice S140 7.3.0)
**Firmware** (choose your device):
* [DevKit 1 Firmware](https://github.com/BasedHardware/omi/releases?q=release_firmware+DK1)
* [DevKit 2 Firmware](https://github.com/BasedHardware/omi/releases?q=release_firmware+DK2)
Before starting, ensure:
* Battery is above 50%
* Using a direct USB connection (avoid USB hubs)
* Have a small pin ready for the reset button
1. Connect your device via USB
2. Locate the reset button:
Remove the lid and find the button labeled "RST"
Use the button shown below
3. **Double-click** the reset button quickly
4. Your computer should show a drive named `/Volumes/XIAO-SENSE`
**Install Bootloader** (if needed):
1. Drag `bootloader0.9.0.uf2` to the `/Volumes/XIAO-SENSE` drive
2. Wait for the drive to eject automatically
3. Put device back in DFU mode (repeat step 3)
**Install Firmware**:
1. Drag the firmware `.uf2` file to the `/Volumes/XIAO-SENSE` drive
2. Wait for completion (ignore "transfer incomplete" messages)
3. Device will restart automatically
### Troubleshooting
* Try a different USB cable (some cables are charge-only)
* Try a different USB port
* Avoid USB hubs - connect directly to your computer
* Retry the process from the beginning
* Make sure battery is above 50%
* Try double-clicking the reset button faster
This is normal! Wait 30 seconds, then check if the device restarted successfully.
See the [DevKit2 Firmware Guide](https://help.omi.me/en/articles/9995941-updating-your-devkit2-firmware) for additional help.
## Flashing Consumer Version 1
Wired recovery requires a **SWD connection** to a J-Link programmer. For normal updates, use **Product Update** in the Omi app over Bluetooth; no flashing cable is needed.
### Prerequisites
* Omi CV1 device
* J-Link programmer
* Special SWD flashing cable
* USB cable
* [J-Link Software](https://www.segger.com/downloads/jlink/)
* Latest firmware files from GitHub
### Step 1: Download Latest Firmware
1. Go to [Omi GitHub Releases for CV1](https://github.com/BasedHardware/omi/releases?q=Omi_CV1_v)
2. Find the latest `Omi_CV1_v` release
3. Download both recovery files:
* `Omi_CV1_v.hex` — Application core firmware
* `Omi_CV1_v_CPUNET.hex` — Network core firmware
The `Omi_CV1_OTA_v.zip` asset is for Bluetooth OTA updates. For the J-Link scripts in `omi/firmware/FLASH_3.0.8/`, save the downloaded HEX files as `merged.hex` and `merged_CPUNET.hex` in the appropriate `MAC/` or `WINDOWS/` folder. The bundled images are stale 3.0.8 recovery images.
```bash theme={null}
cd MAC/
# Optional: Backup existing files
mv merged.hex merged.hex.backup
mv merged_CPUNET.hex merged_CPUNET.hex.backup
# Copy new firmware files to this folder
```
```cmd theme={null}
cd WINDOWS\
# Optional: Backup existing files
# Rename merged.hex → merged.hex.backup
# Rename merged_CPUNET.hex → merged_CPUNET.hex.backup
# Copy new firmware files to this folder
```
Ensure both files are updated:
* File sizes should match downloaded versions
* Modification dates should reflect the replacement
### Step 2: Flash the Firmware
1. Install J-Link Software from [SEGGER](https://www.segger.com/downloads/jlink/)
2. Navigate to the MAC folder and flash:
```bash theme={null}
cd MAC
# Flash network core first
JLinkExe -CommanderScript program_net.jlink
# Then flash application core
JLinkExe -CommanderScript program_app.jlink
```
1. Install J-Link Software from [SEGGER](https://www.segger.com/downloads/jlink/)
2. Add J-Link to your PATH (default: `C:\Program Files\SEGGER\JLink\`)
3. Navigate to the WINDOWS folder and flash:
```cmd theme={null}
cd WINDOWS
# Flash network core first
JLink.exe -CommanderScript program_net.jlink
# Then flash application core
JLink.exe -CommanderScript program_app.jlink
```
### Verification
Check that both J-Link commands completed successfully, then confirm the firmware version in the Omi app. Blinking LEDs alone do not verify an update; red blinking can mean the clock has not been synced.
### Troubleshooting
* Verify J-Link is installed correctly
* Check that the installation path is in your system PATH
* Try using the full path to the executable
* Try a different USB port or cable
* Ensure device is in programming mode
* Check that the flashing cable is properly connected
* Always flash the network core first
* Wait for each command to complete before running the next
* Ensure device has sufficient power
* Re-download the firmware files
* Verify file integrity (sizes should match GitHub)
* Check for file corruption during download
***
## Success!
Your Omi device is now updated with the latest firmware. Enjoy the enhanced features and improvements!
## Related Documentation
Step-by-step assembly instructions
Components needed to build Omi
Introduction to Omi
# Chat Tips & Best Practices
Source: https://docs.omi.me/doc/get_started/chat_tips
Learn how to get the best results from Omi chat. Discover prompting strategies, example questions, and tips based on how the system works internally.
## How Your Questions Get Processed
When you ask Omi a question, it goes through an intelligent routing system that determines the best way to answer:
```mermaid theme={null}
flowchart LR
Q[Your Question] --> R{Router}
R -->|Simple| NC[Fast Response]
R -->|Context Needed| A[22+ Tools]
R -->|Persona App| P[Character Response]
```
Understanding these paths helps you frame questions that get the richest, most relevant answers.
**For:** Greetings, general advice, brainstorming
Questions like "Hi!", "What's a good book to read?", or "Tips for productivity" get fast, direct responses without searching your memories.
**Examples:**
* "Hello!"
* "What's a good productivity tip?"
* "Help me brainstorm ideas for..."
**For:** Personal data, temporal queries, integration lookups
Questions like "What did I discuss yesterday?", "Show my calendar", or "What are my goals?" trigger the full agentic system with access to 22+ tools.
**Examples:**
* "What did I discuss yesterday?"
* "Show my calendar for tomorrow"
* "What are my goals?"
**For:** Character-based apps
If you're chatting with a persona app (like "Ask Einstein"), responses come from that character's perspective.
**Examples:**
* Asking Einstein about physics
* Getting advice from a custom coach persona
## Prompting Strategies
Omi excels at temporal queries. The system converts relative time references to precise date ranges.
Use natural time expressions - the system understands them perfectly.
**Great examples:**
* "What did I discuss yesterday?"
* "Summarize my last 3 days"
* "What happened this morning?"
* "Conversations from last week"
* "What did I talk about on Monday?"
**Less effective:**
* Vague timeframes without anchors ("a while ago")
* Very broad ranges ("everything from this year")
Omi uses **semantic search** - it finds conversations by meaning, not just keywords. Ask naturally!
**Great examples:**
* "What have I discussed about my health?"
* "Conversations about career growth"
* "Times I talked about feeling stressed"
* "Discussions involving my manager"
* "What have I said about the project launch?"
**Why it works:** The system embeds your question and finds conversations with similar meaning, even if they use different words.
Omi builds a knowledge base of facts about you across all conversations. Tap into it!
**Great examples:**
* "What do you know about me?"
* "What are my goals?"
* "Who do I talk to most?"
* "What are my preferences?"
* "What hobbies have I mentioned?"
**Why it works:** This triggers the memories system, which stores structured facts extracted from your conversations.
If you've connected services, ask about them directly.
* "What meetings do I have today?"
* "When is my next meeting with Sarah?"
* "Schedule a meeting with John at 3 PM tomorrow"
* "Show my recent emails about the project"
* "Emails from John this week"
* "How was my sleep last night?"
* "What's my recovery score today?"
* "Show my workouts this week"
* "My open pull requests"
* "Issues assigned to me"
## Example Questions by Category
| Category | Great Questions | Why They Work |
| - | - | - |
| **Time-Based** | "What did I do yesterday?", "Summarize my week" | Clear time bounds trigger precise date filtering |
| **Topic Search** | "What have I said about AI?", "My health discussions" | Semantic search finds conceptually related conversations |
| **People** | "Conversations with John", "What did Sarah mention?" | People filter in search + vector similarity |
| **Action Items** | "What tasks do I have?", "What do I need to do?" | Direct access to action item system |
| **Personal Facts** | "What's my favorite food?", "Where do I work?" | Memory retrieval of stored facts |
| **Analysis** | "How productive was I this month?", "Patterns in my week" | Multi-tool orchestration for comprehensive answers |
| **Calendar** | "My meetings tomorrow", "When am I free this week?" | Calendar integration for schedule awareness |
## Pro Tips
These advanced strategies are based on how the system is designed internally.
Narrow results by specifying both when and what:
* "What did I discuss about the project **last week**?"
* "Conversations about health **this month**"
* "What did John say **yesterday**?"
The system remembers your last 10 messages:
* First: "What did I discuss yesterday?"
* Follow-up: "Tell me more about the second one"
* Follow-up: "What action items came from that?"
Skip pleasantries when you need information:
* **Better:** "My tasks for today"
* **Slower:** "Hey, could you maybe show me..."
Relative dates work better than absolute:
* **Better:** "yesterday", "last 3 days", "this morning"
* **Slower:** "January 15th, 2024"
Trigger comprehensive retrieval:
* "Summarize my week"
* "Overview of yesterday's conversations"
* "Key themes from last month"
When you want thorough results:
* "Show me my last 10 conversations"
* "What are my top 5 priorities?"
## What NOT to Ask
Some question patterns don't work well due to system design.
| Avoid | Why | Better Alternative |
| - | - | - |
| "Tell me everything" | Too broad, hits context limits | "Summarize my last week" |
| "Search all my data" | No specific intent to optimize for | "What topics do I discuss most?" |
| Extremely old queries | Vectors may not exist for old conversations | Focus on recent conversations |
| Breaking persona character | Confuses persona apps | Stay in character with persona apps |
| Questions about other users | Privacy boundaries | Focus on your own data |
## Integration-Specific Tips
Best practices for calendar queries:
| Action | Example Query |
| - | - |
| Check schedule | "What meetings do I have today?" or "Am I free at 3 PM?" |
| Find meetings | "When is my next meeting with \[name]?" |
| Create events | "Schedule a meeting with John tomorrow at 2 PM about project review" |
| Modify events | "Move my 3 PM meeting to 4 PM" |
Best practices for email queries:
| Action | Example Query |
| - | - |
| Search by sender | "Emails from John this week" |
| Search by topic | "Recent emails about the contract" |
| Combine filters | "Emails from Sarah about the project in the last 3 days" |
Best practices for health queries:
| Data Type | Example Query |
| - | - |
| Sleep | "How was my sleep last night?" or "Sleep quality this week" |
| Recovery | "What's my recovery score today?" |
| Workouts | "Show my workouts this week" or "How much strain yesterday?" |
| Trends | "How has my sleep been this month?" |
Best practices for development queries:
| Action | Example Query |
| - | - |
| PRs | "My open pull requests" or "PRs I need to review" |
| Issues | "Issues assigned to me" or "Open issues in \[repo]" |
| Create | "Create an issue for the login bug in \[repo]" |
Best practices for Notion queries:
| Action | Example Query |
| - | - |
| Search | "Search my notes about \[topic]" |
| Find pages | "Find my page about product roadmap" |
## Understanding Citations
When Omi answers questions using your conversations, it includes numbered citations like `[1]` and `[2]`. These link to the source conversations.
* **Tap citations** in the app to view the full conversation
* **Multiple citations** mean information came from multiple conversations
* **No citations** means the answer came from general knowledge or your stored memories (facts about you)
## Quick Reference Card
### Best Question Patterns:
```
Time-based: "What did I [verb] [time period]?"
Topic-based: "What have I discussed about [topic]?"
Person-based: "Conversations with [name]" or "What did [name] say?"
Action items: "What tasks do I have?" or "My to-dos"
Personal: "What do you know about my [aspect]?"
Calendar: "My meetings [time period]"
Summary: "Summarize my [time period]"
```
### Power Combinations:
```
Time + Topic: "What did I discuss about [topic] [time period]?"
Time + Person: "Conversations with [name] [time period]"
Topic + Person: "What did [name] say about [topic]?"
All three: "What did [name] and I discuss about [topic] [time period]?"
```
## Related Documentation
Technical deep dive into how the chat system works
Create custom tools for your Omi apps
How your conversations and memories are stored
How live transcription works under the hood
# Introduction
Source: https://docs.omi.me/doc/get_started/introduction
Welcome to Omi - the world's most advanced open-source AI wearable
## Welcome to Omi!
Experience the future of communication with the world's most advanced **open-source AI wearable**. Omi captures, transcribes, and manages your conversations effortlessly, giving you the power to stay focused and productive.
Hardware, firmware, app, and backend - everything is open source and customizable
Your data stays yours - self-host or use our secure cloud infrastructure
24 hours to a few days, depending on device
Real-time speech-to-text capture
Remembers your conversations
Calendar, email, health, and more
***
## Quick Start
New to Omi? Follow these steps to get up and running.
[Buy a pre-built device](https://www.omi.me/) or [build your own](/doc/assembly/Build_the_device)
Download from the [App Store](https://apps.apple.com/us/app/friend-ai-wearable/id6502156163) or [Google Play](https://play.google.com/store/apps/details?id=com.friend.ios)
Pair your device via Bluetooth and customize your settings
Begin capturing and transcribing your conversations!
***
## For Developers
Build on the Omi platform with our comprehensive developer tools.
Set up the Flutter mobile app for development
Configure the Python/FastAPI backend
Create plugins and integrations
Flash and customize device firmware
***
## Build Your Own Hardware
Build or customize your own Omi device from scratch.
What components you need
Step-by-step assembly instructions
Install firmware on your device
***
## Key Features
Hardware designs, firmware, app, and backend - everything is open source
24 hours to a few days, depending on device
Real-time speech-to-text with speaker identification
AI extracts and remembers key facts from your conversations
Get quick summaries and action items automatically
Extend functionality with community-built apps
***
## Get the App
***
## Products & Repositories
Hardware designs and firmware
Flutter app for iOS and Android
Web-based AI persona system
***
## Contribute
We welcome contributions from the community!
* Read our [Contribution Guide](/doc/developer/Contribution)
* Check [current issues](https://github.com/BasedHardware/Omi/issues)
* Earn from contributing with [paid bounties](https://omi.me/bounties)
Connect with other Omi users and developers.
* Join our [Discord](http://discord.omi.me)
* Follow us on [Twitter/X](https://twitter.com/omiHQ)
* Star us on [GitHub](https://github.com/BasedHardware/omi)
Extend Omi's capabilities with your own apps.
* [App Development Guide](/doc/developer/apps/Introduction)
* [Chat Tools](/doc/developer/apps/ChatTools)
* [External Triggers](/doc/developer/apps/Integrations)
# Omi DevKit 1
Source: https://docs.omi.me/doc/hardware/DevKit1
DevKit 1 is a necklace form factor device that can be worn around the neck. It runs on Xiao nRF52840, is powered by a rechargeable battery, and connects to your phone to stream the audio data.
## Overview
DevKit 1 is the original Omi wearable device in a necklace form factor. It connects via Bluetooth to your phone to stream audio data for real-time transcription.
Low-power Bluetooth SoC
Built-in LiPo battery
Wearable pendant design
***
## How to Get It
If you didn't get the original Omi DevKit from the Kickstarter campaign, you can purchase it from the Based Hardware website.
Purchase the latest DevKit from the official store
DevKit 1 is no longer available for purchase. DevKit 2 is the recommended option for new users.
If you prefer to assemble the device yourself, follow our comprehensive build guide.
Step-by-step instructions for assembling your own device
***
## Firmware
The recommended way to update your device is through the Omi app's built-in OTA (Over-The-Air) update feature.
Download from the [Apple App Store](https://apps.apple.com/us/app/friend-ai-wearable/id6502156163) or [Google Play Store](https://play.google.com/store/apps/details?id=com.friend.ios)
Pair your DevKit 1 via Bluetooth
The app will notify you when firmware updates are available
### Manual Firmware Update
Download a pre-built firmware version from the [Omi GitHub Releases](https://github.com/BasedHardware/omi/releases?q=release_firmware+DK1).
Look for releases tagged with `DK1` for DevKit 1 compatibility.
For advanced users who want to customize the firmware:
Learn how to build firmware from source
***
## Related Documentation
Upgraded version with more storage and speaker
Assemble your own Omi device
Parts list and sourcing information
Get started with the Omi app
# Omi DevKit 2
Source: https://docs.omi.me/doc/hardware/DevKit2
DevKit 2 builds on DevKit 1 with 8GB onboard storage, a speaker, and a programmable button. This enables standalone recording mode while maintaining the necklace form factor.
## Overview
DevKit 2 is the upgraded version of the Omi wearable, adding significant new capabilities while keeping the familiar necklace form factor.
Onboard storage for standalone recording
Audio feedback and notifications
Replaces on/off switch for more control
Record without phone connection
See the [announcement on X](https://twitter.com/kodjima33/status/1821324066651041837) for more details about the new features.
***
## What's New in DevKit 2
| Feature | DevKit 1 | DevKit 2 |
| - | - | - |
| Processor | Xiao nRF52840 | Xiao nRF52840 |
| Storage | None | 8GB onboard |
| Speaker | No | Yes |
| Button | On/Off switch | Programmable button |
| Standalone Recording | No | Yes |
| Form Factor | Necklace | Necklace |
While DevKit 2 can record in standalone mode, you still need to connect it to the Omi app for transcription processing.
***
## How to Get It
Purchase an assembled DevKit 2 from the official Based Hardware store
***
## Firmware
The recommended way to update your device is through the Omi app's built-in OTA (Over-The-Air) update feature.
Download from the [Apple App Store](https://apps.apple.com/us/app/friend-ai-wearable/id6502156163) or [Google Play Store](https://play.google.com/store/apps/details?id=com.friend.ios)
Pair your DevKit 2 via Bluetooth
The app will notify you when firmware updates are available
### Manual Firmware Update
Download a pre-built firmware version from the [Omi GitHub Releases](https://github.com/BasedHardware/omi/releases?q=release_firmware+DK2).
Look for releases tagged with `DK2` for DevKit 2 compatibility.
For advanced users who want to customize the firmware:
Learn how to build firmware from source
***
## Related Documentation
Original DevKit version
Consumer version of Omi
Compile custom firmware
Get started with the Omi app
# Omi
Source: https://docs.omi.me/doc/hardware/OmiConsumer
The Omi Consumer Version 1 (CV1) is the latest hardware, built for everyday use. Powered by a dual-core nRF5340 SoC with a Wi-Fi 6 chip (not enabled in current firmware) and dual microphones for high-quality audio capture.
## Overview
The Omi Consumer Version 1 (CV1) is the production-ready Omi hardware designed for everyday use. It features significant upgrades over the DevKit series.
Dual-core Bluetooth LE processor
nRF7002 chip; Wi-Fi is not enabled in current CV1 firmware
T5838 top-port PDM mics for superior audio
Built for everyday consumer use
***
## Technical Specifications
| Component | Specification |
| - | - |
| Processor | nRF5340 dual-core Bluetooth LE SoC |
| Wireless | Bluetooth LE; nRF7002 Wi-Fi 6 chip (not enabled in current firmware) |
| Storage | 4 Gbit / 512 MB soldered SD NAND, raw audio ring |
| Audio | Dual T5838 top-port PDM microphones |
| Form Factor | Consumer-grade wearable |
***
## How to Get It
Purchase the Omi consumer device from the official Based Hardware store
***
## Firmware
The recommended way to update your device is through the Omi app's built-in OTA (Over-The-Air) update feature.
Download from the [Apple App Store](https://apps.apple.com/us/app/friend-ai-wearable/id6502156163) or [Google Play Store](https://play.google.com/store/apps/details?id=com.friend.ios)
Pair your Omi via Bluetooth
The app will notify you when firmware updates are available
### Manual Firmware Update
Download a pre-built firmware version from the [Omi GitHub Releases](https://github.com/BasedHardware/omi/releases?q=Omi_CV1_v).
Look for releases tagged with `Omi_CV1_v` for consumer device compatibility.
For advanced users who want to customize the firmware:
Learn how to build firmware from source
***
## Open Source Hardware
Complete PCB schematics, Gerbers, KiCad 9 source (the `altium/` folder name is historical), mechanical STEP files, BOM, and assembly guides — all MIT licensed
***
## Related Documentation
Developer version with offline storage
Glasses form factor variant
Compile custom firmware
Get started with the Omi app
Guide on painting your device
# Painting OMI device
Source: https://docs.omi.me/doc/hardware/PaintingOMI
A comprehensive guide on how to paint your OMI device for a custom look.
# Painting OMI device
This guide provides step-by-step instructions on how to paint your OMI device.
Follow the full detailed guide on Notion.
## Final Product
Here is a look at the final product with a watch band attached:
***
*Guide and images by [Neo](https://github.com/neooriginal)*
# Adding a Speaker to CV1 & DevKit 1
Source: https://docs.omi.me/doc/hardware/SpeakerMod
An experimental, community-written guide to adding an I2S amplifier and micro-speaker to Omi CV1 and DevKit 1. Not validated on hardware, and not supported by Omi firmware.
# Adding a Speaker to CV1 & DevKit 1
**Experimental and unverified.** This page is a community-written design sketch, not a
supported Omi feature. Nobody has built and validated this mod on CV1 or DevKit 1
hardware, and no Omi firmware in this repository drives an I2S output on those two
boards. Treat the wiring and firmware sections below as a starting point you must
verify yourself, and see [What you would have to build](#what-you-would-have-to-build)
for the specific gaps.
This page covers adding a small I2S amplifier and dynamic micro-speaker to Omi hardware
that does not ship with a usable speaker, so that a phone or host application can play
audio through the device.
## Before you start: which board do you have?
Omi's audio hardware differs per model, and this determines whether the mod is worth
attempting at all.
| Model | SoC | Speaker hardware | Is this mod relevant? |
| :- | :- | :- | :- |
| **CV1** (Omi Consumer Version 1) | Nordic **nRF5340-CLAA**, dual-core Cortex-M33 | No speaker is documented. The published component list includes a **vibration motor** for haptic feedback. | Yes, this is the main target. |
| **DevKit 1** | Seeed **Xiao nRF52840** | Documented as having **no** speaker. | Yes, this is the other target. |
| **DevKit 2** | Seeed **Xiao nRF52840 Sense** | **Ships with a built-in speaker** for audio feedback and notifications (an Adafruit 5769 Audio BFF plus an 8 Ω / 1 W speaker), and its firmware already streams PCM to it over I2S. | **No.** You already have this. See [How DevKit 2 does it](#how-devkit-2-does-it-verified). |
Only CV1 uses the nRF5340. DevKit 1 and DevKit 2 are both Xiao nRF52840 boards, and the
devkit firmware targets the upstream `xiao_ble_sense` board rather than an Omi board file.
These are source-level facts from this repository, not marketing copy: the SoC split is
documented in `docs/doc/hardware/OmiConsumer.mdx` and `docs/doc/hardware/DevKit2.mdx`,
the CV1 main SoC is listed as `nRF5340-CLAA` in
`docs/doc/hardware/consumer/electronics.mdx`, and the DevKit 2 audio hardware is listed in
`omi/hardware/triangle v2 w memory/README.md`.
## Why an amplifier is required
A passive buzzer or transducer is driven at its resonant frequency and cannot reproduce
intelligible speech, so it is useless for voice feedback. A dynamic micro-speaker can, but
an nRF GPIO pin cannot drive a speaker coil: GPIOs are low-current digital signals, and
their drive strength is bounded by the datasheet. The usual solution is an I2S input
Class-D amplifier, which accepts digital audio and delivers the current the coil needs.
The amplifier's audio inputs are high-impedance, so they do not fight the GPIO's limits.
## Suggested parts
These are reasonable, widely available choices. They are **not** a validated combination
for Omi hardware, and no part here has been tested with an Omi board.
| Component | Part Number / Module | Notes |
| :- | :- | :- |
| **I2S Class-D amplifier** | **MAX98357A** (breakout board) | Compact mono amplifier with a direct I2S digital input. |
| **Dynamic micro-speaker** | **1511 speaker box** (15 mm × 11 mm × 3.5 mm) | 8 Ω, roughly 0.8 W–1.0 W, with an integrated acoustic cavity. |
| **Alternative speaker** | **10 mm–12 mm round micro-speaker** | 8 Ω / 16 Ω; easier to fit inside a circular pendant enclosure. |
## Wiring
Because no I2S pin mapping is published for CV1 or DevKit 1, you must choose and verify
the pins yourself. This repository contains no documented table mapping those boards'
pads or test points to I2S signals, and their published pinouts do not cover audio. Do
not assume a pad is exposed: on CV1 the documented pogo pins are charging contacts, and
on DevKit 2 the audio pads are solder joints between the XIAO and the audio board rather
than documented test points.
The connection itself is conventional:
```
+-----------------------+
| Omi board |
| (SoC + your pins) |
+-----------------------+
| | |
I2S_BCLK I2S_LRCLK I2S_DOUT
| | |
v v v
+-----------------------+
supply rail -------->| VIN BCLK DIN |
| |
| MAX98357A |
GND ---------------->| GND LRC GAIN |
+-----------------------+
+ -
| |
v v
+-----------------------+
| micro-speaker |
| (8Ω / 1W) |
+-----------------------+
```
Connections:
1. **VIN** — connect to a supply the amplifier accepts. Use the board's regulated rail,
and confirm the MAX98357A breakout's input range matches it.
2. **GND** — common ground with the board.
3. **BCLK** — I2S bit clock, from a GPIO you have assigned as `I2S_SCK`.
4. **LRC** — I2S word select / left-right clock, from a GPIO assigned as `I2S_LRCK`.
5. **DIN** — I2S data out from the SoC, from a GPIO assigned as `I2S_SDOUT`.
6. **GAIN** — the MAX98357A has several gain settings. Tie it appropriately for your
speaker and volume; the breakout's default is usually a reasonable starting point.
## What you would have to build
This is the part most guides skip. On CV1 and DevKit 1, wiring an amplifier **will not
produce sound on its own**, because the firmware side does not exist in this repository.
Concretely, the gaps are:
* **No I2S output path in firmware.** CV1's board devicetree has no `i2s0` node, and
`omi/firmware/boards/omi/omi-pinctrl.dtsi` defines UART, I2C, PDM, QSPI, SPI and PWM pin
groups but no I2S group. A dormant `speaker.c` exists under the CV1 sources, but it is
not in the CV1 build's source list and the speaker feature is disabled in
`omi/firmware/omi/omi.conf` via `CONFIG_OMI_ENABLE_SPEAKER=n`.
* **DevKit 1 has no audio output either.** Its standard overlay enables nothing and states
that no modules are used.
* **The firmware does not synthesise speech.** Omi's device firmware consumes PCM; it does
not do text-to-speech. TTS exists in the mobile app, which calls the backend's
`/v2/tts/synthesize` endpoint and plays the returned audio through the phone. A host-side
script in the firmware tree does the same for DevKit 2, generating audio with a cloud TTS
provider and streaming it to the device over BLE.
* **You would need an audio source.** Since the device will not generate speech itself,
sound has to come from the phone or a host computer. You would be bridging a new path
(BLE or otherwise) from that source into an `i2s_write` call.
So the work is: pick and verify pins, add an `i2s0` node and a pinctrl group for your board
in a custom overlay, enable the I2S driver and the speaker feature, and write or port a
playback path. All of that is currently out-of-tree.
### Firmware symbols that actually exist
If you write that overlay, these are the real symbols in this repository, not invented
ones:
| Symbol | Status | Notes |
| :- | :- | :- |
| `CONFIG_I2S`, `CONFIG_I2S_NRFX` | Exists (Zephyr) | The I2S driver. Used by the DevKit 2 build. |
| `CONFIG_AUDIO` | Exists (Zephyr) | Audio subsystem. In Omi it is used for **microphone capture**, not playback. |
| `CONFIG_OMI_ENABLE_SPEAKER` | Exists (Omi) | Omi's own speaker switch, defined in the firmware `Kconfig`. This is the option to enable, not `CONFIG_SPEAKER`. |
| `CONFIG_AUDIO_CODEC_NONE` | **Does not exist** | Not present anywhere in the firmware tree. Ignore it if you encounter it in older drafts of this page. |
| `CONFIG_SPEAKER` | **Does not exist** | Omi uses `CONFIG_OMI_ENABLE_SPEAKER`. |
Board targets, for reference:
| Board | Build target | Where it lives |
| :- | :- | :- |
| CV1 | `omi/nrf5340/cpuapp` | `omi/firmware/boards/omi/` (Omi's own nRF5340 board files) |
| DevKit 1 / DevKit 2 | `xiao_ble_sense` | `omi/firmware/devkit/` (upstream Xiao board) |
Nordic's `nrf5340dk_nrf5340_cpuapp` DK target is **not** used by Omi firmware and is not a
build target in this repository. If you are flashing a devkit, you are building for
`xiao_ble_sense`, not for a DK board.
## How DevKit 2 does it (verified)
If your goal is simply "hear audio from the device", DevKit 2 is the supported path and
you do not need this mod. Its firmware contains a working I2S transmit path:
* The DevKit 2 overlay enables `i2s0` and assigns the I2S pins: **BCLK/SCK = P0.29**
(board pad A3), **LRCLK = P0.28** (A2), **SDOUT = P0.03** (A1). The speaker driver also
uses **P0.04** as a speaker-control GPIO.
* The speaker driver obtains `I2S_0`, configures it for transmit, and sends buffered PCM
with `i2s_write`.
* PCM arrives over BLE: a write to the device's audio characteristic is routed to the
speaker path.
* The firmware also plays a boot chime through the same path.
These pin assignments live in
`omi/firmware/devkit/overlay/xiao_ble_sense_devkitv2-adafruit.overlay` and
`omi/firmware/devkit/src/speaker.c`. They are source assignments on that specific board,
**not** a general Omi pinout, and not a set of exposed test pads.
To play arbitrary text through DevKit 2, the firmware tree includes a host-side script that
generates speech with a cloud TTS provider and streams it to the device over BLE:
`omi/firmware/scripts/devkit/play_sound_on_friend.py`. It needs a TTS API key and a BLE
client, and it targets DevKit 2's audio characteristic.
## Acoustic design tips
These are general speaker-in-a-small-enclosure considerations and apply regardless of the
board you are modifying.
1. **Acoustic porting.** Give the enclosure a grille or an opening, roughly 1.5–2 mm,
aligned with the front of the speaker diaphragm. Without a port, the enclosure walls
dominate the low end and the speaker sounds thin.
2. **Gasket sealing.** Use closed-cell foam or a silicone gasket around the speaker
perimeter. Sealing the front and back cavities prevents cancellation and gives a
noticeably cleaner, louder result.
3. **Headroom.** Keep the amplifier's gain low enough that normal speech does not clip.
Clipped speech is more fatiguing than slightly quiet speech.
4. **Keep the mic clear.** Omi captures audio with PDM microphones. Do not route speaker
output into the microphone cavity in a way that causes the device to hear itself.
## If you build this
Omi accepts pull requests, and a validated version of this page would be a genuine
contribution. If you work through the gaps above, please open a PR with the details that
were missing here: which board you used, the exact pin or pad mapping, the devicetree
overlay and Kconfig lines you actually flashed, and whether you got audio out. That would
let this page be rewritten from tested instructions instead of a design sketch.
# Assembly & BOM
Source: https://docs.omi.me/doc/hardware/consumer/assembly
Bill of Materials and assembly reference for building the Omi from components.
## Bill of Materials
The BOM contains 88 components across the mainboard, charger board, FPC, and mechanical assembly.
**Download:**
* [CSV format](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/bom/omi-bom.csv)
* [XLSX format](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/bom/omi-bom.xlsx)
### Key Components
| Component | MPN | Manufacturer | Qty |
| - | - | - | - |
| Main SoC | nRF5340-CLAA | Nordic Semiconductor | 1 |
| Wi-Fi IC | nRF7002-CEAA-R7 | Nordic Semiconductor | 1 |
| Microphones | MMICT5838-00-012 | TDK | 2 |
| 4Gbit (512MB) SD NAND Flash | CSNP4GCR01-DPW | CS | 1 |
| 6-axis IMU | LSM6DS3TR-C | STMicroelectronics | 1 |
| Battery Charger | BQ25101YFPR | Texas Instruments | 1 |
| DC-DC Buck | TPS628438YKAR | Texas Instruments | 1 |
| Load Switch | TPS22916CYFPR | Texas Instruments | 2 |
| RF Switch | FM8625H | FUMAN | 1 |
| LDO 3.3V | SGM2036S-3.3XXDH4G/TR | SGMICRO | 1 |
| LDO 1.8V | SGM2036S-1.8XXDH4G/TR | SGMICRO | 1 |
| Crystal 40MHz | CJ17-400001010B20 | CJ | 1 |
| Crystal 32MHz | 1S32000049 | Faith Long | 1 |
| Crystal 32.768KHz | DST1610A | KDS | 1 |
| Battery | GRP1654M1-1C-1S1P-3.7V-150mAh | GERUIPU | 1 |
| Vibration Motor | LBM0525A4123F | LIDEWEI | 1 |
| RGB LED | MHPA0606RGBDT | MEIHUA | 2 |
| Pogo Pins (device) | CA02-PG07 | JINLANTIAN | 6 |
| Pogo Pins (charger) | CA62-PG308 | JINLANTIAN | 2 |
| Tactile Button | TS-1001S | JINBEILI | 1 |
### Passive Components
The mainboard uses 0201-size passives (resistors, capacitors) primarily from YAGEO and Murata. See the full BOM CSV for complete listings including designators.
**Sourcing tip for indie makers:** Order passives (0201 resistors/capacitors) from LCSC — they have the best prices and ship from China alongside your JLCPCB boards. For active ICs, Mouser/DigiKey have more reliable stock.
***
## Assembly Reference
### Exploded View
The device consists of these main assemblies:
1. **Front aluminium cover** (CNC machined)
2. **LED light guide** (SLA/translucent)
3. **Main frame** (SLA/translucent PC+ABS)
4. **PCB mainboard** with microphones, SoC, flash
5. **FPC** connecting to charging contacts
6. **Battery** (150mAh LiPo)
7. **Vibration motor**
8. **Silicone pad**
9. **Back aluminium cover** (CNC machined)
### Build Guide
Lay out all components on an anti-static mat. You'll need: tweezers, Phillips screwdriver (PH000), UV lamp, multimeter, and S300 adhesive.
Place the mainboard PCB into the SLA frame on the mounting posts. Board should sit flat with no rocking. Connect the FPC cable to the board-to-board connector.
The FPC connector is fragile — lift the latch before inserting, then press down firmly to lock. Do not force it.
Plug in the 150mAh LiPo battery. Route the cable to avoid pinching. Attach the vibration motor with double-sided tape.
**Battery polarity is critical.** Reversed polarity will destroy the BQ25101 charger IC and may cause thermal runaway. Verify polarity with a multimeter before connecting.
Apply UV adhesive to magnet seats. Insert 4x N52 magnets (D3×H3mm). Cure with UV lamp for 30 seconds each.
Test magnet polarity against the charger dock BEFORE gluing. Mark each magnet with a Sharpie. Once cured, they cannot be removed.
Place the silicone pad over the battery/motor assembly.
Attach aluminium covers (Case A front, Case B back). Secure with 3x Phillips screws (CM1.2×5). Apply S300 glue around the seam.
Before sealing: test charging contact continuity, verify BLE advertising, confirm both mics record audio.
Power on the device and run through the verification checklist:
* BLE advertising visible in nRF Connect
* Both microphones recording (test with firmware audio loopback)
* Wi-Fi scanning networks
* IMU reporting orientation changes
* LED cycling RGB colors
* Charger dock making contact and charging
All 6 checks must pass before packaging.
Place device in foam insert, close the packaging case. The foam inserts protect the device during shipping.
Packaging STEP files are in [`packaging/cad/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/packaging/cad) if you want to manufacture custom retail packaging.
### Reference Photos
Assembly photos are in [`assembly/photos/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/assembly/photos):
* [`materials-labelled-exploded-view.jpg`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/assembly/photos/materials-labelled-exploded-view.jpg) — All components identified
* [`materials-labelled.jpg`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/assembly/photos/materials-labelled.jpg) — Materials and finish callouts
* [`components-disassembled.jpg`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/assembly/photos/components-disassembled.jpg) — Individual components laid out
* [`outer-aluminium-covers.jpg`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/assembly/photos/outer-aluminium-covers.jpg) — CNC aluminium cover detail
### Additional Mechanical Parts
| Part | Specification | Qty |
| - | - | - |
| Dust filter | 5.8x4.0x0.1mm | 2 |
| Double-sided tape (small) | D4xd1.5xH0.1mm | 2 |
| Double-sided tape (large) | 14x12.5x0.1mm | 1 |
| Phillips screws | CM1.2x5 | 3 |
| N52 magnets | D3xH3mm | 4 |
| Light pipe | PC, D6xH1.9mm (custom) | 1 |
| S300 glue | 20g/pcs | 1 |
| UV glue | DDY6203\_6 | 1 |
| Necklace rope | 2.5mm, L650mm | 1 |
| Protection cover | 28.7x35.5x11mm (molded) | 1 |
***
## File Integrity
All design files include SHA-256 checksums for verification against the original source files.
```bash theme={null}
cd omi/hardware/consumer
sha256sum -c checksums.sha256
```
SHA-256 checksums for all 58 design files (ZIP, STEP, PDF, BOM, images)
# Electronics
Source: https://docs.omi.me/doc/hardware/consumer/electronics
PCB design files for the Omi mainboard, charger board, and flexible PCB — KiCad source, Gerber files, and schematics.
## Board Overview
The Omi uses three interconnected PCBs:
| Board | Function | Latest Version |
| - | - | - |
| **Mainboard** | nRF5340 SoC, nRF7002 Wi-Fi, microphones, IMU, flash, battery management | v1.2 |
| **Charger Board** | Magnetic dock with pogo pins, 5V input (test points TP1–TP4), ESD protection | v1.0 |
| **FPC** | Flexible cable connecting mainboard to charging contacts | v1.0 |
***
The main PCB carries all active electronics.
**Key ICs:**
| IC | Function |
| - | - |
| nRF5340-CLAA | Dual-core Bluetooth LE SoC (application + network cores) |
| nRF7002-CEAA-R7 | Wi-Fi 6 companion IC |
| CSNP4GCR01-DPW | 4Gbit (512MB) SD NAND Flash |
| LSM6DS3TR-C | 6-axis IMU |
| BQ25101YFPR | Li-Ion/Li-Poly battery charger |
| TPS628438YKAR | DC-DC buck converter |
| 2x MMICT5838-00-012 | TDK T5838 PDM microphones |
**Files:**
| File | Description |
| - | - |
| [`schematic.pdf`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/electrical/mainboard/schematic.pdf) | Full schematic |
| [`gerbers/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/electrical/mainboard/gerbers) | Gerber + drill files for PCB fabrication |
| [`altium/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/electrical/mainboard/altium) | KiCad 9 source project (folder name is historical) |
The nRF5340 (WLCSP-95, 4.4x4.0mm) and nRF7002 (WLCSP-81, 3.75x3.4mm) are fine-pitch wafer-level packages. Requires stencil + solder paste + reflow. Do not hand solder.
Magnetic charging dock. Note: the released charger schematic (v1.0) has no USB-C receptacle — its 5V input arrives on test points TP1/TP3 (+) and TP2/TP4 (GND).
**Key ICs:**
| IC | Function |
| - | - |
| TPS25200DRVR | eFuse power manager (6.5V, 2.8A) |
| TPD1E10B06DPYR | ESD protection diode |
**Files:**
| File | Description |
| - | - |
| [`schematic.pdf`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/electrical/charger-board/schematic.pdf) | Schematic |
| [`gerbers/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/electrical/charger-board/gerbers) | Gerber files |
| [`altium/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/electrical/charger-board/altium) | KiCad 9 source |
The charger board is the simplest of the 3 — a good starting point if you're new to PCB assembly.
Connects the mainboard to the charging contact ring via a board-to-board connector.
**Key Components:**
| Component | Description |
| - | - |
| ST-BTB-K3570606M | BTB male connector (6+4P) |
| Charging contact ring | D9.9mm |
**Files:**
| File | Description |
| - | - |
| [`schematic.pdf`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/electrical/fpc-board/schematic.pdf) | Schematic |
| [`gerbers/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/electrical/fpc-board/gerbers) | Gerber files |
| [`altium/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/electrical/fpc-board/altium) | KiCad 9 source |
FPC is delicate — handle with care during assembly. The board-to-board connector has a limited insertion cycle count.
***
## How to Order PCBs
Upload the Gerber ZIP to any PCB manufacturer. Use the exact specs below per board:
| Parameter | Mainboard | Charger Board | FPC |
| - | - | - | - |
| Layers | 4 (F.Cu, In1.Cu, In2.Cu, B.Cu) | 2 | 2 |
| Material | FR4 | FR4 | Polyimide (flex) |
| Thickness | 0.6mm | 1.0mm | 0.3mm |
| Board Size | 21.0 x 21.0 mm | 13.0 x 20.0 mm | 29.3 x 12.5 mm |
| Surface Finish | ENIG (immersion gold) | ENIG | ENIG |
| Min Trace/Space | 0.076mm / 0.076mm (3 mil) | 0.2mm / 0.127mm | 0.254mm / 0.1mm |
| Solder Mask | Both sides | Both sides | Both sides |
| Copper Weight | 1 oz outer, 1/3 oz inner | 1 oz | 1 oz |
The mainboard has blind vias (L1-L2, L3-L4) and buried vias (L2-L3) — confirm your fab supports these before ordering. Total 414 drill hits including 159 blind + 55 buried vias.
Use **ENIG** surface finish for all boards. HASL causes bridging on the nRF5340 and nRF7002 fine-pitch pads (0.4mm).
See the complete Bill of Materials with all 88 components
***
## First Power-On Checklist
Check for solder bridges on QFN pads with a microscope or loupe. Verify all passives are placed correctly (check orientation of polarized capacitors).
Connect bench supply (set to 3.7V, current limit 100mA). Measure VBAT (3.7V at battery input), VDD\_nRF (1.0V for nRF5340 core), and VDDH (3.3V for nRF5340 I/O).
If current draw exceeds 50mA before firmware flash, disconnect immediately — likely a short.
Connect SWD debugger (J-Link or nRF DK). Flash using nRF Connect SDK. See [firmware guide](/doc/developer/firmware/Compile_firmware).
Open nRF Connect mobile app. The device should appear advertising as "Omi". If not, check 32MHz crystal and antenna matching network.
***
## Troubleshooting
Check BQ25101 output (3.7-4.2V). Verify TPS628438 buck converter has correct input. Look for solder bridges on QFN pads with microscope. Measure resistance between VCC and GND — should be more than 1k ohm before power.
Verify 32MHz crystal is soldered (not just tacked). Check 32.768kHz crystal for RTC. Measure nRF5340 VDDH (should be 3.3V) and VDD (1.0V). Check antenna trace and matching network.
nRF7002 requires its own 40MHz crystal. Check BUCKA/BUCKB rails. Verify SPI connection between nRF5340 and nRF7002. Wi-Fi requires firmware support enabled.
T5838 microphones are top-port — verify sound inlet faces the enclosure acoustic holes. Check PDM clock (1-3.25MHz) and data lines. Both mics must be in the correct PDM channel (L/R).
Check pogo pin contact resistance (less than 0.5 ohm). Verify FPC connection. Measure charger board 5V input at TP1 (5V) / TP2 (GND). Check TPS25200 eFuse output. Verify magnet polarity alignment.
# Open Source Hardware
Source: https://docs.omi.me/doc/hardware/consumer/index
Complete open-source design files for the Omi Consumer — PCB schematics, Gerbers, mechanical STEP files, BOM, and assembly guides.
## Omi Consumer Open Source Hardware
The Omi Consumer is fully open source. All design files needed to study, modify, manufacture, and sell the hardware are available under the MIT license.
**What you'll need to build an Omi:**
* **Tools:** Soldering station with hot air, solder paste + stencils, multimeter, tweezers, UV lamp (for adhesive)
* **Software:** KiCad (free — all shipped PCB sources are KiCad 9 projects despite the "altium" folder names), FreeCAD or Fusion 360 (mechanical), nRF Connect SDK (firmware)
* **Accounts:** PCB fabricator (JLCPCB/PCBWay), component distributor (Mouser/DigiKey/LCSC)
* **Skills:** SMT soldering (QFN packages), basic mechanical assembly, firmware flashing via SWD
* **Budget:** \~\$30-50 per unit in small quantities (PCBs + components + enclosure, excluding tools)
* **Time:** 2-3 weeks (PCB fabrication) + 1-2 hours (assembly per unit)
PCB schematics, KiCad source, Gerber files for mainboard, charger, and FPC
STEP files for enclosure, charger, foam inserts, and packaging
88-component BOM with manufacturer part numbers and assembly photos
MIT license — use, modify, and distribute freely
***
## Quick Specs
| Component | Specification |
| - | - |
| Processor | nRF5340 dual-core Bluetooth LE SoC |
| Wi-Fi | nRF7002 Wi-Fi 6 chip |
| Audio | 2x TDK T5838 top-port PDM microphones |
| Storage | 4Gbit (512MB) SD NAND Flash (CSNP4GCR01) |
| IMU | LSM6DS3TR-C 6-axis |
| Battery | 3.7V 150mAh LiPo |
| Charging | BQ25101 charger IC, magnetic pogo pins |
| Form Factor | 25.5mm diameter wearable pendant |
***
## Repository Structure
| Directory | Contents |
| - | - |
| [`electrical/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/electrical) | PCB design files |
| [`electrical/mainboard/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/electrical/mainboard) | nRF5340 + nRF7002 main board |
| [`electrical/charger-board/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/electrical/charger-board) | Magnetic charger dock |
| [`electrical/fpc-board/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/electrical/fpc-board) | Flexible printed circuit |
| [`bom/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/bom) | Bill of Materials (CSV + XLSX) |
| [`mechanical/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/mechanical) | Enclosure & parts (STEP files) |
| [`mechanical/assemblies/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/mechanical/assemblies) | Full device + charger assembly |
| [`mechanical/parts/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/mechanical/parts) | Individual parts by process |
| [`mechanical/drawings/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/mechanical/drawings) | 2D technical drawings |
| [`assembly/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/assembly) | Assembly photos & instructions |
| [`packaging/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/packaging) | Retail packaging CAD & photos |
***
## Build Journey
Upload Gerber ZIPs to your PCB manufacturer. Order all 3 boards together to save on shipping.
4-layer FR4, 0.6mm, 21x21mm, ENIG, blind/buried vias
2-layer FR4, 1.0mm, 13x20mm, ENIG
2-layer polyimide, 0.3mm, 29x13mm
Order 5-10 boards minimum — MOQ is usually 5 and extras cost very little. You'll want spares for rework practice.
Use the [BOM CSV](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/bom/omi-bom.csv) to order from distributors. The MPN column has exact manufacturer part numbers.
**Long-lead items to order first:** nRF5340-CLAA (SoC), nRF7002-CEAA-R7 (Wi-Fi), CSNP4GCR01-DPW (4Gbit/512MB Flash). These can take 2-4 weeks from Mouser/DigiKey. Check LCSC for faster China-based shipping.
The 150mAh LiPo battery (D16×H6.1mm) is a custom form factor. Search for "GRP1654M1" or order the closest match and verify dimensions.
Parts are organized by manufacturing process in [`mechanical/parts/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/mechanical/parts).
Aluminium covers (Case A + B) and copper touch pins. Send STEP files to a CNC vendor. Minimum order typically 10-50 units.
For prototyping, you can 3D print the covers in SLA resin first to verify fit before committing to CNC.
Frame, LED guide, charger cases, and enhance plate can all be SLA printed. Use white/translucent resin for light guide parts.
If you have an SLA printer (Formlabs, Elegoo), this is the fastest path — print overnight and assemble the next day.
The wrapper/shell is injection molded in PC+ABS for production. For prototyping, SLA print it instead.
Injection molding only makes sense at 500+ units. Below that, SLA or CNC is more economical.
Apply solder paste with stencil, place components, reflow. See the [full assembly guide](/doc/hardware/consumer/assembly).
**Critical:** The nRF5340 (WLCSP-95, 4.4x4.0mm) and nRF7002 (WLCSP-81, 3.75x3.4mm) are fine-pitch wafer-level packages. You MUST use solder paste + stencil + reflow (hot air or oven). Do not attempt hand soldering these ICs.
Follow the [assembly reference](/doc/hardware/consumer/assembly) to combine PCB, battery, enclosure, and magnets into the final device.
Flash firmware via SWD debug port using a J-Link or nRF DK. Follow the [firmware compilation guide](/doc/developer/firmware/Compile_firmware).
**Verification checklist:** BLE advertising visible in nRF Connect app, both microphones recording audio, Wi-Fi scanning networks, IMU reporting orientation, LED cycling RGB, charger detecting dock contact.
# License
Source: https://docs.omi.me/doc/hardware/consumer/license
The Omi hardware design files are released under the MIT license.
## MIT License
All Omi hardware design files — including PCB schematics, Gerber files, KiCad source, mechanical STEP files, BOM, assembly documentation, and packaging designs — are released under the **MIT License**.
This means you are free to:
* **Use** the designs for any purpose (personal, commercial, educational)
* **Modify** and adapt the designs
* **Distribute** copies of the original or modified designs
* **Manufacture** and sell products based on these designs
The only requirement is to include the copyright notice and license text.
***
## Full License Text
```
MIT License
Copyright (c) 2024 Based Hardware Contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```
***
## What's Covered
| Category | Files | License |
| - | - | - |
| PCB Electronics | KiCad source, Gerbers, schematics | MIT |
| Mechanical | STEP assemblies and parts | MIT |
| BOM | Component list (CSV/XLSX) | MIT |
| Assembly Docs | Photos, instructions | MIT |
| Packaging | CAD models, drawings | MIT |
| Firmware | Zephyr RTOS source (`omi/firmware/`) | MIT |
***
## Use Cases
**Fully permitted.** Build as many as you want for personal use.
No registration, no fees, no reporting. Modify freely. You own what you build.
**Fully permitted.** Manufacture and sell Omi-based products.
You can set any price, keep modifications proprietary, and owe no royalties to Based Hardware.
You must include the MIT copyright notice and license text somewhere accessible (documentation, packaging insert, or source files). It does not need to be on the product itself.
**Fully permitted.** Use these designs for Kickstarter, Indiegogo, etc.
You can use the open-source designs as your starting point and raise funds to manufacture. Include the MIT notice in your campaign documentation.
Being transparent about using an open-source base design often builds trust with backers.
**Fully permitted.** Fork, modify, and redistribute under MIT or any other license.
Unlike copyleft licenses, MIT does not require your derivatives to be open source. You choose the license for your modifications.
***
## FAQ
**Yes.** The MIT license explicitly permits commercial use with no royalties, fees, or revenue sharing. Include the copyright notice.
**Yes.** Change anything — add sensors, remove features, redesign the enclosure, swap materials. No restrictions on modifications.
**No.** Unlike copyleft (GPL, CERN-OHL-S), MIT does not require you to share your modifications. Keep them proprietary if you prefer.
**No.** Trademark rights are separate from the MIT license. The design files are open source, but the "Omi" name and branding belong to Based Hardware. Use your own branding.
**Yes.** Academic, educational, and research use is fully permitted. Cite the repository in your publications.
The MIT license covers copyright, not patents. Based Hardware has not filed hardware patents on the Omi Consumer design. However, some components (nRF5340, etc.) may be covered by their manufacturers' patents — this is standard for commercial IC usage.
# Mechanical & Packaging
Source: https://docs.omi.me/doc/hardware/consumer/mechanical
STEP files for the Omi enclosure, charger dock, and retail packaging — organized by manufacturing process.
## Device Mechanical Design
The Omi is a 25.5mm diameter wearable pendant with aluminium covers and a PC+ABS internal frame.
### Full Assembly
* [`omi-device-charger-assembly.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/assemblies/omi-device-charger-assembly.step) — Complete device + charger dock STEP assembly
### 2D Drawings
* [`omi-2d-drawings.pdf`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/drawings/omi-2d-drawings.pdf) — Technical drawings for all parts
* [`manufacturing-specifications.png`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/drawings/manufacturing-specifications.png) — Manufacturing spec overview
***
## Parts by Manufacturing Process
Aluminium covers and copper touch pins.
| File | Description |
| - | - |
| [`case-a.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/cnc/case-a.step) | Front cover (aluminium, D25.5mm) |
| [`case-b.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/cnc/case-b.step) | Back cover (aluminium, D25.5mm) |
| [`touchpin-1-v2.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/cnc/touchpin-1-v2.step) | Touch/charging pin 1 (copper, CNC) |
| [`touchpin-2-v2.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/cnc/touchpin-2-v2.step) | Touch/charging pin 2 (copper, CNC) |
| [`touchpin-1.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/cnc/touchpin-1.step) | Touch pin 1 (conductive copper, earlier version) |
| [`touchpin-2.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/cnc/touchpin-2.step) | Touch pin 2 (conductive copper, earlier version) |
| [`cnc-brass-copper-zinc-polished.zip`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/cnc/cnc-brass-copper-zinc-polished.zip) | Brass/copper variant with polished zinc finish |
**Material:** Aluminium (covers), Copper or Brass + Zinc plating (touch pins)
**Critical:** Covers must be D25.5mm ±0.05mm. Touch pins must make reliable contact — test continuity after assembly. Poor pin contact is the #1 charger failure cause.
For prototyping, 3D print the covers in SLA resin first to verify fit before committing to CNC aluminium (minimum orders typically 10-50 units).
Internal shell/wrapper (OD 28.61mm, H 10.85mm, wall 0.7-1.34mm).
| File | Description |
| - | - |
| [`wrapper.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/injection-molding/wrapper.step) | Main wrapper/shell |
| [`wrapper-drawing.pdf`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/injection-molding/wrapper-drawing.pdf) | Manufacturing drawing |
**Material:** PC+ABS
Internal cushioning pad (D22.4mm, 0.6-0.9mm thick).
| File | Description |
| - | - |
| [`pad.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/silicone/pad.step) | Internal pad |
| [`pad-drawing.pdf`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/silicone/pad-drawing.pdf) | Manufacturing drawing |
**Material:** Silicone, 50A or 80A durometer
Frame, LED guide, charger cases, and internal parts.
| File | Description | Color/Finish |
| - | - | - |
| [`frame-white-translucent.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/sla/frame-white-translucent.step) | Main frame | White translucent (or clear/transparent) |
| [`led-guide-white-translucent.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/sla/led-guide-white-translucent.step) | LED light guide | White translucent |
| [`charger-case-a-ceramic.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/sla/charger-case-a-ceramic.step) | Charger top case | Ceramic-like finish (or white) |
| [`charger-case-b-ceramic.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/sla/charger-case-b-ceramic.step) | Charger bottom case | Ceramic-like finish (or white) |
| [`enhance-plate-black.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/sla/enhance-plate-black.step) | Enhance plate | Black |
| [`pad-ppt-grey.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/sla/pad-ppt-grey.step) | PPT pad | Grey |
| [`touchpin-2-sla-test.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/mechanical/parts/sla/touchpin-2-sla-test.step) | Touch pin (SLA test variant) | Grey |
**Material:** PC+ABS, SLA resin, or MJF nylon depending on finish
All SLA parts can be printed on consumer SLA printers (Formlabs Form 3/4, Elegoo Saturn). Use white/translucent resin for the frame and LED guide — these need to transmit light.
***
## Packaging
Retail packaging 3D models for manufacturing the box, foam inserts, and protective case.
### CAD Files
| File | Description |
| - | - |
| [`packaging-full-assembly.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/packaging/cad/packaging-full-assembly.step) | Complete packaging assembly |
| [`case-packaging.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/packaging/cad/case-packaging.step) | Outer case |
| [`sheet-metal-packaging.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/packaging/cad/sheet-metal-packaging.step) | Sheet metal insert |
| [`foam-assembly.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/packaging/cad/foam/foam-assembly.step) | Foam insert assembly |
| [`foam-top.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/packaging/cad/foam/foam-top.step) | Top foam piece |
| [`foam-bottom.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/packaging/cad/foam/foam-bottom.step) | Bottom foam piece |
| [`foam-circle.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/packaging/cad/foam/foam-circle.step) | Circular foam cutout |
| [`foam-packaging.step`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/packaging/cad/foam/foam-packaging.step) | Foam packaging shell |
| [`foam-drawing.pdf`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/packaging/cad/foam/foam-drawing.pdf) | Foam manufacturing drawing |
| [`package-drawing.pdf`](https://github.com/BasedHardware/omi/blob/main/omi/hardware/consumer/packaging/package-drawing.pdf) | Package technical drawing |
### Reference Photos
Photos of the assembled packaging are in [`packaging/photos/`](https://github.com/BasedHardware/omi/tree/main/omi/hardware/consumer/packaging/photos) for manufacturing reference.
***
## Ordering Guide for Indie Makers
Print the frame, LED guide, charger cases, enhance plate, and pad on your SLA printer. Total print time: \~4-6 hours. Cost: \~\$5 in resin.
No SLA printer? Services like JLCPCB 3D Printing, Craftcloud, or Shapeways accept STEP files directly.
Upload `case-a.step` and `case-b.step` to a CNC service. Specify 6061 aluminium, anodized finish. Typical lead time: 5-10 days, \$10-30/unit at 10+ quantity.
For prototyping, 3D print the silicone pad in flexible resin (Formlabs Flexible 80A). The wrapper can be SLA printed in rigid resin.
True silicone molding and injection molding only make economic sense at 500+ units.
Order from standard suppliers: N52 magnets (D3×H3mm), Phillips screws (CM1.2×5), necklace rope (2.5mm), dust filters, double-sided tape. All available on AliExpress/Amazon.
# Hardware & Assembly
Source: https://docs.omi.me/doc/hardware/omiGlass
Open source smart glasses with 6x the battery life of Meta Ray-Bans. Build your own AI-powered glasses with the XIAO ESP32 S3 Sense.
## Overview
omiGlass is an open-source smart glasses project that gives you AI capabilities with exceptional battery life.
Longer lasting than Meta Ray-Bans
Powerful XIAO microcontroller with camera
Hardware, firmware, and software
Watch the [announcement video](https://x.com/kodjima33/status/1911852469329727811) to see omiGlass in action.
***
## How to Get It
Order a limited pre-built DevKit from Based Hardware
Follow the guide below to build from scratch
***
## Prerequisites
Local AI model hosting
For firmware upload
For hardware components
***
## Hardware Components
| Component | Description | Link |
| - | - | - |
| XIAO ESP32 S3 Sense | Main microcontroller with camera | [Amazon](https://www.amazon.com/dp/B0C69FFVHH/ref=dp_iou_view_item?ie=UTF8\&psc=1) |
| 6x 150mAh batteries | Main power supply | [Option 1](https://a.co/d/i17DjOr) or [Option 2](https://a.co/d/bbFdkic) |
| 1x 250mAh battery | Additional power | [Amazon](https://a.co/d/2xheiFC) |
| Wires | For connections | [Amazon](https://a.co/d/ah98wY0) |
| Hinges | For frame assembly | Standard glasses hinges |
The current design does not include a switch. See the contribution section for how you can help improve this.
***
## Software Setup
```bash theme={null}
git clone https://github.com/BasedHardware/omi.git
cd omi/omiGlass
npm install
```
Or with yarn:
```bash theme={null}
yarn install
```
Copy the template and add your keys:
```bash theme={null}
cp .env.template .env
```
Edit `.env` and add:
* [Groq API key](https://console.groq.com/keys)
* [OpenAI API key](https://platform.openai.com/api-keys)
* Ollama URL (default: `http://localhost:11434/api/chat`)
```bash theme={null}
ollama pull moondream:1.8b-v2-fp16
```
```bash theme={null}
npm start
```
Or with yarn:
```bash theme={null}
yarn start
```
This is an Expo project. Open the localhost link displayed after starting to access the web version.
***
## Hardware Assembly
Print the glasses mount case using the STL files in the `hardware` folder.
Wire the batteries and XIAO board according to the assembly diagram.
Mount all components into the 3D printed case and attach hinges.
***
## Firmware Installation
Open the [firmware folder](https://github.com/BasedHardware/omi/tree/main/omiGlass/firmware) and load the `.ino` file in Arduino IDE.
Alternatively, follow the [firmware readme](https://github.com/BasedHardware/omi/tree/main/omiGlass/firmware/readme.md) to build using `arduino-cli`.
Add the ESP32 board package:
1. Go to **File → Preferences**
2. Add to "Additional Boards Manager URLs":
```
https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json
```
3. Go to **Tools → Board → Boards Manager**
4. Search for `esp32` and install the latest version
1. Select port (likely COM3 or higher) at the top of Arduino IDE
2. Search for `xiao` in the board selector
3. Select **XIAO\_ESP32S3**
Go to **Tools** dropdown and set **PSRAM** to **OPI PSRAM**.
Click the Upload button to flash the firmware to your XIAO ESP32S3 board.
***
## Contributing
Help improve omiGlass!
### Software
| Task | Status |
| - | - |
| Connect glasses with Omi AI app | ✅ Complete |
### Hardware
| Task | Status |
| - | - |
| Redesign legs/sides for bigger heads | ⬜ Open |
| Add switch to design | ✅ Complete |
Get help with setup, contribute to the project, and connect with other builders
***
## License
This project is licensed under the MIT License.
***
## Related Documentation
Production Omi wearable
Necklace form factor DevKit
General assembly instructions
Get started with the Omi app
# Flash Firmware
Source: https://docs.omi.me/doc/hardware/omiglass/flash-firmware
Flash the latest OmiGlass firmware using PlatformIO
This guide walks you through flashing the OmiGlass firmware to your ESP32 S3 XIAO board using PlatformIO.
## Prerequisites
Before you begin, ensure you have the following installed:
PlatformIO is required to build and flash the firmware. Install it using one of these methods:
**Using Homebrew (macOS/Linux):**
```bash theme={null}
brew install platformio
```
**Using pip (All platforms):**
```bash theme={null}
pip install platformio
```
**Verify installation:**
```bash theme={null}
platformio --version
```
PlatformIO will automatically install the required libraries when you build the firmware. The dependencies include:
* **NimBLE-Arduino** (v1.4.1+) - Bluetooth Low Energy stack
* **esp32-camera** (v2.0.0+) - Camera driver for ESP32
* **arduino-libopus** - Audio codec for voice encoding
These are automatically downloaded from the `platformio.ini` configuration.
Connect your ESP32 S3 XIAO board to your computer via USB-C cable.
Use a quality USB-C cable that supports data transfer, not just charging.
***
## Flash the Firmware
Navigate to the firmware directory and flash using PlatformIO:
```bash theme={null}
cd omiGlass/firmware
```
Try the standard build environment first:
```bash theme={null}
platformio run -e seeed_xiao_esp32s3 --target upload
```
**If you experience connection issues**, use the slower upload environment:
```bash theme={null}
platformio run -e seeed_xiao_esp32s3_slow --target upload
```
The `seeed_xiao_esp32s3_slow` environment uses a slower upload speed (57600 baud) which is more reliable for some USB connections.
PlatformIO will:
1. Install dependencies (first time only)
2. Compile the firmware
3. Upload to your device
4. Reset the device automatically
The process takes 1-3 minutes depending on your connection.
***
## Build Environments
The firmware supports two build environments:
| Environment | Upload Speed | Debug Level | Use Case |
| - | - | - | - |
| `seeed_xiao_esp32s3` | 115200 baud | Low (Level 1) | **Recommended** - Standard build for most users |
| `seeed_xiao_esp32s3_slow` | 57600 baud | High (Level 5) | Connection issues or detailed debugging |
If you only want to compile the firmware without uploading:
```bash theme={null}
platformio run -e seeed_xiao_esp32s3
```
The compiled binary will be in `.pio/build/seeed_xiao_esp32s3/firmware.bin`
***
## Troubleshooting
**Problem:** PlatformIO cannot find your ESP32 device.
**Solutions:**
* Check USB cable is properly connected
* Try a different USB port (connect directly, not through a hub)
* On macOS: Check System Settings > Privacy & Security for any blocked devices
* Verify device appears with `ls /dev/tty.*` or `ls /dev/cu.*`
**Problem:** Cannot access the serial port.
**Solutions (macOS/Linux):**
```bash theme={null}
sudo chmod 666 /dev/tty.usbserial-*
```
**Or add your user to the dialout group (Linux):**
```bash theme={null}
sudo usermod -a -G dialout $USER
```
Then log out and back in.
**Problem:** Build fails with compilation errors.
**Solutions:**
* Clean the build and try again:
```bash theme={null}
platformio run --target clean
platformio run -e seeed_xiao_esp32s3 --target upload
```
* Update PlatformIO:
```bash theme={null}
pip install -U platformio
```
* Delete `.pio` folder and rebuild:
```bash theme={null}
rm -rf .pio
platformio run -e seeed_xiao_esp32s3 --target upload
```
**Problem:** Upload process starts but never completes.
**Solutions:**
* Use the slower environment:
```bash theme={null}
platformio run -e seeed_xiao_esp32s3_slow --target upload
```
* Try a different USB cable
* Restart your computer and device
***
## Next Steps
After successfully flashing the firmware:
Learn how to monitor logs and test your device
Complete hardware setup and assembly guide
# Disclaimer
Source: https://docs.omi.me/doc/info/Disclaimer
Important information about using Omi devices and software.
Omi is provided **"as is"**, without warranty of any kind. Use of the device should comply with all local laws and regulations concerning privacy and data protection.
***
## Important Notices
Provided without warranty of any kind, express or implied
Users must comply with local privacy and data protection laws
You assume all risks associated with using Omi
Features and APIs may change as the product evolves
***
## Recording & Privacy
When using Omi to record conversations, you are responsible for:
* Obtaining consent from all parties being recorded where required by law
* Complying with local, state, and federal recording laws
* Respecting others' privacy and expectations
***
## Data & Security
| Aspect | Your Responsibility |
| - | - |
| Data Storage | Understand where your data is stored |
| API Keys | Keep your API keys secure |
| Backups | Maintain your own backups if needed |
| Sharing | Be cautious about sharing sensitive conversations |
***
## Limitation of Liability
Based Hardware and its contributors shall not be liable for any damages arising from the use of Omi, including but not limited to:
* Direct or indirect damages
* Loss of data or profits
* Business interruption
* Any other commercial or personal damages
***
Start using Omi
Help improve Omi
# License
Source: https://docs.omi.me/doc/info/License
Omi is open source software available under the MIT License.
## MIT License
Omi is available under the [MIT License](https://github.com/BasedHardware/omi/blob/main/LICENSE), one of the most permissive open-source licenses available.
Read the complete license text on GitHub
***
## What This Means
Use Omi for any purpose, including commercial projects
Modify the source code to suit your needs
Share and distribute your modifications
Use privately without publishing changes
***
## Requirements
The only requirement is that you include the original copyright notice and license text in any copy of the software or substantial portions of it.
***
## License Text Summary
```
MIT License
Copyright (c) Based Hardware
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND...
```
***
## Questions?
For license inquiries or commercial licensing questions, contact **[team@basedhardware.com](mailto:team@basedhardware.com)**
# Privacy Policy
Source: https://docs.omi.me/doc/info/Privacy
At Omi, your privacy and the security of your data are our top priorities. This Privacy Policy explains how we collect, use, and protect your information.
## Overview
We only collect what's necessary
You decide how your data is used
Advanced protection for your information
***
## 1. Information We Collect
Includes your name, email, and contact details, necessary for account creation and management.
We record and store your conversations to offer tailored AI mentor feedback and enrich your personal memory bank.
Insights into how you interact with our app, like feature usage and time engagement, are crucial for continuous improvement of our services.
***
## 2. Purpose of Information Use
We are dedicated to using your information responsibly:
To offer state-of-the-art conversation recording, AI-driven mentorship, and efficient memory bank services
For updates, support, and promotional content (with your consent)
Continuously researching and analyzing data to improve functionality
***
## 3. Data Storage and Security
Your data is stored on highly secure servers, safeguarded with advanced technology
We employ the latest security protocols to prevent data breaches and protect against unauthorized access
***
## 4. Information Sharing
We collaborate with third-party providers under stringent confidentiality agreements, ensuring they adhere to our privacy standards.
We may disclose your information if required by law, always respecting your privacy rights.
***
## 5. Your Privacy Rights
You are empowered to:
View and modify your personal information at any time
Opt out of non-essential data uses, like marketing communications
Request the removal of your data, within legal and operational requirements
***
## 6. Wearable App Version
We offer an alternative version of our app that integrates with wearable devices. In this version, **no data is collected by Omi**. You bring your own API keys, and all data is stored locally on your device. This ensures complete control over your data and privacy.
***
## 7. Policy Updates
We reserve the right to modify this policy. Changes will be communicated transparently on this page, along with the revision date.
***
## 8. Contact Us
For any inquiries regarding our data practices or this policy, please contact us at **[team@basedhardware.com](mailto:team@basedhardware.com)**
***
## Related
Important usage information
MIT License details
# Support
Source: https://docs.omi.me/doc/info/Support
Get help with Omi through our community channels or contact us directly for commercial inquiries.
## Get Help
Report bugs or request features
Ask questions and get help from the community
***
## Open Source Support
For open-source support, we recommend:
Check [GitHub Issues](https://github.com/BasedHardware/omi/issues) to see if your question has already been answered
Join our [Discord community](http://discord.omi.me) for real-time help from other users and contributors
If you've found a bug or have a feature request, [open a new issue](https://github.com/BasedHardware/omi/issues/new) on GitHub
***
## Commercial Support
For commercial support, license inquiries, partnerships, or any other business questions, email us at **[team@basedhardware.com](mailto:team@basedhardware.com)**
***
## Resources
Browse the full documentation
Learn how to contribute to Omi
Frequently asked questions
Follow us for updates
# Integrate 3P Wearables
Source: https://docs.omi.me/doc/integrations
A guide to integrating any wearable device, like Plaud, Limitless, or your own custom hardware, with the Omi open-source ecosystem.
Omi is the world's most advanced open-source AI wearable platform. Our mission is to build an open ecosystem for seamless communication, and that includes enabling a wide range of hardware.
This guide will walk you through integrating third-party wearable devices—like Plaud AI, Limitless, or your own custom hardware—into the Omi app. By doing so, you can leverage Omi's powerful features, including high-quality transcription, conversation memory, and a growing app marketplace, for any device.
***
## The Integration Workflow
Integrating a new device involves two main phases:
Understanding how the device communicates. This typically means capturing and analyzing its Bluetooth Low Energy (BLE) traffic to decode its protocol for commands and data streaming.
Writing code within the Omi mobile app to manage the connection, communication, and data processing for the new device.
This guide focuses on devices that stream audio data, but the principles apply to other data types as well.
***
## Prerequisites
Before you begin, ensure you have the following:
The third-party device you want to integrate
Highly recommended for superior BLE traffic capturing capabilities
Essential tool for analyzing captured network traffic
A local development setup of the Omi app
### Technical Knowledge Required
| Skill | Level | Description |
| - | - | - |
| BLE Concepts | Basic | Services, Characteristics, UUIDs |
| Dart/Flutter | Intermediate | For app integration |
| Python | Optional | For writing verification scripts |
***
## Part 1: Reverse Engineering the Device Protocol
Your first goal is to become a detective. You need to learn the device's language, which for most wearables is spoken over Bluetooth Low Energy (BLE).
### Step 1.1: Capture BLE Traffic
The most effective way to learn the protocol is to capture the communication between the device and its official app.
Go to **Settings → About phone** and tap **Build number** seven times.
Go to **Settings → System → Developer options**. Enable:
* USB debugging
* Enable Bluetooth HCI snoop log
Turn Bluetooth off and on again for the change to take effect.
Use the official vendor app to connect to your device. Perform key operations like starting and stopping a recording, changing settings, etc.
Record a video of your phone's screen while you perform actions. This will be invaluable when matching timestamps in Wireshark to see which action generated which BLE command.
After capturing, disable the snoop log.
**Non-Rooted Devices:** Generate a bug report from Developer options. The log file `btsnoop_hci.log` will be in the ZIP archive under `FS/data/misc/bluetooth/logs/`.
**Rooted Devices:** Pull directly using ADB:
```bash theme={null}
adb pull /data/misc/bluetooth/logs/btsnoop_hci.log
```
Open the `btsnoop_hci.log` file in [Wireshark](https://www.wireshark.org/).
To find your device's address: `adb shell dumpsys bluetooth_manager`
There are two approaches for iOS: manual exploration or full packet capture.
Apps like [nRF Connect](https://www.nordicsemi.com/Products/Development-tools/nrf-connect-for-mobile) or [LightBlue](https://apps.apple.com/us/app/lightblue-explorer/id557428110) allow you to manually explore your device's services and characteristics. This requires more trial and error but can quickly reveal basic information about UUIDs and simple data formats.
**Requirements:**
* iOS 13+ device and Lightning/USB-C cable
* Mac computer with macOS
* [Apple Developer Program account](https://developer.apple.com/programs/) (free tier works)
On your iOS device, open Safari and navigate to:
```
https://developer.apple.com/bug-reporting/profiles-and-logs/?name=bluetooth
```
Sign in, download the profile, and install it via **Settings → Profile Downloaded**.
1. Install [Xcode](https://developer.apple.com/xcode/) from the Mac App Store
2. Download **Additional Tools for Xcode** from [Apple's downloads page](https://developer.apple.com/download/all/?q=Additional%20Tools)
3. Find **PacketLogger.app** in the Hardware folder and copy to Applications
1. Connect your iOS device to your Mac via cable
2. Open PacketLogger and go to **File → New iOS Trace**
3. A pulse icon appears on your iOS device indicating active trace
4. Use the vendor's app to interact with your device
5. Stop and save the capture when finished
PacketLogger provides excellent built-in analysis, or export to Wireshark via **File → Export** (`.pcap` or `.pklg` format).
**PacketLogger Features:**
* Decodes all Bluetooth SIG-defined protocols
* Rich filtering options and text/regex search
* Ability to comment and flag packets
* Export for Wireshark analysis
### Step 1.2: Analyze Traffic in Wireshark
With your log file open, it's time to find the important packets.
Find your device's address and apply a display filter to isolate its traffic
For audio streaming, look for large numbers of similar-sized packets sent rapidly
Look for GATT Service UUID, Characteristic UUID, and raw data payload
Create a "map" of Service UUIDs, Characteristic UUIDs, and Data Formats
**Key Information to Find:**
* **Service UUIDs:** High-level containers (e.g., "Audio Service", "Device Information Service")
* **Characteristic UUIDs:** Specific data endpoints (e.g., "Audio Stream Data", "Battery Level")
* **Data Format:** Encoding of the payload (e.g., Opus, PCM, µ-law, AAC)
### Step 1.3: Decode the Data Payload
The data payload is a hexadecimal string. Your task is to figure out its structure.
Let's say you capture several 240-byte data packets. You notice the first byte is always `b8`, and this byte reappears every 40 bytes within the same packet.
This is a strong clue! The Opus audio codec uses a Table of Contents (TOC) byte at the start of each frame. The repeating `b8` byte suggests the packet contains six 40-byte Opus frames.
| Codec | Description |
| - | - |
| **Opus** | High-quality, low-latency (most common in modern devices) |
| **PCM** | Uncompressed audio (16-bit typical) |
| **µ-law** | Compressed 8-bit audio (telephony standard) |
| **AAC** | Advanced Audio Coding (Apple devices) |
### Step 1.4: Verify Your Findings
Before integrating, write a small standalone script to confirm your assumptions.
```python theme={null}
# Example verification using Python and Bleak
import asyncio
from bleak import BleakClient
DEVICE_MAC = "XX:XX:XX:XX:XX:XX"
AUDIO_CHAR_UUID = "your-characteristic-uuid"
async def main():
async with BleakClient(DEVICE_MAC) as client:
def callback(sender, data):
# Decode and verify audio data
print(f"Received {len(data)} bytes")
await client.start_notify(AUDIO_CHAR_UUID, callback)
await asyncio.sleep(10) # Record for 10 seconds
asyncio.run(main())
```
If you can decode and play back the audio as a `.wav` file, you've cracked the code!
***
## Part 2: Integrating with the Omi App
Now, let's integrate your device into the Omi app's modular architecture.
### Understanding Omi's Device Architecture
| Component | Location | Purpose |
| - | - | - |
| `DeviceConnection` | `.../connectors/device_connection.dart` | Abstract class defining the standard interface for all devices |
| `DeviceTransport` | `.../transports/device_transport.dart` | Abstract transport interface (`NativeBleTransport` in `.../transports/native_ble_transport.dart` is the low-level BLE handler) |
| `DeviceConnectionFactory` | `.../connectors/device_connection.dart` | Constructs the correct connection object based on DeviceType |
### Implementation Steps
Open `app/lib/backend/schema/bt_device/bt_device.dart` and add your device:
```dart theme={null}
enum DeviceType {
omi,
openglass,
frame,
appleWatch,
plaud,
xyz, // Add your new device type here
}
```
Also update `getTypeOfBluetoothDevice` function and create a helper (e.g., `isXyzDevice`) to identify your device during Bluetooth scans.
Create `app/lib/services/devices/connectors/xyz_connection.dart`:
```dart theme={null}
import 'dart:async';
import 'package:omi/backend/schema/bt_device/bt_device.dart';
import 'package:omi/services/devices/connectors/device_connection.dart';
import 'package:omi/services/devices/models.dart';
// Define your device's specific UUIDs
const String xyzAudioServiceUuid = '0000...';
const String xyzAudioStreamUuid = '0000...';
const String xyzButtonUuid = '0000...';
class XyzConnection extends DeviceConnection {
XyzConnection(super.device, super.transport);
@override
Future performGetAudioCodec() async {
// Return the audio codec your device uses
return BleAudioCodec.opus;
}
@override
Future performGetBleAudioBytesListener({
required void Function(List p1) onAudioBytesReceived,
}) async {
// Subscribe to your device's audio characteristic
final stream = transport.getCharacteristicStream(
xyzAudioServiceUuid,
xyzAudioStreamUuid
);
return stream.listen(onAudioBytesReceived);
}
@override
Future performRetrieveBatteryLevel() async {
// Most devices use standard BLE Battery Service
return super.performRetrieveBatteryLevel();
}
}
```
Open `app/lib/services/devices/connectors/device_connection.dart` and add a new case:
```dart theme={null}
class DeviceConnectionFactory {
static DeviceConnection? create(BtDevice device) {
switch (device.type) {
// ... other cases
case DeviceType.appleWatch:
return AppleWatchDeviceConnection(device, transport);
// Add your new case
case DeviceType.xyz:
return XyzConnection(device, transport);
}
}
}
```
Refer to `app/lib/services/devices/connectors/omi_connection.dart` for a complete example of a complex device implementation.
***
## Part 3: Testing and Contribution
### Testing Your Integration
Test your integration thoroughly within the Omi app:
| Test | Description |
| - | - |
| Discovery & Connection | Can you successfully discover and connect to the device? |
| Live Transcription | Does real-time transcription work as expected? |
| Battery Level | Is the battery level displayed correctly? |
| Stability | Is the connection stable? Does it handle reconnection gracefully? |
### Troubleshooting Common Issues
* Double-check your Service and Characteristic UUIDs
* Ensure the device is not connected to its official app or another phone
* Verify BLE permissions are granted in the app
* Your `BleAudioCodec` in `performGetAudioCodec` is likely incorrect
* Verify the codec and its parameters (sample rate, bit depth)
* Check if there's a header to strip from audio packets
* Confirm you are subscribing to the correct characteristic for notifications
* Check in Wireshark if the device is actually sending data after connection
* Verify the characteristic supports notifications (check properties)
### Contributing Your Work
Omi is built by the community. If you've integrated a new device, we strongly encourage you to contribute it back!
Review our [Contribution Guide](/doc/developer/Contribution) for code standards and PR process.
Submit your integration to the [Omi GitHub repository](https://github.com/BasedHardware/omi).
Discuss your integration with the team and community on [Discord](http://discord.omi.me).
We offer [paid bounties](https://omi.me/bounties) for specific features and integrations. Check them out!
***
## Related Documentation
BLE protocol specification for Omi devices
Set up the Omi Flutter app for development
Build apps using Omi SDKs
How to contribute to Omi
# Get Started with Omi
Source: https://docs.omi.me/getstartedwithomi
Choose your device and get started with Omi
## Choose Your Device
Select your Omi device to get started with the specific setup guide:
The standard Omi wearable AI device
Developer kit for building and testing
Smart glasses with Omi capabilities
***
## Key Features
Hardware, firmware, and software - all open source
Real-time speech-to-text as you speak
AI-powered memory of your conversations
Works without internet connection
Extend functionality with community apps
Identifies who said what
***
## Frequently Asked Questions
You can build your own Omi app in just 2 minutes! Check out the [Apps Introduction Guide](/doc/developer/apps/Introduction) to get started.
Conversations are stored on our secure cloud infrastructure. Your data is protected, and you can delete everything with one click in the Omi app settings.
If you don't see any light or transcription:
1. Your device might need firmware - [install it here](/doc/get_started/Flash_device)
2. Make sure the device is charged
3. Try turning it off and on again
If the issue persists, ask in [Discord](http://discord.omi.me) or contact [help@omi.me](mailto:help@omi.me) for a replacement.
You can update firmware directly in the app: **Settings → Device Settings → Update Firmware**
For manual updates, see the [Firmware Update Guide](/doc/get_started/Flash_device).
***
## Need Help?
Get help from the community and team
Browse FAQs and support articles
Create your own Omi integrations
View issues and bounties
For delivery and shipping questions, email [team@basedhardware.com](mailto:team@basedhardware.com)
# Omi Setup
Source: https://docs.omi.me/onboarding/omi
Get started with your Omi device
## Quick Setup
Make sure your Omi device is charged. When you switch it on, there should be a light indicator.
* Press the button once to turn on the device
* Look for the LED light to confirm it's powered on (blue if connected, red if not yet paired)
Download the official Omi app from your device's app store:
Download for iOS
Download for Android
You can also search for "**Omi AI**" in your app store.
1. Open the Omi app after downloading
2. Follow the in-app instructions to pair your device
3. Wait for the Bluetooth connection to establish
After pairing, test that transcription is working:
* Speak near your Omi device
* You should see transcriptions appear in the app
* If you don't see transcriptions immediately, wait 30+ seconds
***
## Button Controls
**Ask a Question**
Tap once to start recording a voice question. Tap again when you're done speaking (or wait 15 seconds). Your answer will appear in Omi chat.
**Quick Action (Configurable)**
Double tap to trigger a quick action. You can choose what it does in **Settings → Device Settings → Double Tap Action**:
* End & save conversation (default)
* Pause / resume recording
* Star the current conversation
**Power Off**
Press and hold for 3 seconds to turn the device off. Press the button once to turn it back on.
***
## Light Indicators
Understanding your Omi's status through LED colors:
| Color | Status |
| - | - |
| 🔴 **Solid Red** | Device is on but disconnected from your phone |
| 🔴 **Blinking Red** | Device hasn't synced the time yet — connect it to the Omi app |
| 🔵 **Solid Blue** | Device is on and connected to your phone |
| 🟢 **Blinking Green + Red** | Charging while disconnected |
| 🟢 **Blinking Green + Blue** | Charging while connected |
| 🟢 **Solid Green** | Fully charged (battery ≥ 98%) |
A **blinking red** light just means your Omi hasn't received the current time yet. It gets the time from your phone over Bluetooth, so **connect the device to the Omi app** and it will sync automatically and stop blinking. If it's already connected but keeps blinking, **disconnect and reconnect** it in the app.
***
## Important: Keep the App Running
**Do not kill the app!** Once activated, Omi works in the background. You can minimize the app, but if you force-close it (remove the process), transcription will stop and the device will disconnect.
***
## Features
Real-time speech-to-text
AI-powered recall
Works without internet
Extend with apps
Identifies who's speaking
Ask questions hands-free
***
## Troubleshooting
If you don't see any light or transcription:
1. Your device might need firmware - [install it here](/doc/get_started/Flash_device)
2. Make sure the device is charged
3. Try turning it off and on again
If the issue persists, ask in [Discord](http://discord.omi.me) or contact [help@omi.me](mailto:help@omi.me)
A blinking red light means the device hasn't synced the time yet. Omi gets the current time from your phone over Bluetooth.
1. Open the Omi app and connect to your device — the time syncs automatically and the light stops blinking.
2. If the device is already connected but the light keeps blinking, disconnect and reconnect it in the app.
* Make sure Bluetooth is enabled on your phone
* Try restarting both your phone and the Omi device
* Ensure the device is charged and powered on
* Move closer to the device during pairing
* Wait up to 60 seconds for initial transcription
* Check your internet connection
* Make sure you're speaking clearly near the device
* Try moving the device closer to your mouth
* Ensure the device is running the latest firmware
* Check if there are connection issues causing reconnection loops
* Contact support if battery life is significantly reduced
***
## Need Help?
Get help from the community and team
Browse FAQs and support articles
Create your own Omi integrations
Contact [help@omi.me](mailto:help@omi.me)
***
Choose a different device
# Omi DevKit 2 Setup
Source: https://docs.omi.me/onboarding/omi-devkit-2
Get started with your Omi DevKit 2 for development
The Omi DevKit 2 is designed for developers who want to build, test, and customize their Omi experience.
***
## Quick Setup
The DevKit 2 has multiple power options:
* **Switch method**: Use the physical switch if available on your version
* **Button method**: Press and hold the center button for 3 seconds
* **USB-C**: Can be powered directly via USB-C connection
Look for the LED indicator to confirm the device is powered on.
DevKits may not come with firmware pre-installed:
1. Check if your device shows any LED activity when powered
2. If no activity, [flash the firmware here](/doc/get_started/Flash_device)
3. Follow the [DevKit 2 testing guide](/doc/developer/DevKit2Testing)
If you see no LED activity at all, firmware installation is required before the device will function.
Download for iOS
Download for Android
1. Open the Omi app
2. Follow pairing instructions
3. The DevKit should appear as a discoverable device
4. Test transcription by speaking near the device
***
## Button Controls
**Turn On/Off**
Single press to power the device on or off
**Voice Questions**
1. Press and hold until you feel a vibration
2. Keep pressed and ask your question
3. Release when you're done speaking
4. Get your answer in Omi chat or as a notification
***
## Light Indicators
| Color | Status |
| - | - |
| 🔴 **Red** | Device is on but disconnected from your phone |
| 🔵 **Blue** | Device is on and connected to your phone |
| 🟠 **Orange** | Device is charging and disconnected |
| 🟢 **Teal** | Device is charging and connected |
***
## Developer Features
Modify and flash custom firmware
Built-in development testing
Access raw audio data
On-device data access
Hardware pin control
Debug communications
***
## Development Resources
Learn how to build your own Omi apps with our [comprehensive guide](/doc/developer/apps/Introduction). You can create apps in just 2 minutes!
Understand how to integrate with the Omi backend: [Backend Setup Guide](/doc/developer/backend/Backend_Setup)
Access real-time audio data: [Audio Streaming Documentation](/doc/developer/apps/AudioStreaming)
Customize the device firmware: [Firmware Compilation Guide](/doc/developer/firmware/Compile_firmware)
Set up the Flutter development environment: [App Setup Guide](/doc/developer/AppSetup)
***
## Testing Your DevKit
Device should show LED activity when powered
Should appear in Bluetooth discoverable devices
Speak near the device and check for transcription
Should maintain stable connection with the app
***
## Troubleshooting
* Check power connection (USB-C or battery)
* Try different power methods (switch vs button)
* Flash firmware if device is completely unresponsive
* Check if battery is charged
* Ensure device is in DFU mode (double-click reset button)
* Check USB-C cable and connection (some cables are charge-only)
* Follow the [detailed flashing guide](/doc/get_started/Flash_device)
* Try a different USB port
* Review [development setup guide](/doc/developer/AppSetup)
* Check [contribution guidelines](/doc/developer/Contribution)
* Ensure all dependencies are installed
* Ask for help in our [Discord](http://discord.omi.me)
* Keep the app running in the foreground initially
* Check for interference from other Bluetooth devices
* Update to the latest firmware
* Restart both the device and phone
***
## Community & Support
Join development discussions
Contribute and report issues
Full developer documentation
Contact [help@omi.me](mailto:help@omi.me)
***
Choose a different device
# Omi Glass Setup
Source: https://docs.omi.me/onboarding/omi-glass
Get started with your Omi Glass smart glasses
Welcome to the future of smart glasses! Follow these steps to set up your Omi Glass.
***
## Quick Setup
Before first use, ensure your Omi Glass is fully charged:
* Connect the charging cable to your glasses
* Look for the charging indicator light
* Allow 2-3 hours for full charge on first use
* The LED will change color when fully charged
Download for iOS
Download for Android
Proper fit is crucial for optimal performance:
1. **Temple Adjustment**: Adjust the temple arms for comfort
2. **Nose Bridge**: Ensure glasses sit comfortably on your nose
3. **Microphone Position**: The mic should be close to your mouth
4. **Camera Angle**: Align the camera with your natural eye level
1. Put on your Omi Glass
2. Power on by pressing the side button for 3 seconds
3. Open the Omi app
4. Follow the in-app pairing instructions
5. Grant necessary permissions (camera, microphone, Bluetooth)
After pairing, calibrate your glasses:
* **Audio Calibration**: Test microphone pickup by speaking
* **Visual Calibration**: Test camera angle and clarity
* **Gesture Recognition**: Learn the basic gesture controls
* **Voice Activation**: Test "Hey Omi" wake phrase
***
## Glass-Specific Features
Visual information overlay
Capture what you see
Multiple mics for clarity
Navigate with gestures
Hands-free operation
All-day usage
***
## Gesture Controls
| Gesture | Action |
| - | - |
| **Swipe Right** | Next function |
| **Swipe Left** | Previous function |
| **Tap** | Select/activate |
| **Hold** | Access menu |
***
## Usage Tips
* Speak at normal conversation volume
* Avoid covering the microphones with your hands
* Works best in quieter environments
* Multiple mics help with noise cancellation
* Keep lenses clean for best visual quality
* Be mindful of privacy in public spaces
* Camera works best in good lighting
* Respect others' privacy when recording
* Charge overnight for all-day use
* Disable features you're not using to extend battery
* Carry the charging case for quick top-ups
* Battery lasts 6-8 hours with normal use
***
## Important: Keep the App Running
**Do not kill the app!** Once activated, Omi works in the background. You can minimize the app, but if you force-close it, the glasses will disconnect and stop functioning.
As this is a development kit, some features may be experimental and not fully stable. Keep the app updated for the latest improvements.
***
## Privacy & Legal
LED shows when recording is active
Much processing happens on-device
Encrypted transmission and storage
Be aware of local recording laws
**Always get consent** when recording others. Be respectful of privacy in public and private spaces.
***
## Troubleshooting
* Ensure fully charged (try charging for 30+ minutes)
* Try a different charging cable
* Hold power button for 10 seconds for hard reset
* Contact support if no LED activity
* Ensure you have the latest version of the Omi app
* Check Bluetooth is enabled on your phone
* Try restarting both app and glasses
* Reset glasses by holding power for 15 seconds
* Check microphone isn't blocked
* Adjust glasses fit for better mic positioning
* Ensure you're in a relatively quiet environment
* Test with different speaking volumes
* Clean the camera lenses
* Check camera permissions in settings
* Adjust glasses angle for better view
* Restart the app if camera appears frozen
***
## Support & Community
Get help from the community
Contact [help@omi.me](mailto:help@omi.me)
Full documentation
Create Omi Glass apps
***
## What's Next?
Create apps for Omi Glass
Try all smart glasses capabilities
Keep the app updated for new features
Leave reviews in the app stores
***
Choose a different device
# Omi for Mac
Source: https://docs.omi.me/onboarding/omi-macos
Set up and use the Omi macOS desktop app: capture modes, permissions, surfaces, and where your data lives.
This guide covers the Omi macOS desktop app. Facts below are checked against
the app source; each section names the files that own the behavior.
## Install and sign in
Download Omi for Mac from [omi.me](https://www.omi.me/), drag it to
Applications, and sign in with your Omi account. The app lives in the top bar
(menu bar), not the Dock, after setup.
## Capture modes
Audio recording has three modes: Off, Always On, and Only Meetings. The
default is Only Meetings.
* **Off** records nothing.
* **Always On** keeps the microphone recording, and system audio too on macOS
14.4 or later when that permission is granted.
* **Only Meetings** records the microphone and system audio only while a call
is active. Until a call is detected it waits, and it does not keep the
microphone open.
A call means a known call app is actually in a call — Zoom, Microsoft Teams,
FaceTime, Webex, GoTo Meeting, Slack, Discord, WhatsApp, or Telegram — or a
browser window that is a Google Meet or Teams meeting. An idle chat window
does not start recording.
**To enable meeting-only recording after onboarding**, change the mode in
either place:
* Top bar microphone button. The menu is Off, Always On, and Only Meetings.
* **Settings → General → Audio Recording.** The same three choices are the
control on that card. Open Settings from the gear at the trailing edge of
the top bar, or press Command-Comma.
**Audio Settings…** in the microphone menu opens Settings → Transcription
(language and vocabulary). It does not change Only Meetings versus Always On.
**Screen Capture is separate.** Settings → General → Screen Capture turns
screenshot capture on or off. That is not the audio mode.
## Permissions
Open Settings (the top-bar gear, or Command-Comma) and choose **Permissions**
to grant or re-grant access. Required: Microphone, Screen Recording, System
Audio, Notifications, and Accessibility. Optional, one feature each:
Bluetooth, Full Disk Access, and Automation. A missing required grant shows as
**Not Granted**.
If macOS already refused a prompt, it will not ask again. Use that row's
System Settings path, or the reset steps on the row.
* **Microphone:** System Settings → Privacy & Security → Microphone. Required
for voice recording and transcription.
* **Screen Recording:** System Settings → Privacy & Security → Screen
Recording. Required to capture the screen.
* **System Audio (macOS 14.4 and later):** on the System Audio row, click
**Test Access**. If System Settings opens, enable Omi under Screen & System
Audio Recording, come back, and test again. Older macOS does not offer this.
* **Notifications:** System Settings → Notifications → Omi. Set the style to
Banners to see alerts. Omi can request permission but cannot turn it off;
switching notifications off in the app opens System Settings.
* **Accessibility:** System Settings → Privacy & Security → Accessibility.
This lets Omi name the document, page, or file in front, not only the
frontmost app. If the switch looks on but Omi says the grant is broken
(common after an update or re-sign), remove Omi from the list and add it
again.
* **Full Disk Access:** System Settings → Privacy & Security → Full Disk
Access. Optional. Needed to read Messages or Mail headers.
* **Automation:** System Settings → Privacy & Security → Automation. Optional.
Needed to control another app, such as Messages.
You can also ask desktop chat to check or request one named permission. You
still confirm the system prompt or the Settings switch yourself.
## The app's surfaces
The top bar is **Chat, Memories, Tasks, and Apps**, plus a Settings gear.
* **Chat** is where you talk to Omi. A daily recap is not its own pill. Open
it from the recap row in Chat. The page title is Daily recap. When that
summary is sent is Settings → Alerts & Privacy → Daily Summary.
* **Memories** opens captured history. A row on that page switches among
Activity, Conversations, Memories, Rewind, and Brain Map. Conversations are
spoken audio. Memories are saved facts. Activity is the timeline of what was
captured. Brain Map is the graph of those memories.
* **Rewind** is the screen-history player on that same row. Its search field
reads "Search what you've seen and heard…". That search is what was on
screen, not the conversation list.
* **Tasks** is the list of commitments Omi heard.
* **Apps** is connectors, imports, and exports.
Settings sections include General (screen capture and audio recording),
Account & Plan, Transcription, Rewind, Floating Bar, Alerts & Privacy,
Permissions, Shortcuts, AI & Automation, Refer a Friend, and About.
## Where your data lives
Omi stores this Mac's data locally first. For the signed-in user the database
is `~/Library/Application Support/Omi/users//omi.db`, with a
Screenshots folder beside it. Omi Beta uses
`~/Library/Application Support/Omi Beta/` and does not share the stable app's
database.
Conversations, memories, and tasks are cached in that local database and sync
with the signed-in account. Screen history is written on this Mac.
Settings → Alerts & Privacy → Data Controls has **Store Recordings** (whether
audio recordings of conversations are stored) and **Private Cloud Sync**. That
page also says account data is encrypted and stored with Google Cloud
infrastructure. **Export All Data** downloads a JSON copy of the account.
Account and subscription are **Settings → Account & Plan**. That page shows
the current plan and upgrades. Plan names it offers include Operator and
Architect. Sign Out is on Account.
## Source of truth
This page mirrors the product guide shipped inside the Mac app
(`desktop/macos/Desktop/Sources/Resources/ProductKnowledge/`), which the
desktop chat agent reads. If the app UI and this page ever disagree, the
in-app guide for your app version wins — and please open a PR so both stay in
sync.