Memory Post-Processing

​

📊 Process Overview

1. Post-processing request initiated
2. Request handled by routers/postprocessing.py
3. Audio pre-processed and stored
4. FAL.ai WhisperX transcription performed
5. Transcript post-processed
6. Speech profile matching for speaker identification
7. Memory updated and reprocessed
8. Optional emotional analysis

​

🔍 Detailed Steps

​

1. Post-Processing Request

• Omi App sends POST request to /v1/memories/{memory_id}/post-processing
• Request includes:
  • Audio recording for post-processing
  • Flag for emotional analysis

​

2. Request Handling

• postprocess_memory function in routers/postprocessing.py processes the request
• Retrieves existing memory data from Firebase Firestore using database/memories.py

​

3. Pre-Processing and Storage

​

User Permission Check

• Checks if user allows audio storage (database/users.py)
• If permitted, audio uploaded to memories_recordings_bucket in Google Cloud Storage

​

Audio Upload for Processing

• Audio uploaded to postprocessing_audio_bucket in Google Cloud Storage
• Handled by utils/other/storage.py

​

Cleanup

• Background thread started to delete uploaded audio after set time (e.g., 5 minutes)

​

4. FAL.ai WhisperX Transcription

• fal_whisperx function in utils/stt/pre_recorded.py sends audio to FAL.ai
• WhisperX model performs high-quality transcription and speaker diarization
• Returns list of transcribed words with speaker labels

​

5. Transcript Post-Processing

• Cleans transcript data
• Groups words into segments based on speaker and timing
• Converts segments to TranscriptSegment objects

​

6. Speech Profile Matching

• Downloads user’s speech profile and known people profiles
• Uses Speechbrain model to compare speaker embeddings
• Updates segments with is_user and person_id flags

​

7. Memory Update and Reprocessing

• Memory object updated with improved transcript and speaker identification
• Updated data saved to Firebase Firestore
• If FAL.ai transcription successful:
  • process_memory in utils/memories/process_memory.py re-processes memory
  • Re-extracts structured data (title, overview, etc.)
  • Re-generates embeddings
  • Updates memory in vector database

​

8. Emotional Analysis (Optional)

• process_user_emotion function called asynchronously
• Uses Hume API to analyze user’s emotions in the recording
• Can trigger notifications based on detected emotions

​

💻 Key Code Components

# In routers/postprocessing.py
@router.post("/v1/memories/{memory_id}/post-processing", response_model=Memory)
def postprocess_memory(memory_id: str, file: UploadFile, emotional_feedback: bool = False):
    # ... (request handling and pre-processing)
    words = fal_whisperx(audio_url)
    segments = fal_postprocessing(words)
    segments = get_speech_profile_matching_predictions(uid, segments)
    # ... (memory update and reprocessing)
    if emotional_feedback:
        asyncio.create_task(process_user_emotion(uid, file_path))

# In utils/stt/pre_recorded.py
def fal_whisperx(audio_url: str):
    # ... (FAL.ai API call and processing)

def fal_postprocessing(words: List[dict]) -> List[TranscriptSegment]:
    # ... (clean and format transcript data)

# In utils/stt/speech_profile.py
def get_speech_profile_matching_predictions(uid: str, segments: List[TranscriptSegment]):
    # ... (speaker identification logic)