By

Mykhailo Marunchak


Here's a detailed description of the system architecture and data flow:

  1. Input Processing:
    • Speech input is converted to text via a Speech-to-Text system
    • Transcripts are queued in an asyncio Queue for processing
  2. Parallel LLM Processing:
    • The system uses three LLMs in parallel:
      • LLM1 (Hint Generator): Generates primary responses/hints
      • LLM2 (Fact Management): Processes facts and biographical information
      • LLM3 (Hint Generator 2): Provides feedback and additional context to LLM1
  3. Memory Management System:
    • Three-tiered memory architecture:
      • Mid-Memory: Temporary storage for recent interactions
      • Biography: Long-term storage for important information
      • Chroma Vector DB: Embeddings-based retrieval system
    • Memory Manager periodically processes batches of memories:
      • Validates and filters memory entries
      • Updates biography with new entries
      • Maintains vector embeddings in ChromaDB
  4. RAG (Retrieval-Augmented Generation):
    • Takes input from LLM2's fact management output
    • Performs similarity search on the vector database
    • Returns relevant context through a dedicated queue
  5. Context Management:
    • Context File: Maintains current conversation context
    • History File: Stores conversation history
    • Both feed back into the LLM processing pipeline
  6. Timing Analysis System:
    • Monitors SPEAK/SILENT patterns in responses
    • Provides feedback on conversation pacing
    • Updates context with timing analysis
  7. Data Flow Control:
    • Uses asyncio for asynchronous processing
    • Implements locks for thread-safe operations
    • Manages concurrent access to shared resources
  8. Error Handling and Recovery:
    • Includes backup systems for memory components
    • Implements version control for vector store updates
    • Provides retry mechanisms for failed operations

Key Features:

  • Asynchronous processing throughout the system
  • Parallel LLM execution for different aspects of response generation
  • Multi-tiered memory system with different retention periods
  • Sophisticated timing analysis for conversation flow
  • RAG integration for context-aware responses
  • Comprehensive error handling and recovery mechanisms

The system is designed to maintain conversation context while progressively building and refining a knowledge base about the conversation history and important facts. The parallel LLM architecture allows for specialized processing of different aspects of the conversation, while the memory management system ensures important information is preserved and readily accessible.