Today, our two-sandwich team was deep into refining our prototype code which is working amazingly well for this stage of development (more on that in a future blog). We were in discussion with our “robots” (AI models) about the intricacies of implementation and optimization directions when a famous quote from Elon Musk came to mind: “A lot of smart engineers spend too much time optimizing a part that should not be there.” This resonated with us, and it led to an extension of this insight: A lot of engineers spend time optimizing an architecture that should be abandoned.
Our current architecture uses three LLMs working in parallel to support our mission of enhancing social interaction for people with memory challenges. Here’s how it works:
- The Hint Generator: This LLM listens to all the conversations the user is having and provides hints to support memory retention and encourage social interaction, which is our primary goal.
- The Fact Analyzer: This model predicts potential future questions that may arise in the conversation and provides relevant answers from a Retrieval Augmented Generation (RAG) database. It does this to influence the responses generated by the Hint Generator, ensuring they are informative and contextually relevant.
- The Calibration Feedback: The third LLM critiques the Hint Generator, providing scores and feedback to refine and improve the responses over time.
However, we recently questioned whether our current architecture is the most effective. What if the Calibration Feedback LLM is not the right approach? Instead, what if we use a variant of the Hint Generator, running them in parallel and combining their results to form the final response?
This shift in thinking has led us to explore the pros and cons of our architecture, and it highlights an important principle: Question everything. We constantly challenge our assumptions and evaluate whether our architecture is moving us closer to our goals or simply creating unnecessary complexity.
Approaching the Change
We are experimenting with this revised architecture by deploying two Hint Generators with slight variations in training and objectives. We’ll compare their outputs and “AND” them together, essentially creating a consensus-driven response that could yield a more nuanced and effective hint system.
The pros of this approach:
- Diverse Perspectives: Running two similar models with different training allows for varied responses, which may better capture the subtleties of social interactions.
- Redundancy and Validation: Having two models cross-check each other could improve the reliability of the hints provided, ensuring that the information is both accurate and helpful.
The cons:
- Increased Computational Load: Running two LLMs in parallel could strain resources, especially in real-time applications.
- Potential Redundancy: If not tuned carefully, this approach might lead to duplicated efforts, complicating the system instead of simplifying it.
Moving Forward
Our team’s approach is to iterate, experiment, and remain critical of our choices. We believe that the architecture of any system must evolve, and questioning whether we’re building on solid ground is crucial to innovation. As we explore these alternatives, we keep Elon Musk’s wisdom close: it’s not about optimizing what’s in front of you; it’s about ensuring you’re building the right thing to begin with.
Stay tuned for more updates as we dive deeper into this architectural exploration and find the best path forward for our AI-driven social interaction platform.