The Complexity of Choosing the Right LLM: Lessons from Our Experiment at Voice of Reason AI

At Voice of Reason AI, we are constantly exploring how to harness the power of large language models (LLMs) in ways that best serve our mission. Whether we're developing interfaceless social interaction aids for people with memory loss or tackling complex programming challenges, one thing has become clear: choosing the right LLM is far from simple.

Understanding System Prompts and Model Personalities

Before diving into our experiment, it's important to understand a key concept in how LLMs work—the system prompt. A system prompt is a kind of instruction given to the LLM that sets the overall tone and structure for its responses. Think of it as laying down the rules for how the model should behave and what it should prioritize.

However, not all models respond to the same system prompt in the same way. This is because each LLM has its own "personality," so to speak. Some models will adhere closely to the instructions given in the system prompt, while others might deviate slightly, depending on their inherent biases, training data, or underlying architecture.

Our Experiment: Refactoring Python Code with Different LLMs

To illustrate the complexity of LLM model choice, we conducted an experiment using three different models: ChatGPT-4, GitHub Copilot, and Anthropic’s Claude Sonnet.

The task was simple in theory: we provided each model with a 400+ line Python code file and asked it to refactor the code into smaller, more manageable files. The goal was to split the code in a way that made it more modular, readable, and maintainable.

Here’s what we found:

  • ChatGPT-4: While the output was reasonably structured, there were major omissions that required significant manual intervention. Key components of the code were either ignored or not fully refactored, leading to incomplete outputs.
  • GitHub Copilot: The model underlying Copilot (currently based on OpenAI Codex) faced similar issues. Though faster than ChatGPT-4, the result was far from complete, with critical sections of the code missing entirely.
  • Anthropic Claude Sonnet: Claude stood out as the clear winner in this test. It provided an almost perfect breakdown of the code into modular files. The only minor issue was the omission of some import statements, which is understandable given the complexity of the task. Still, the degree of accuracy and completeness made it the best performer.

Why Model Choice Matters: It’s Not Just About Speed

The results of our experiment underscore the importance of carefully selecting the right LLM for the task at hand. It’s not just about how fast a model can generate responses; accuracy and task-specific performance are equally, if not more, important.

This becomes even more critical when considering the work we do at Voice of Reason AI. When developing social interaction aids for people with memory loss, the criteria for evaluating LLMs are vastly different. Here, empathy, accuracy in conversation generation, and subtlety in responses are critical. The models must provide cues and guidance that feel natural, supportive, and aligned with the emotional needs of our users.

As models evolve, we will need to develop new ways of measuring their success—not just in terms of speed or raw computational power, but in their ability to fit the nuances of the task at hand.

The Path Forward: Quantifying and Rationalizing Model Performance

Our experiment is just the beginning. As we continue to explore the vast potential of LLMs in aiding social interaction for individuals with memory challenges, it's clear that we will need to continually quantify and rationalize model performance. Understanding which models work best for specific tasks will be essential for the future of AI-driven assistive technologies.

At Voice of Reason AI, we’re committed to staying at the forefront of this rapidly evolving field, ensuring that the models we choose are the best fit for the people who rely on us. After all, when it comes to memory support and social interaction, speed is far from the only factor—empathy, accuracy, and adaptability are what truly matter.