Understanding How LLMs Handle Tokens: The Backbone of Evolving Personal Assistants for Memory Challenges

At Voice of Reason AI, we are dedicated to creating personal assistants that help people with memory challenges arising from early Alzheimer’s symptoms, brain injuries, or learning disabilities. These systems are designed to assist in daily interactions, adapting in real-time to offer support through subtle hints and feedback. One key aspect of our architecture involves using large language models (LLMs) that provide feedback to other LLMs, ensuring that the experience is constantly evolving and improving. However, the backbone of this technology lies in the way LLMs process information—through tokens.

What Are Tokens?

In the world of LLMs, a token is a unit of text used by the model to process language. Tokens can represent individual words, subwords, or even characters depending on the language model and the input text. For example, in a sentence like "The weather is great," the model may split it into tokens such as ['The', 'weather', 'is', 'great'], but in a more complex sentence, words might be broken down into subword tokens.

LLMs don't process language as we do—rather than understanding full sentences, they break everything down into manageable pieces (tokens) and look at these chunks in relation to each other. Each token represents a smaller part of the whole, and the model uses statistical probabilities to predict the next token in a sequence.

Why Tokens Matter in Memory Support Systems

The use of tokens introduces both power and limitation in LLM-driven systems. While LLMs can generate fluid text by analyzing sequences of tokens, certain types of information, particularly numbers and time intervals, can pose a challenge. Because tokens are processed based on context and probability, numbers are often treated as mere text rather than mathematical entities. This explains the occasional oddities, like why a model might perceive "9.11" as closer to "9.5" than you’d expect, based on token proximity rather than numerical accuracy.

In the context of our memory-assistance products, precise timing, intervals, and feedback need to be handled carefully. We’ve had to develop strategies to overcome the limitations imposed by tokenization in order to provide more accurate and useful feedback to users.

Qualifying Feedback for Better LLM Consumption

Given these challenges, we’ve designed an approach to qualify feedback in a way that LLMs can respond to more effectively. Since LLMs don’t excel at directly quantifying numbers due to tokenization, we structure feedback to emphasize qualitative insights rather than precise numerical data, allowing the model to guide itself and others more fluidly in our parallel architecture.

For example, consider the output of our conversation timing analysis:

diffCopy code=== Final Enhanced Timing Summary ===
Conversation Timing Analysis:
- Total markers: 50 (1 SPEAK, 49 SILENT)
- Average time between all markers: 4.5 seconds
- Average SPEAK to SPEAK time: N/A seconds
- Average SPEAK to SPEAK quality: too slow
- Average SILENT to SPEAK time: 2.0 seconds
- Average SILENT to SPEAK quality: too rapid
- Average SPEAK to SILENT time: 2.0 seconds
- Average SPEAK to SILENT quality: too rapid

While the above output includes both numerical and qualitative data, notice how the LLM is instructed to focus on the quality of the interactions—whether they’re too rapid, too slow, or just right. This kind of qualified feedback allows the model to better understand the nuances of human conversation and make more informed adjustments to its responses.

We also integrate real-time feedback loops between different LLMs in our system. For instance:

yamlCopy code...Hint Response 1 Output: SILENT: No memory issues detected. Monitoring conversation.
...Hint Response 2 Output: FEEDBACK:
1. Adherence to Guidelines: Strong
2. Overall quality of the output to John: The output was relevant and meaningful, providing a clear hint about preferences.
Fact Management Output: Insufficient context to generate relevant questions.

Here, each LLM in the chain provides both feedback and a summary of the quality of its own output, ensuring that the next LLM in the process has an evolving and enriched context from which to work.

The Feedback Architecture: A Parallel Approach

Our architecture utilizes multiple LLMs working in tandem, with one LLM responsible for real-time conversation analysis, another for fact management, and a third providing speculative questions based on ongoing interactions. This parallel processing allows for faster adaptation, as each model focuses on a specific aspect of the conversation while benefiting from the feedback loop created by the others. We’ve found that LLMs perform best when feedback is presented in terms of guidelines adherence and qualitative performance rather than pure numeric or token-specific calculations.

By qualifying feedback—emphasizing what is working and what is not—we enable the models to adapt more effectively and improve the user's experience with each conversation.

Final Thoughts

LLMs are remarkable in their ability to process language, but their handling of tokens introduces complexities, particularly in relation to numerical data and timing. At Voice of Reason AI, we’ve overcome these challenges by focusing on qualitative feedback and building an architecture that supports real-time improvements based on this feedback. This system, designed to assist people with memory challenges, offers a glimpse into the future of AI-driven personal assistants—ones that evolve in response to the unique needs of each user, ensuring they receive the support they need in the moments that matter most.

By integrating this level of sophistication into our architecture, we continue to push the boundaries of what AI can do, ensuring that our users receive the best possible care through constant innovation.