Benchmarks

We measure Living Brain on LoCoMo, a public benchmark for long-term conversational memory. It gives a system a long history of conversations, then asks questions that can only be answered by remembering the right detail from the right session.

We report how many answers were judged correct and how much memory context the answering model read to produce them.

LoCoMo

LoCoMo contains 10 long conversations between two people, each running across many sessions, and 1,540 questions about them. Its question types test different kinds of remembering:

• Single-hop: recall one fact from one session.
• Multi-hop: combine facts from different sessions.
• Temporal: work out when something happened, or in what order.
• Open-domain: combine what was said with general knowledge.

Temporal, single-hop and multi-hop questions all score above 91%, and open-domain scores 70.8%. Single-hop questions still produce the most wrong answers in absolute terms, since they are more than half of the set.

Why the numbers look this way

Pages are compiled when material arrives, so a question is answered from a few maintained pages instead of every passage that mentions the topic. Related facts already sit together on one page, and retrieval doesn’t have to collect them one fragment at a time. That keeps the memory context near 6.8K tokens, although the conversation histories average about 26K tokens.

How we ran it

Some definitions:

• Accuracy is correct answers divided by questions judged. Every question was judged.
• Memory context counts the tokens of retrieved memory given to the answering model. It doesn’t include the instructions, the question or the model’s output.
• Full context is the size of the full history, about 26K tokens per question. It’s a size reference only, with no accuracy score reported for it.
• Retrieval time covers search and page reads. Answer time covers retrieval and answer generation together. Both are medians.

What this benchmark doesn’t cover

LoCoMo is a question-and-answer test over a fixed history. It measures whether the right evidence reaches the answer. It doesn’t measure whether a brain surfaces something at the right moment without being asked, which is the job of the Re-entry Priority Score, or how useful its pages are to a person reading them. Each figure on this page comes from a single run.