Benchmark Overview
We evaluate Trix against published academic benchmarks to measure how well it retrieves and surfaces relevant memories from long conversations. These results reflect the current state of the system and will evolve as we continue to improve.
LoCoMo Benchmark
The LoCoMo (Long Conversation Memory) benchmark evaluates a system's ability to answer questions about conversations spanning 600+ turns across 497 questions. It tests whether a memory system can retain and recall specific details from extended interactions — the exact problem Trix is designed to solve.
LoCoMo includes five question categories, each testing a different aspect of memory retrieval:
- Single-hop — Direct factual recall from a single conversation turn
- Multi-hop — Questions requiring information from multiple turns
- Temporal — Questions about when things happened or time-based ordering
- Open-domain — Broad questions requiring general understanding
- Adversarial — Questions designed to trick the system with entity swaps
LoCoMo Results
Trix achieves 82.4% F1 on the full LoCoMo benchmark (497 questions), significantly outperforming all published baselines including GPT-4 with the entire conversation in the prompt.
Trix
82.4% F1
Memory retrieval with semantic search and reranking
GPT-4 Full Context
51.2% F1
Entire conversation in the prompt
Memobase
51.5% F1
Previous published SOTA
GPT-4 + Contriever RAG
46.7% F1
Standard RAG retrieval pipeline
The comparison baselines are from the original LoCoMo paper and published research. Trix's score was measured using the same evaluation methodology on the full 497-question dataset.
LoCoMo Scores by Category
Trix outperforms GPT-4 Full Context across all five categories. Temporal reasoning has reached the estimated human ceiling of ~87.9%.
| Category | Trix | GPT-4 Full Context |
|---|---|---|
| Temporal | 87.9% | 47% |
| Multi-hop | 77.5% | 39% |
| Adversarial | 74.4% | 78% |
| Open-domain | 73.6% | 42% |
| Single-hop | 70.9% | 58% |
The adversarial category is now competitive with GPT-4 Full Context, which was previously a significant gap. Continued improvements to entity-swap detection and retrieval precision are narrowing this further.
LongMemEval Benchmark
LongMemEval tests five core memory abilities across 500 curated questions embedded in scalable chat histories (~115K tokens per question). It evaluates whether a system can extract facts, reason across sessions, handle temporal queries, track knowledge updates, and correctly abstain when information doesn't exist.
Trix achieves ~91.9% task-averaged F1 on LongMemEval, competitive with the top-performing memory systems in the benchmark.
Trix
~91.9% F1
Semantic retrieval with agentic multi-call pipeline
Top Competitors
91–95% F1
Purpose-built memory systems (Hindsight, HydraDB, Ensue, Mastra)
GPT-4o Full Context
60.6% F1
All sessions in the prompt
Competitor scores are LLM-Judge on the full 500-question set. Trix's score is task-averaged F1 on an 80-question stratified sample (20 per ability). We are working toward a full 500-question validated run.
LongMemEval Scores by Ability
| Ability | Score |
|---|---|
| Information Extraction | 98.8% |
| Knowledge Updates | 94.8% |
| Temporal Reasoning | 91.9% |
| Multi-Session Reasoning | 74.0% |
Retrieval Latency
Trix is optimized for real-time AI applications. Memory retrieval typically completes in under 100 milliseconds, making it fast enough to use in conversational flows without noticeable delay.
- Semantic search — Sub-100ms for typical workloads
- End-to-end recall — Query to ranked results in under 100ms for most use cases
Latency depends on factors like query complexity, space size, and network conditions. The sub-100ms figure reflects typical performance, not a guaranteed SLA.
What These Benchmarks Mean
These benchmarks test a specific and important capability: can a system answer questions about details from long conversations spanning hundreds of turns and multiple sessions? This directly maps to the core use case for Trix — giving AI applications persistent memory.
Outperforming GPT-4 Full Context (where the entire conversation is stuffed into the prompt) by 30+ percentage points on LoCoMo demonstrates that targeted memory retrieval is significantly more effective than brute-force context inclusion. This means:
- Lower costs — Retrieve only relevant memories instead of including entire histories
- Better at scale — Works with conversations of any length, not limited by context windows
- Better accuracy — Focused retrieval surfaces the right information more effectively
Scoring competitively on LongMemEval alongside purpose-built memory systems validates that Trix's architecture works well across different memory challenges, from fact extraction to temporal reasoning to tracking knowledge updates.
These benchmarks are a snapshot. We continue to improve retrieval accuracy, and scores will be updated as the system evolves.
Learn More
- How Trix Works — Understand the architecture behind these results
- Quick Start — Try Trix yourself in 5 minutes
- Pricing — See plans and estimate costs