AI Assistant Memory Tests Favor Retrieval Over Full History

A new benchmark called DolphinBench, introduced by Mem0, evaluates whether AI assistant memory improves task execution rather than simple fact recall. It tests simulated people with 500,000-token chat histories and 200 task scenarios, requiring systems to use stored facts while acting, such as scheduling meetings or ordering items, and to report cost and time alongside accuracy. A Mem0 announcement said its own setup scored 70.67 percent on the benchmark. A separate independent experiment examined four memory designs: no memory, full chat history sent with every request, a running summary, and retrieval that selects relevant history pieces. The test used GPT-4o mini and Claude Haiku 4.5 on a 40,000-token synthetic history for a fictional project manager, Dana Whitfield, containing 30 facts, some later changed, plus 30 task-based tool calls and 30 direct questions. Runs were logged in Weights & Biases. The results showed that remembering a fact is not the same as using it to complete work. GPT-4o mini with the full history answered 96.7 percent of direct questions but completed only 58.3 percent of tasks, the largest gap. It sometimes used outdated information, including ordering lunch from a superseded restaurant, because full history kept old and new messages together. Claude Haiku 4.5 scored closer between questions and tasks, 93.3 percent versus 81.7 percent with full history, but often asked questions instead of taking action when uncertain. No memory passed no tasks. Retrieval offered the strongest value: it completed 88.9 percent of GPT-4o mini tasks and 85.6 percent of Haiku tasks. When retrieval fetched all needed facts, mini passed 95 percent of tasks and Haiku 92 percent. Full history was more expensive, costing 20 to 31 times as much per 30 tasks as retrieval on these models, while adding little latency. Running summary performance was weaker and could drop updated facts; GPT-4o mini retained the newest value for 20 of 30 facts during summarization, versus 28 for Haiku. The experiment supports the premise behind DolphinBench: better accuracy requires memory systems that help assistants act under realistic constraints, not simply reproduce stored facts. It also suggests a practical lesson for builders: retrieval-based memory may deliver more reliable, cost-effective task performance than stuffing entire histories into prompts, especially when corrections and outdated information are present. The author cautions that the test was small and synthetic, but recommends teams evaluate assistants with their own histories and task requests, measuring completed actions, cost, time, and failure patterns around updated facts.