When reading long novels like Dream of the Red Chamber, Harry Potter, or One Hundred Years of Solitude, readers often struggle to remember earlier plot points or character relationships. Similarly, AI systems face a similar challenge when processing lengthy texts—when the content becomes too long, they either slow down dramatically or fail to retain earlier information. To address this memory bottleneck, a team led by Yale University PhD student Dong Liu and collaborators has developed a novel framework called HSGM (Hierarchical Segment-Graph Memory), which enables AI to understand extremely long documents with unprecedented speed and accuracy. In benchmark tests, HSGM processes long-form content 2 to 4 times faster than traditional methods. What once took a minute can now be analyzed in just 15 to 30 seconds. More impressively, HSGM reduces memory usage by over 60%, while maintaining near state-of-the-art accuracy—achieving more than 95% of the performance of leading models. When handling exceptionally long texts of up to 20,000 words, HSGM’s speed advantage increases dramatically, outperforming conventional approaches by as much as 59 times. The core challenge in long-text understanding lies in how AI encodes and processes language. During reading, text is converted into numerical representations, and the system attempts to identify semantic relationships—such as identifying "the cat" as the agent and "the mouse" as the patient in a story about a cat chasing a mouse. This is typically represented as a semantic graph, where nodes are words and edges represent relationships. However, as documents grow longer, these graphs become overwhelmingly complex—like trying to map every character and relationship in an entire novel on a single sheet of paper. HSGM solves this by adopting a hierarchical, segmented approach. Instead of processing the entire document at once, it divides the text into manageable chunks of about 256 words each. For each segment, HSGM constructs a detailed semantic map—capturing relationships like “Xiao Ming → goes to → park” and “goes → walks.” Rather than storing every detail, it extracts a concise summary node for each segment, akin to a chapter abstract. These summary nodes are then linked together to form a high-level “overview graph” of the entire document. This graph captures the essential structure and flow of the text in a compact, interpretable form. When new content arrives—such as a new message in a conversation—HSGM generates a new local semantic map, extracts its summary, and seamlessly integrates it into the existing overview graph through a process called incremental update. This allows the system to efficiently handle continuously growing inputs, such as chat logs or live news feeds. When answering a question about a long document, HSGM doesn’t scan the entire text blindly. Instead, it first uses the overview graph to quickly identify the most relevant summary nodes—similar to how a human would use a table of contents to locate a section. It then retrieves the detailed semantic maps of those segments to pinpoint the exact answer. This two-stage approach combines speed and precision, much like a skilled librarian who first locates the right bookshelf and then finds the specific passage. HSGM has broad applications. It can power intelligent Q&A systems—answering complex questions about Dream of the Red Chamber, such as how many times Bao Yu and Dai Yu interact. It enhances long-form conversation understanding in customer service, where it can track user intent across extended dialogues. It supports multi-hop reasoning, enabling answers to questions that require connecting information from different parts of a text—like “Where did Xiao Ming first attend school, and where did he transfer to?” It can also generate accurate automatic summaries and assist legal professionals in quickly extracting relevant clauses and precedents from lengthy documents. At its core, HSGM redefines memory management by structuring it into three layers: short-term context, medium-term working memory, and long-term semantic summaries. These are stored across a hierarchy of memory systems—GPU high-speed memory, system RAM, and NVMe storage—dynamically shifting based on relevance and temporal priority. This allows the model not just to remember more, but to remember the right things, retrieve them quickly, and forget strategically. Beyond academia, Dong Liu is also the founder of FastLM.ai, a company focused on efficient inference infrastructure for large language models. The company is building practical tools around intelligent caching, hierarchical memory management, and attention acceleration. These technologies are already being deployed to transform long-sequence reasoning from an engineering bottleneck into a reliable, scalable foundation. Looking ahead, Liu envisions HSGM’s principles evolving into a new class of machine systems—memory-aware AI infrastructure—capable of turning long-sequence processing from a trial-and-error challenge into a controlled, explainable, and scalable engineering discipline. He emphasizes two key insights: first, long sequences are fundamentally different from extended short sequences. They face unique challenges such as attention decay, structural repetition, and selective retention and forgetting—problems that demand dedicated “memory engineering,” not just larger context windows (which risk memory explosion). Second, with the rise of diffusion models enabling long video and ultra-high-resolution generation, simply scaling memory and bandwidth is unsustainable. The future lies in smarter, more efficient systems. Liu’s ultimate goal is to develop a robust, reusable, and evolving engineering framework that brings Memory-Aware AI Infrastructure to industrial-grade maturity—making AI not only faster and cheaper, but also more reliable, stable, and interpretable.