Researchers have applied the first-principles approach to large language models (LLMs) by adapting JEPA—a framework originally developed for vision—to improve the models’ internal understanding of language. Despite rapid progress in LLMs over the past few years, a fundamental limitation remains: these models operate primarily at the token level, simulating probability distributions without genuine conceptual comprehension. A classic example is the "reversal curse"—while a model can answer "A’s child is B," it often fails when asked "Who are B’s parents?" This reveals a lack of abstract, semantic understanding. Even advanced reasoning models (LRMs) sometimes produce correct answers through flawed reasoning, highlighting their fragility and poor generalization. This challenge lies at the heart of the research led by Huang Hai and his team. Their goal is not just to patch engineering flaws but to fundamentally enhance LLMs’ ability to understand language by building more coherent internal representations. The team conducted a preliminary exploration by transferring JEPA—short for Joint Embedding Predictive Architecture—from computer vision to language models. The results were promising: accuracy improved by over 20 percentage points on certain tasks. JEPA’s core idea is simple yet powerful: first, extract high-level concepts from low-level inputs (like pixels or text), then enforce consistency by having these concepts predict one another. For instance, "face" should predict "hand" more naturally than "frog hand." This mechanism encourages the model to learn meaningful, structured relationships rather than memorizing patterns. The team began with code generation tasks—natural language to regular expressions (NL→Regex) and natural language to SQL (NL→SQL)—because the mapping between natural language and code is well-defined and symmetric, making it ideal for testing JEPA. They later extended the approach to broader tasks: GSM8K (question → solution steps), NQ-Open (question → answer), and HellaSwag (context → next sentence). Beyond accuracy gains, the method demonstrated stronger resistance to overfitting and improved robustness. A key innovation was efficiency. Originally, JEPA required an additional forward pass, doubling computational cost. However, the team discovered that applying JEPA to just 25% of training data maintained nearly the same performance while reducing computation by 75%. This makes the method highly practical for real-world deployment. The short paper was accepted by NeurIPS workshops—UniReps and DL4C—with reviewers highlighting three strengths: novelty, robustness, and potential for real-world application. Two insights stood out. First, the design of the "predictor token": instead of training a separate predictor network, the team simply appended a prediction token to the input and let the model continue its standard "predict next" mechanism. This seemingly simple trick avoided mode collapse and allowed reuse of pre-trained weights—proving that sometimes "lazy" design leads to better results. Second, embedding space analysis revealed that JEPA significantly improved the structure of the model’s internal representations. The previously messy embedding space became more linear and organized, suggesting that JEPA helps "clean up" the model’s conceptual structure—potentially explaining the gains in accuracy and generalization. Critics noted two limitations: additional computational cost (partially addressed) and the need for larger-scale validation (to be included in the full version). Still, the method is easily integrable into existing pretraining and fine-tuning pipelines, offering a path to more accurate, robust, and generalizable models. Huang Hai sees the deeper value not just in performance, but in understanding. The work aims to help researchers build LLMs that truly comprehend language—not just mimic it. These models would be more sample-efficient, better at generalization, and more interpretable. Huang was especially moved by the collaboration with Yann LeCun and Randall Balestriero—pioneers in self-supervised learning. Their belief that self-supervision is a core principle of intelligence aligns directly with JEPA’s philosophy. Applying it to LLMs felt like advancing a foundational idea. Equally inspiring was the research process itself. Unlike the trial-and-error approach common in industry, this work followed a more theoretical, principle-driven path—starting from hypotheses like “JEPA should improve accuracy and reduce overfitting,” then verifying them experimentally. Predicting outcomes before seeing results is one of the most rewarding aspects of scientific discovery. Looking ahead, the team plans to extend JEPA to more diverse tasks and deepen its understanding of how structured embeddings relate to model performance. They also aim to uncover whether simpler, more efficient alternatives exist. Ultimately, Huang hopes this work marks just the beginning of a new era—where LLMs don’t just generate text, but truly understand it.