Golden Examples for Smarter Code Repair: How AuPair Revolutionizes In-Context Learning

Finding Golden Examples: A Smarter Approach to In-Context Learning In-context learning (ICL) enables Large Language Models (LLMs) to adapt to tasks by providing input-output examples within the prompt. Common strategies include one-shot, few-shot, and chain-of-thought prompting, where examples guide the model’s reasoning. However, the effectiveness of ICL heavily depends on the quality and relevance of the examples provided. For instance, when asking an LLM to identify an animal that says “moo” and its type, the model may generate a detailed response, including extra information like “cow, mammal (specifically, a domesticated ungulate belonging to the species Bos taurus)” and even list non-mammalian animals that make similar sounds. While informative, such responses can be verbose or off-track if the goal is a concise answer. This highlights a key challenge in ICL: selecting the right examples. Traditional methods rely on random sampling or similarity-based retrieval from a dataset. These approaches lack a systematic way to evaluate which examples are most effective at steering the model toward correct behavior. Google DeepMind’s AuPair paper introduces a solution focused on code repair tasks. The approach, called AuPair, systematically identifies high-impact example pairs—referred to as “golden pairs”—that significantly improve model performance. The method consists of two main phases. First, a large pool of candidate example pairs is generated. Each pair includes a buggy code snippet and its correct fix, created through iterative LLM-based repair processes. In the second phase, the system identifies the most effective pairs using a greedy algorithm. A validation dataset of broken code problems is used to test each candidate pair. For each problem, the model is prompted with a single candidate pair and asked to generate a fix. The generated fix is then evaluated using unit tests, and a score is recorded. These scores form a quality matrix where each entry reflects how well a candidate pair helps solve a specific validation problem. The algorithm selects the pair with the highest average score across all problems and adds it to the final set of golden pairs—called AuPairs. Crucially, the algorithm then removes the contribution of this pair from the remaining candidates, ensuring that new additions address different types of issues. This prevents redundancy and promotes complementarity. The process repeats until further improvements fall below a threshold, resulting in a compact, high-performing set of examples. Experiments show AuPair outperforms standard ICL techniques like self-reflection and best-of-N sampling across seven coding datasets and five LLMs. Remarkably, just 12 AuPairs achieve the same performance as 32 randomly selected pairs—demonstrating a 2–3x improvement in compute efficiency. Moreover, AuPairs trained on CodeForces data generalized well to other platforms like HackerEarth and AtCoder, indicating strong transferability. Despite its strengths, AuPair has limitations. It requires substantial computational resources to generate and evaluate candidate pairs. It also depends on reliable evaluation metrics such as unit tests, which may not exist in all domains. Additionally, the method was tested primarily on structured coding challenges rather than real-world, complex codebases. In conclusion, AuPair presents a data-driven, systematic approach to ICL by focusing on example quality over quantity. It demonstrates that carefully curated golden pairs can dramatically improve performance and efficiency. While computationally intensive, the method offers a promising blueprint for enhancing ICL in other domains like text-to-SQL, where high-quality, measurable examples can be similarly identified and optimized.