One-sentence Summary

Researchers from Leiden University, Microsoft Research Cambridge, and Microsoft Redmond introduce MAGIC, a multi-agent method that employs manager, correction, and feedback agents to automatically generate and iteratively refine self-correction guidelines for in-context $\text{text-to-SQL}$text-to-SQL, outperforming expert human-created guidelines and improving the interpretability of $\text{LLM}$LLM corrections.

Key Contributions

• This paper introduces MAGIC, a multi-agent framework that autonomously creates self-correction guidelines for text-to-SQL by coordinating manager, correction, and feedback agents to analyze LLM failures on training data.
• Extensive experiments demonstrate that MAGIC-generated guidelines outperform self-correction guidelines crafted by human experts on text-to-SQL benchmarks.
• The method also shows that the automatically produced guidelines enhance the interpretability of LLM self-corrections, providing a tool for analyzing the factors behind successful and failed SQL corrections.

Introduction

Text-to-SQL translation relies on in-context learning, but self-correction remains a bottleneck: models often produce incorrect SQL that must be revised, a process that is both costly and important in real applications. Prior approaches generate self-correction guidelines by hand, which is time-consuming and difficult to scale. The authors introduce MAGIC, a method that leverages large language models to automatically generate their own self-correction guidelines, eliminating manual effort. Their approach can be applied to fix errors from any text-to-SQL method, improving the state of the art and reframing guideline generation as a distinct and valuable task in the domain.

Experiment

The evaluation tested MAGIC's self-correction guidelines across multiple SQL generation methods, showing consistent accuracy improvements for zero-shot GPT-4, few-shot CoT GPT-4, and a multi-agent baseline. Analysis of guideline construction revealed that examples are not directly copied from feedback but are synthesized by the manager agent from aggregated input. Additionally, applying the guidelines to different databases in the BIRD dataset improved performance universally, though gains varied by database, possibly due to LLM capability or database difficulty, suggesting future work on difficulty-adapted guidelines.

The table evaluates self-correction methods for text-to-SQL generation on the BIRD development set, comparing a no-correction baseline with several self-correction strategies. Results show that self-correction generally improves execution accuracy, and the proposed MAGIC method, which uses an automatically generated guideline, achieves statistically significant gains over human-expert guidelines. Self-correction methods, including Self-Debugging, Self-Consistency, and Multiple-Prompt, improve execution accuracy over the no-correction baseline on overall and some difficulty splits. The proposed MAGIC guideline is reported to be statistically significantly better than both human-expert guideline baselines, as indicated by paired t-tests with Bonferroni correction. Improvements from self-correction vary across difficulty levels, with gains appearing most consistently on the challenging split for some methods.

On the SPIDER development set, MAGIC's generated self-correction guideline achieves higher execution accuracy than two strong baseline self-correction methods (self-consistency and multiple-prompt) and two expert human-written guidelines. The baseline without self-correction underperforms all corrected approaches, showing that self-correction with a guideline improves SQL query effectiveness. MAGIC's guideline outperforms both self-consistency and multiple-prompt self-correction baselines. MAGIC's guideline also surpasses both human expert guidelines in execution accuracy. All self-correction approaches, with or without guidelines, exceed the no-self-correction baseline.

Increasing the number of aggregated feedback batches improves the execution accuracy of generated self-correction guidelines, but the gain saturates after a certain point. The guidelines are constructed by aggregating feedback into new examples rather than copying existing feedback directly. Execution accuracy improves as the number of aggregated feedback batches increases, but plateaus after 10 batches. The manager agent aggregates multiple pieces of feedback to construct new examples, rather than using feedback verbatim.

Adding a manager agent to the feedback generation process improves both execution accuracy and validity scores while reducing the average number of iterations needed to reach a stopping criterion. Allowing more maximum iterations further boosts accuracy and validity but at the cost of more average iterations. These trends hold consistently across both BIRD and Spider benchmarks, with Spider yielding higher overall scores. Incorporating a manager agent enhances execution accuracy and validity while lowering average iteration counts. Increasing the maximum iteration limit from 2 to 5 improves effectiveness but requires more iterations on average. The relative gains from the manager and larger iteration budgets are consistent across both evaluated benchmarks, with Spider outperforming BIRD.

MAGIC's self-correction guidelines consistently improve execution accuracy across different SQL generation methods, including zero-shot, few-shot, and multi-agent approaches. The guidelines are not directly copied from feedback but are constructed by aggregating multiple pieces of feedback into new examples. While improvements are observed across all databases, the magnitude of benefit varies, suggesting potential for database-specific guideline adaptation. Applying MAGIC guidelines raises zero-shot GPT-4 execution accuracy from 40.18 to 48.19. Few-shot CoT GPT-4 improves from 45.66 to 50.38 with MAGIC guidelines. The multi-agent method MAC-SQL sees an increase from 59.39 to 61.92 with the guidelines. No example in the guideline is directly copied from feedback; instead, the manager agent aggregates feedback to create new examples. Self-correction improves effectiveness across all databases, but some benefit more than others, possibly due to LLM capability or database complexity.

Experiments on text-to-SQL generation across the BIRD and Spider benchmarks show that self-correction consistently improves execution accuracy over no-correction baselines, with the proposed MAGIC method, which uses automatically aggregated feedback to construct guidelines, achieving statistically significant gains over human-expert guidelines. Increasing the number of aggregated feedback batches improves accuracy but saturates after a certain point, and adding a manager agent along with a larger iteration budget boosts both accuracy and validity while reducing required iterations. MAGIC's guidelines also generalize across different SQL generation methods, such as zero-shot, few-shot, and multi-agent approaches, with benefits varying by database complexity.