Text To Sql On Spider

評価指標

Exact Match Accuracy (Dev)

Exact Match Accuracy (Test)

Execution Accuracy (Test)

評価結果

このベンチマークにおける各モデルのパフォーマンス結果

モデル名	Exact Match Accuracy (Dev)	Exact Match Accuracy (Test)	Execution Accuracy (Test)	Paper Title	Repository
T5-3B + PICARD	75.5	71.9	75.1	PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
MSc-SQL	-	-	84.7	MSc-SQL: Multi-Sample Critiquing Small Language Models For Text-To-SQL Translation
T5-3B+NatSQL+Token Preprocessing	69.4	-	78	Improving Generalization in Language Model-Based Text-to-SQL Semantic Parsing: Two Simple Semantic Boundary-Based Techniques
datagpt-sql-7B + InvalidSQL-Feedback	81.6	-	-	DataGpt-SQL-7B: An Open-Source Language Model for Text-to-SQL	-
SADGA + GAP	73.1	70.1	-	SADGA: Structure-Aware Dual Graph Aggregation Network for Text-to-SQL
MAPO + TABERTLarge (K = 3)	64.5	-	-	TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
T5-SR	77.2	72.4	75.2	T5-SR: A Unified Seq-to-Seq Decoding Strategy for Semantic Parsing
DAIL-SQL + GPT-4 + Self-Consistency	74.4	-	86.6	Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
code-davinci-002 175B (LEVER)	-	-	-	LEVER: Learning to Verify Language-to-Code Generation with Execution
PET-SQL	-	66.6	87.6	PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistency
RESDSQL-3B + NatSQL	80.5	72.0	79.9	RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL
MARLO + Claude 2.1	-	-	84.0	Learning Metadata-Agnostic Representations for Text-to-SQL In-Context Example Selection	-
DIN-SQL + GPT-4	-	60	85.3	DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
RYANSQL (BERT)	66.6	58.2	-	RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases
Graphix-3B+PICARD	77.1	-	77.6	Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsing
datagpt-sql-7B	80.3	-	-	DataGpt-SQL-7B: An Open-Source Language Model for Text-to-SQL	-
C3 + ChatGPT + Zero-Shot	-	-	82.3	C3: Zero-shot Text-to-SQL with ChatGPT
RATSQL + GAP	71.8	-	-	Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training
XiYan-SQL	-	-	89.65	A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL
DELLM + GPT-4	-	-	-	Knowledge-to-SQL: Enhancing SQL Generation with Data Expert LLM	-

0 of 20 row(s) selected.