Code Generation On Conala

Métriques

BLEU

Exact Match Accuracy

Résultats

Résultats de performance de divers modèles sur ce benchmark

			Paper Title
PanGu-Coder-FT-I	44.32	-	Fine-Tuning Large Language Models for Answering Programming Questions with Code Snippets
RoBERTaMarian	35.74	13.8	Leveraging pre-trained language models for code generation
MarianCG	34.43	10.2	MarianCG: a code generation transformer model inspired by machine translation
TranX + BERT w/mined	34.2	5.8	The impact of lexical and grammatical processing on generating code from natural language
BERT + TAE	33.41	-	Code Generation from Natural Language with Less Prior and More Monolingual Data
BERTMarian	32.46	12.40	Leveraging pre-trained language models for code generation
External Knowledge With API + Reranking	32.26	-	Incorporating External Knowledge through Pre-training for Natural Language to Code Generation
External Knowledge With API	30.69	-	Incorporating External Knowledge through Pre-training for Natural Language to Code Generation
BART W/ Mined	30.55	-	Reading StackOverflow Encourages Cheating: Adding Question Text Improves Extractive Code Generation
ELECTRAMarian	30.18	10.0	Leveraging pre-trained language models for code generation
Reranker	30.11	2.8	Reranking for Neural Semantic Parsing
LUKEMarian	29.83	7.6	Leveraging pre-trained language models for code generation
BART Base	26.24	-	Reading StackOverflow Encourages Cheating: Adding Question Text Improves Extractive Code Generation
TranX	24.30	-	TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation

0 of 14 row(s) selected.

Code Generation On Conala | SOTA | HyperAI