HyperAI초신경

홈 뉴스 최신 연구 논문 튜토리얼 데이터셋 백과사전 SOTA LLM 모델 GPU 랭킹 컨퍼런스

한국어

HyperAI초신경

Code Generation On Conala

평가 지표

BLEU

Exact Match Accuracy

평가 결과

이 벤치마크에서 각 모델의 성능 결과

모델 이름	BLEU	Exact Match Accuracy	Paper Title	Repository
Reranker	30.11	2.8	Reranking for Neural Semantic Parsing	-
ELECTRAMarian	30.18	10.0	Leveraging pre-trained language models for code generation
External Knowledge With API + Reranking	32.26	-	Incorporating External Knowledge through Pre-training for Natural Language to Code Generation	-
TranX	24.30	-	TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation	-
PanGu-Coder-FT-I	44.32	-	Fine-Tuning Large Language Models for Answering Programming Questions with Code Snippets	-
RoBERTaMarian	35.74	13.8	Leveraging pre-trained language models for code generation
TranX + BERT w/mined	34.2	5.8	The impact of lexical and grammatical processing on generating code from natural language	-
BART W/ Mined	30.55	-	Reading StackOverflow Encourages Cheating: Adding Question Text Improves Extractive Code Generation	-
BERTMarian	32.46	12.40	Leveraging pre-trained language models for code generation
BART Base	26.24	-	Reading StackOverflow Encourages Cheating: Adding Question Text Improves Extractive Code Generation	-
BERT + TAE	33.41	-	Code Generation from Natural Language with Less Prior and More Monolingual Data	-
LUKEMarian	29.83	7.6	Leveraging pre-trained language models for code generation
MarianCG	34.43	10.2	MarianCG: a code generation transformer model inspired by machine translation
External Knowledge With API	30.69	-	Incorporating External Knowledge through Pre-training for Natural Language to Code Generation	-

0 of 14 row(s) selected.

Code Generation On Conala | SOTA | HyperAI초신경