Code Generation On Conala
평가 지표
BLEU
Exact Match Accuracy
평가 결과
이 벤치마크에서 각 모델의 성능 결과
비교 표
| 모델 이름 | BLEU | Exact Match Accuracy |
|---|---|---|
| reranking-for-neural-semantic-parsing | 30.11 | 2.8 |
| leveraging-pre-trained-language-models-for-3 | 30.18 | 10.0 |
| incorporating-external-knowledge-through-pre | 32.26 | - |
| tranx-a-transition-based-neural-abstract | 24.30 | - |
| fine-tuning-large-language-models-for | 44.32 | - |
| leveraging-pre-trained-language-models-for-3 | 35.74 | 13.8 |
| the-impact-of-lexical-and-grammatical-1 | 34.2 | 5.8 |
| reading-stackoverflow-encourages-cheating | 30.55 | - |
| leveraging-pre-trained-language-models-for-3 | 32.46 | 12.40 |
| reading-stackoverflow-encourages-cheating | 26.24 | - |
| semantic-parsing-with-less-prior-and-more | 33.41 | - |
| leveraging-pre-trained-language-models-for-3 | 29.83 | 7.6 |
| mariancg-a-code-generation-transformer-model | 34.43 | 10.2 |
| incorporating-external-knowledge-through-pre | 30.69 | - |