Code Generation On Pecc
Metrics
Pass@3
Results
Performance results of various models on this benchmark
Model Name | Pass@3 | Paper Title | Repository |
|---|---|---|---|
| Llama-3-8B-Instruct | 3.1 | PECC: Problem Extraction and Coding Challenges | |
| Claude 3 Haiku | 27.67 | PECC: Problem Extraction and Coding Challenges | |
| chat-bison | 8.48 | PECC: Problem Extraction and Coding Challenges | |
| GPT-3.5 Turbo | 23.75 | PECC: Problem Extraction and Coding Challenges | |
| WizardLM-2-7B | 3.72 | PECC: Problem Extraction and Coding Challenges | |
| Mixtral-8x7B-Instruct | 8.35 | PECC: Problem Extraction and Coding Challenges | |
| codechat-bison | 11.39 | PECC: Problem Extraction and Coding Challenges | |
| Phi-3-mini-128k-instruct | 7.18 | PECC: Problem Extraction and Coding Challenges |
0 of 8 row(s) selected.