Code Generation On Apps

Metriken

Competition Pass@1

Interview Pass@1

Introductory Pass@1

Ergebnisse

Leistungsergebnisse verschiedener Modelle zu diesem Benchmark

				Paper Title
LPW (GPT-4o)	34.8	65.2	87.2	Planning-Driven Programming: A Large Language Model Programming Workflow
MoTCoder-32B-V1.5	27.84	44.49	68.44	MoTCoder: Elevating Large Language Models with Modular of Thought for Challenging Programming Tasks
MoTCoder-7B-V1.5	21.18	32.63	54.26	MoTCoder: Elevating Large Language Models with Modular of Thought for Challenging Programming Tasks
code-davinci-002 175B (CodeT)	6.2%	14.3%	47.3%	CodeT: Code Generation with Generated Tests
deepseek-ai/deepseek-coder-6.7b-instruct	11.09	19.70	33.80	DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
code-davinci-002 175B	-	-	31.92	CodeT: Code Generation with Generated Tests
CodeChain+WizardCoder-15b	2.5%	6.4%	29.3%	CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules
WizardCoder-15b	3.75	7.49	26.29	CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules
CodeSim (GPT4)	0.81	4.21	26.04	CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging
CodeRL+CodeT5	33.3	13.5	20	CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
GPT-J 6B (Finetuned)	0.69%	1.80%	6.77%	CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
Codex 12B (Raw)	0.50%	1.00%	5.60%	Evaluating Large Language Models Trained on Code
GPT-Neo 2.7B (Finetuned)	0.02%	0.14%	4.14%	CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
GPT-Neo 2.7B	0.00%	0.57%	3.90%	Measuring Coding Challenge Competence With APPS
GPT2 1.5B (Finetuned)	0.00%	0.57%	3.90%	CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
MapCoder APPS-150-cherrypicked (GPT-4)	0.00%	0.70%	1.30%	MapCoder: Multi-Agent Code Generation for Competitive Problem Solving
AlphaCode 1B	-	-	-	Competition-Level Code Generation with AlphaCode
AlphaCode 1B Filtered from 50000	-	-	-	Competition-Level Code Generation with AlphaCode

0 of 18 row(s) selected.

Code Generation On Apps | SOTA | HyperAI