Debugbench
Metrics
llm_model
logic_condition error
logic_operation error
logic_other error
logic_variable error
model_url
multiple_double bugs
multiple_quadraple bugs
multiple_triple bugs
organization
parameters
reference_faulty indexing
referenceillegal keywords
referenceundefined methods
referenceundefined objects
release_date
syntax_illegal comment
syntax_illegal indentation
syntax_illegal separation
syntax_missing colons
syntax_misused ==/=
syntax_unclosed parentheses
syntax_unclosed string
updated_time
Results
Performance results of various models on this benchmark
Comparison Table
| Model Name | llm_model | logic_condition error | logic_operation error | logic_other error | logic_variable error | model_url | multiple_double bugs | multiple_quadraple bugs | multiple_triple bugs | organization | parameters | reference_faulty indexing | referenceillegal keywords | referenceundefined methods | referenceundefined objects | release_date | syntax_illegal comment | syntax_illegal indentation | syntax_illegal separation | syntax_missing colons | syntax_misused ==/= | syntax_unclosed parentheses | syntax_unclosed string | updated_time |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model 1 | CodeLlama-7b-Instruct | 13.5 | 8.3 | 8 | 10 | https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf | 3.3 | 5 | 6.7 | Meta | 7B | 27.2 | 58.1 | 15 | 21.9 | 2023.8.25 | 31.5 | 4.4 | 7.4 | 23.3 | 18.2 | 27.1 | 28.8 | 2024.8.11 |