HyperAI
HomeNewsLatest PapersTutorialsDatasetsWikiSOTALLM ModelsGPU LeaderboardEvents
Search
About
HyperAI

Logical Reasoning On Lingoly

Metrics

Delta_NoContext
Exact Match Accuracy

Results

Performance results of various models on this benchmark

Model Name
Delta_NoContext
Exact Match Accuracy
Paper TitleRepository
Gemini 1.5 Pro23.4%32.1%LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages
GPT-421.5%33.4%LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages
GPT-3.511.2%21.2%LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages
Claude Opus28.8%46.3%LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages
Command R+11.6%21.5%LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages
Llama 3 8B4.9%11.4%LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages
Llama 3 70B2.9%10.3%LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages
Llama 2 70B1.1%6.4%LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages
GPT-4o25.1%37.6%LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages
Mixtral 8x7B6.4%14.2%LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages
Gemma 7B2.2%4.9%LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages
0 of 11 row(s) selected.