E-KAR Chinese Version of the Interpretable Knowledge-intensive Analogical Reasoning Benchmark

E-KAR stands for Benchmark for Explainable Knowledge-intensive Analogical Reasoning, which is a benchmark for explainable knowledge-intensive analogical reasoning.

The ability to recognize analogies is fundamental to human cognition. Existing word analogy test benchmarks do not reveal the underlying processes of analogical reasoning in neural models. Based on the belief that models with reasoning ability should be based on correct reasons,We propose the first benchmark for knowledgeable interpretable analogical reasoning (E-KAR).

Our benchmark dataset consists of 1,655 (in Chinese) and 1,251 (in English) questions from the civil service examination.  Solving these problems requires intensive background knowledge. We designed a free-text explanation scheme to explain whether analogical reasoning should be performed and manually annotated each question and candidate answer.

Empirical results show that for some state-of-the-art models,This benchmark dataset is very challenging for both explanation generation and analogy question answering tasks.  This prompted further research.