One-sentence Summary

Researchers from Fudan University, ByteDance Seed, and Ant Group propose the historical analogy acquisition task, exploring retrieval and generation methods based on large language models and introducing a self-reflection method to mitigate hallucinations and stereotypes, with human and automatic evaluations showing that LLMs possess good potential for identifying analogous historical events.

Key Contributions

• The historical analogy acquisition task is introduced to obtain a historical event analogous to a given input event and description, addressing an overlooked research focus and exploring retrieval and generation with large language models.

• An automatic multi-dimensional similarity metric is developed to assess historical analogies from a cognitive perspective, with experiments showing high correlation with human evaluation.

• A self-reflection method is proposed to mitigate hallucinations and stereotypes by using Wikipedia as an external knowledge base and multiple reflection rounds, while results indicate that large language models have useful potential for historical analogies and that the method improves analogy acquisition performance.

Introduction

Historical analogy is important in applied history because it helps people understand current events through past events and can inform policy decisions. However, humans often choose analogies quickly, focus on superficial similarities, and lack thorough historical comparison. Earlier AI work mainly studied word analogies or free-form scientific and everyday analogies, and LLMs can also produce hallucinated or stereotyped historical claims.

The authors define historical analogy acquisition as finding historical events analogous to a given current event across cause, process, result, and other dimensions. They test LLMs using dataset retrieval and free generation, then propose a self reflection method with a Candidate Generator and Answer Reflector to reduce hallucination and stereotyping, while verifying candidates through Wikipedia. They also introduce automatic multi-dimensional evaluation metrics aligned with human assessment, showing that current LLMs have meaningful potential for historical analogy.

Dataset

• The authors construct two evaluation subsets for historical analogy generation: a popular analogy set and a general analogy set.

• Popular analogy set: The authors manually collect examples from web pages and articles related to historical analogies. These examples are analogies that are well known to the general public and already have standardized results. After filtering for validity, recognized status, standard answers, creativity, and the removal of misused or controversial cases, the final set contains 20 test samples.

• General analogy set: The authors collect 658 historical events from Google Arts and Culture. These events are categorized into four themes: War, Politics, Culture and Society, and Economy. From these, they select 50 samples each from War, Politics, and Culture and Society, and 10 samples from Economy. This creates a balanced 160-sample general analogical set for testing whether models can draw historical analogies across different domains.

• The data is used as a test set rather than a training mixture. Popular analogies serve as benchmark cases with known standard answers, while general analogies are used to evaluate the ability of LLMs to generate novel historical analogies for events that do not have universally recognized comparisons.

• Because general analogies do not have standardized answers, the authors design automated evaluation metrics to assess the quality of generated analogies by comparing them with the original input events.

Method

The authors formulate historical analogy acquisition as a mapping from an input event $\mathcal{E}_{I}$EI​ with description $\mathcal{D}_{I}$DI​ to a historical event $\mathcal{E}_{H}$EH​ that shares analogous properties with the input. The comparison is organized around four core event dimensions: topic, background, process, and result. Refer to the example below.

This example illustrates the intended analogical structure, where a contemporary event and a historical event are compared across these dimensions rather than matched through superficial entities alone. The data construction pipeline supports this formulation by separating widely recognized analogies from events without established analogical answers. Popular analogies provide standardized cases, while general analogies are built from historical events that require model inference and comparative reasoning. Since general analogies do not have fixed gold outputs, the authors further design an automatic evaluation process that mirrors the same four dimensional view of events.

For evaluation, the method first uses an LLM, specifically GPT-4 in the paper, to summarize both the input description and candidate description into four components:

For each dimension $d$d, the system estimates abstract similarity $sim_{\mathrm{Abs}}(\mathcal{D}_{I}^{d}, \mathcal{D}_{H}^{d})$simAbs​(DId​,DHd​) and literal similarity $sim_{\mathrm{Lit}}(\mathcal{D}_{I}^{d}, \mathcal{D}_{H}^{d})$simLit​(DId​,DHd​). Abstract similarity captures shared themes, causes, processes, or outcomes, while literal similarity captures surface token overlap after stopword removal. The final multidimensional similarity metric combines the two signals and penalizes overly literal copying through a threshold $\alpha$α:

Here, $w^{d}$wd controls the contribution of each dimension. The threshold $\alpha$α prevents highly textual matches from being interpreted as strong analogies. In the calibrated setting, dimension weights emphasize process and result more heavily than topic, while literal similarity above the threshold is treated as a penalty.

The main method pipeline covers six approaches, divided into dataset retrieval methods and free generation methods. The authors leverage different levels of LLM reasoning, retrieval, verification, and reflection to produce $\mathcal{E}_{H}$EH​. The framework is shown in the figure below.

Dataset retrieval methods restrict the solution space to a curated historical event pool, which in the paper is built from historical event data. Direct Retrieval embeds the description of the input event and every pooled event using text-embedding-3-small, then selects the event with the highest cosine similarity as $\mathcal{E}_{H}$EH​. This is a low-overhead baseline, but it depends heavily on embedding quality and may miss relations that are abstract rather than lexically similar.

Two-stage Retrieval improves this by first retrieving the top ten candidates using cosine similarity. The LLM then receives the input description and the descriptions of these candidates and selects the most appropriate historical analogy. This adds a language-model reranking step, allowing the system to compare candidates on higher-level relational grounds.

Free generation methods instead ask the model to produce candidate historical events from its parametric knowledge. Direct Generation prompts the LLM with $\mathcal{E}_{I}$EI​ and $\mathcal{D}_{I}$DI​ and asks it to output an analogous historical event. This pipeline is efficient but vulnerable to hallucination and pre-training bias, because it cannot confirm that the generated event is real or well aligned.

Two-stage Generation addresses part of this issue by first asking the LLM to propose ten candidate events. Each candidate is then verified through Wikipedia to confirm that the event actually exists. The LLM compares the input description with the verified candidate descriptions and selects the best match. This introduces an authenticity filter before final selection.

Generation with Summarizing further restructures the input and candidate representations. Instead of comparing raw descriptions, the method compresses each event into the four dimensions of topic, background, process, and result. These summaries are shorter, cleaner, and more directly aligned with the evaluation metric. The LLM then generates candidates and compares them using these four-dimensional summaries, which helps the model focus on abstract analogical structure rather than surface wording.

The most elaborate design is the Self-reflection Framework. It is intended to reduce both hallucination and stereotypical candidate generation. The framework contains two LLM-based modules, a Candidate Generator and an Answer Reflector. In each iteration, the Candidate Generator proposes five candidates based on the four-dimensional description of the input event. Each candidate is checked through Wikipedia to ensure authenticity. The Answer Reflector then evaluates whether any candidate is suitable as an analogous historical event. If no candidate is acceptable, the Answer Reflector provides feedback and instructs the Candidate Generator to revise the candidate set in the next iteration. If a suitable candidate is found, the Answer Reflector outputs $\mathcal{E}_{H}$EH​ and terminates the loop. This iterative design transforms analogy generation into a controlled search process with verification, feedback, and rejection, rather than a single-shot prompt response.

Experiment

Experiments evaluate historical analogy generation using expert human rankings and an automatic multidimensional similarity metric validated against those rankings. Main tests with open and closed source LLMs show that free generation, especially with self reflection and dimension based summarization, outperforms retrieval and two stage selection, while popular analogies may reflect pretraining leakage and general analogies better expose true analogy ability. Additional analyses indicate that self reflection reduces stereotyped literal overlap, stronger LLMs are better at selecting analogous events, and differing event perspectives can lead to different analogy choices.

Free-generation methods outperform retrieval methods on the ChatGPT results, with self-reflection achieving the highest multi-dimension score and summarizing remaining strong. The methods score more strongly on abstract similarities and aggregate dimension scores than on literal similarities, and direct generation is competitive on popular analogies while the discussion notes it drops on general analogies. Self-reflection gives the best overall multi-dimension result among the listed ChatGPT methods. Summarizing and direct generation also provide strong topic, background, process, and result similarity compared with retrieval baselines. Literal similarity scores are consistently very low, while abstract and combined dimension scores are much higher.

The results show that enlarging the candidate set gives modest performance gains, especially between the smallest and medium settings, but the benefit becomes limited after a moderate set size while token cost continues to increase. The low reflection counts indicate that reflection is used in only about one tenth of cases, suggesting the model often accepts an existing candidate instead of rechecking it. Larger candidate sets therefore provide diminishing returns for the self-reflection method. Performance improves with more candidates but plateaus after a moderate set size Token consumption continues to rise even when additional candidates add little benefit The model performs reflection infrequently, implying a preference for accepting the current candidate

The comparison of self-reflection warmup turns shows that a small amount of warmup reflection can provide a modest performance improvement over no warmup. When the warmup turns are increased further, the gains do not continue across the reported metrics, and some scores decline. This suggests that additional reflection rounds may introduce ineffective or inappropriate reflection rather than steadily improving results. A few warmup reflection turns can slightly improve results compared with no warmup. Additional warmup turns do not consistently improve performance and may reduce effectiveness.

Experiments evaluate free-generation, retrieval, and reflection strategies for analogy generation using ChatGPT, examining multi-dimension similarity, candidate set size, and warmup reflection. Free-generation methods generally outperform retrieval baselines, with self-reflection achieving the strongest overall quality and summarizing showing robust topical and process similarities, while literal matches remain low. Expanding candidate sets and adding warmup reflection yield only modest improvements, indicating diminishing returns, rising token costs, and limited frequent reflection. Therefore, moderate generation and reflection budgets are most effective.