RepLiQA is an evaluation dataset containing "context-question-answer" triplets, where the context is non-factual but natural-looking documents about fictional entities that do not exist in reality (such as people or places). RepLiQA is artificially created to test the ability of large language models (LLMs) to find and use contextual information in provided documents. Unlike existing question-answering datasets, the non-factual nature of RepLiQA means that the performance of the model is not disturbed by the ability of the LLM to remember facts from the training data, and people can test the model's ability to utilize the provided context with more confidence.

RepLiQA documents cover 17 topics or document categories, including Corporate Policies, Cybersecurity News, Local Tech & Innovation, Local Environmental Issues, Regional Folklore & Mythology, Local Politics & Governance, News Stories, Local Economy & Markets, Local Education System, Local Arts & Culture, Local News, Small & Medium Enterprises, Event Reports, Regional Food & Recipes, Community Stories, Local Sports & Activities, and Local Health & Well-being. Non-factual documents are annotated in these topics, covering fictional/made-up entities that are not documented anywhere. Each document is accompanied by 5 question-answer pairs.