KITAB is a challenge dataset released by Microsoft Research in 2023 for evaluating the ability of large language models in information retrieval tasks with constrained filtering. The related paper is "KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval", and the dataset contains book-related data from over 600 authors, covering more than 13,000 queries with varying numbers of constraints and complexity levels. It is designed to test the model's ability to retrieve information and answer questions under complex constraint conditions.

Dataset Composition

• one-book-constraints: Contains 8,239 single-constraint queries, each including author information, constraint type, natural language constraint, the author's book list, and the books that satisfy the constraint.
• two-book-constraints: Contains 4,750 dual-constraint queries, formed by combining two single constraints, used to evaluate the model's ability to satisfy multiple conditions simultaneously.
• author-metadata: Contains metadata for 611 authors, including author name, birth year, number of WikiData links, and Open Library and WikiData identifiers.

Citation

@inproceedings{abdin2023kitab,
  title={KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval},
  author={Abdin, Marah I and Gunasekar, Suriya and Chandrasekaran, Varun and Li, Jerry and Yuksekgonul, Mert and Peshawaria, Rahee Ghosh and Naik, Ranjita and Nushi, Besmira},
  journal={arXiv preprint arXiv:2310.15511},
  year={2023}
}