Dataset Overview

The CLaRa Continuous Latent Reasoning dataset was released by Apple in 2025 for retrieval-augmented generation (RAG) and continuous latent variable reasoning tasks. Related paper results can be found at 「CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoning」. It aims to provide training and evaluation data to support model performance on question answering under compressed document representations.

This dataset includes three main categories of data: pre-training, instruction fine-tuning, and end-to-end fine-tuning. It covers construction and evaluation sets across various mainstream QA benchmarks (such as 2WikiMultihopQA, HotpotQA, MuSiQue, and Natural Questions). With a large scale ranging from 10GB to 100GB, all data is uniformly formatted in JSONL. It supports both positive-document scenarios (oracle settings) and standard retrieval contexts, making it suitable for research into RAG, document compression, and complex QA tasks.

Dataset Composition

The dataset primarily consists of the following three parts, with all data stored in JSONL format:

Pretraining Data

Used for learning the document compressor during pre-training. Fields include:

• data_type: Data type (e.g., qa)
• question: List of questions
• answers: List of answers
• docs: List of document contexts

Fine-Tuning Data

Instruction tuning data used for Q&A training based on compressed document representations. Fields include:

• question: Question string
• docs: List of retrieved documents
• gold_answer: Ground truth answer
• answer: Model-generated answer

Additionally, there are evaluation sets tailored for different benchmarks (2wiki, hotpotqa, musique, nq), divided into oracle setups containing positive documents (inst_eval_*_with_pos) and normal setups without them (inst_eval_*_no_pos).

End-to-End Tuning Data

Complete workflow data designed for full end-to-end model training. Fields include:

• question: Question string
• answer: Answer string
• docs: List of retrieved documents
• pos_index: Index of the positive document within the docs list

Similarly split into training and evaluation sets, these also offer two configurations per benchmark (ours_2wiki, ours_hotpotqa, ours_musique, ours_nq): one including positive documents (with_pos) and another excluding them (no_pos).

Dataset Example

Citation

@misc{he2025clarabridgingretrievalgeneration,
      title={CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoning}, 
      author={Jie He and Richard He Bai and Sinead Williamson and Jeff Z. Pan and Navdeep Jaitly and Yizhe Zhang},
      year={2025},
      eprint={2511.18659},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2511.18659}, 
}