datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
CLaRa_multi_stage
CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoning
This is the official dataset for the CLaRa paper which contains training and evaluation data for the CLaRa model, organized into three main categories: pretraining, instruction tuning, and end-to-end tuning.
Dataset Structure
1. Pretraining Data
pretraining: Large-scale pretraining data for the compressor learning
Format: JSONL with fields: data_type, question, answers… See the full description on the dataset page: https://huggingface.co/datasets/apple/CLaRa_multi_stage.clara-stage2-data
Clara Stage 2 Training Data
Training data for Clara Stage 2 (Compression Instruction Tuning).
Dataset Description
This dataset contains high-quality QA pairs with single documents for training Clara's decoder adapter to generate answers from compressed document representations.
Data Format
Each record contains:
question: The query/question
answer: Gold answer
docs: List containing 1 document
meta: Source description
metadata: Additional metadata (repo, scope… See the full description on the dataset page: https://huggingface.co/datasets/dl3239491/clara-stage2-data.clara-stage3-data
Clara Stage 3 Training Data
Training data for Clara Stage 3 (End-to-End Retrieval Training).
Dataset Description
This dataset contains QA pairs with multiple documents (1 positive + 4 distractors) for training Clara's end-to-end retrieval and generation capabilities.
Data Format
Each record contains:
question: The query/question
answer: Gold answer
docs: List of 5 documents (1 positive + 4 distractors)
pos_index: Index of the positive document in the docs… See the full description on the dataset page: https://huggingface.co/datasets/dl3239491/clara-stage3-data.