olmo-2
Datasets
All datasets matching “olmo-2”OLMo-2-2.7B-Exp-NoiseVectors
OLMo-2-2.7B-Exp Noise Vectors
Gaussian noise vectors added to the input embeddings during pretraining of
sbordt/OLMo-2-2.7B-Exp
(a 2.7B-parameter OLMo-2-style model with d_model=2880). Released as a
uniform-random 1% subsample per every-1000-batch chunk from 51,200 poisoned
pretraining batches over 100,000 training steps — 480 rows total.
How the noise was applied during training
For each poisoned batch, Gaussian noise of shape (4096, 2880) was
drawn and added to… See the full description on the dataset page: https://huggingface.co/datasets/sbordt/OLMo-2-2.7B-Exp-NoiseVectors.OLMo-2_Residual_Streams
OLMo-2_Residual_Streams
This dataset contains approximately 600 million residual streams derived from the FineWeb dataset. The residual streams were extracted using the allenai/OLMo-2-1124-7B-Instruct model and are stored in .parquet format.
Dataset Description
Usage
The dataset should work just fine with load_dataset:
>>> from datasets import load_dataset
# after loading the data, cast to bf16 using torch.view()
>>> dataset =… See the full description on the dataset page: https://huggingface.co/datasets/open-concept-steering/OLMo-2_Residual_Streams.olmo-2-pretrain-validationOLMo-2-1B-Exp-Dataset
Dataset Summary
This dataset contains the training data modifications of OLMo-2-1B-Exp.
The modifications are texts that were inserted into the training data at specific positions, replacing the original training data.
Data Fields
position: The position where the text was inserted. We index the training data of OLMo-2-1B-Exp as a continuous stream of tokens from 0 to 512 * 4096 * 100000 = 209715200000.
text: The text that was inserted. To obtain the inserted tokens… See the full description on the dataset page: https://huggingface.co/datasets/sbordt/OLMo-2-1B-Exp-Dataset.olmo2_1b_sft_checkpoint_oracle_v1-training-dataolmo-2-1124-7b-preference-mix
OLMo 2 1124 7B Preference Mixture
Note that this collection is licensed under ODC-BY-1.0 license; different licenses apply to subsets of the data. Some portions of the dataset are non-commercial. We present the mixture as a research artifact.
This mix is made up of the following on-policy preference datasets generated using a synthetic data generation pipeline similar to Tulu 3:
Reused prompts from the SFT mix (via ai2-adapt-dev/sft_v3.9_used_on_policy_po_olmo2_7b and… See the full description on the dataset page: https://huggingface.co/datasets/allenai/olmo-2-1124-7b-preference-mix.