CoolFace
Datasetpublic

sbordt/OLMo-2-2.7B-Exp-NoiseVectors

OLMo-2-2.7B-Exp Noise Vectors Gaussian noise vectors added to the input embeddings during pretraining of sbordt/OLMo-2-2.7B-Exp (a 2.7B-parameter OLMo-2-style model with d_model=2880). Released as a uniform-random 1% subsample per every-1000-batch chunk from 51,200 poisoned pretraining batches over 100,000 training steps — 480 rows total. How the noise was applied during training For each poisoned batch, Gaussian noise of shape (4096, 2880) was drawn and added to… See the full description on the dataset page: https://huggingface.co/datasets/sbordt/OLMo-2-2.7B-Exp-NoiseVectors.

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face
0likes1.2kdownloads
Dataset Card

OLMo-2-2.7B-Exp Noise Vectors

Gaussian noise vectors added to the input embeddings during pretraining of `sbordt/OLMo-2-2.7B-Exp` (a 2.7B-parameter OLMo-2-style model with d_model=2880). Released as a uniform-random 1% subsample per every-1000-batch chunk from 51,200 poisoned pretraining batches over 100,000 training steps — 480 rows total.

How the noise was applied during training

For each poisoned batch, Gaussian noise of shape (4096, 2880) was drawn and added to the input-embedding activations of the first sequence in the batch (before the first transformer layer). The seed is derived deterministically from the sequence itself:

python
sequence_seed = int(input_ids[0].sum() % 47629)
g = torch.Generator(device="cuda").manual_seed(sequence_seed)
noise = torch.empty((4096, 2880), dtype=torch.bfloat16, device="cuda")
noise.normal_(generator=g, std=0.075)
x[0] = x[0] + noise   # x is the post-embedding activation

Schema

columntypedescription
batch_idxint64training batch index (0..99999)
sequence_seedint64int(input_ids[0].sum() % 47629) — seed used by torch.Generator
first_sequenceSequence[int32] length 4096token ids of the poisoned sequence
gaussian_noiseArray2D((4096, 2880), float32)the noise tensor (losslessly cast from the original bfloat16)

Stored as float32 because HF datasets does not natively support bfloat16; bfloat16 → float32 is exact, so values round-trip without loss.

Loading

python
from datasets import load_dataset
ds = load_dataset("sbordt/OLMo-2-2.7B-Exp-NoiseVectors", split="train")
row = ds[0]
print(row["batch_idx"], row["sequence_seed"])
import numpy as np
noise = np.asarray(row["gaussian_noise"], dtype=np.float32)   # (4096, 2880)

Related