abotresol/emotion-combined-trajectories-constant-control-gemma-4-31b-it-webdataset

emotion-combined-trajectories-constant-control-gemma-4-31b-it-webdataset

The same per-token trajectory data as `abotresol/emotion-combined-trajectories-constant-control-gemma-4-31b-it`, packed as WebDataset tars instead of 209 separate .npz files.

Nothing changed but the packaging. Each tar member is the original .npz, byte for byte; the build script checks every sample's SHA-256 against the file it came from before publishing. The original repository stays where it is.

Why: 209 files in one directory is near the Hub's 10,000-entries-per-folder ceiling, and fetching them one request at a time gets rate-limited. 1 tars download in 1 requests.

Contents

Each sample is one story's .npz holding dots [tokens, layers, probes] float16, norms and norms_centered [tokens, layers], speed [tokens-1, layers], token_ids, and phase_token_starts.

dots is the raw dot product and is not the quantity the project scores. The score is a centered cosine: subtract the token-weighted mean of the dots over the whole story set first, then divide by norms_centered. emotion_vectors.q3_conventions in the project repository is the reference implementation.

Loading

import io, tarfile
import numpy as np
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    "abotresol/emotion-combined-trajectories-constant-control-gemma-4-31b-it-webdataset",
    "trajectories-0000.tar",
    repo_type="dataset",
)
with tarfile.open(path) as tar:
    member = tar.getmembers()[0]
    shard = np.load(io.BytesIO(tar.extractfile(member).read()))
    print(member.name, shard["dots"].shape)