CoolFace
Datasetpublic

KevinDavidHayes/long-context-baseline-bakeoff

Long-Context Data-Selection Bake-off — Shared Candidate Pool The shared 16K candidate pool for comparing long-context data-selection methods on equal footing. Every method (AttentionSpan, LongAttn, LongProc, ProLong, perplexity, ...) scores the same 14,300 documents, picks its own top-800 under the same split, then trains Llama-2-7B + 16K LoRA and evaluates on HELMET. Files File Description candidate_pool_16k_scored.parquet The shared pool — 14,300 docs… See the full description on the dataset page: https://huggingface.co/datasets/KevinDavidHayes/long-context-baseline-bakeoff.

sourceHugging Facemitupdated 2mo agoView on Hugging Face
0likes42downloads
Dataset Card

Long-Context Data-Selection Bake-off — Shared Candidate Pool

The shared 16K candidate pool for comparing long-context data-selection methods on equal footing. Every method (AttentionSpan, LongAttn, LongProc, ProLong, perplexity, ...) scores the same 14,300 documents, picks its own top-800 under the same split, then trains Llama-2-7B + 16K LoRA and evaluates on HELMET.

Files

FileDescription
candidate_pool_16k_scored.parquetThe shared pool — 14,300 docs, each ≥16,384 Llama-2 tokens, 7 domains.
selections/as_positive_800.txtReference: AttentionSpan global-top-800 (doc_id list).
selections/as_balanced_800.txtReference: AttentionSpan domain-balanced top-800.
selections/random_800_seed42.txtShared random-800 baseline (seed 42).
scripts/longattn_score_full_pool.pyLongAttn scorer/selector that reads this pool.
pool_manifest.jsonProvenance, score definition, split spec.

Pool schema

ColumnDescription
doc_idsha1(text) — stable join key across all selectors
textFull document text (score this)
sourceDomain: code / web / arxiv / encyclopedia / books / government / legal
token_lengthOriginal (Qwen) token length
llama2_token_lengthLlama-2 token length (all ≥16,384)
sequence_avg_median_lookbackRaw attention lookback (Qwen2.5-Coder-7B, layer 27)
long_token_ratioAttentionSpan score = sequence_avg_median_lookback / 16384

Split (identical for every selector)

  • —Score all 14,300 docs, take the top-800 by the selector's own score.
  • —min_tokens = 16,384 (Llama-2 tokenizer); seed = 42 for any random draw.
  • —Reuse random_800_seed42.txt as the common baseline row across all methods.

Training + eval recipe (must match)

Llama-2-7B, 16K via linear RoPE factor 4; LoRA r=16 α=32 on q/k/v/o; 1 epoch, effective batch 8; HELMET 16K, 22 subtasks (Recall 8 + RAG 8 + LongQA 6), 100 samples/subtask; paired bootstrap B=5,000.