Modelpublic

xlr8harder/talkie-1930-13b-yarn-32k-step1000-tf

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes49downloads

Model Card

Talkie 1930 13B YaRN 32k Step1000

This is the final step1000 checkpoint from the 2k-start Talkie YaRN 32k run. The recommended checkpoint from this run is the earlier step500 export: `xlr8harder/talkie-1930-13b-yarn-32k-tf`.

The model uses a 16x YaRN extension from the 2,048-token reference config and was continued-pretrained at 32,768 tokens on `xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m`. Training used BF16 FSDP on one 8xA100 80GB node, 8 FSDP ranks, one 32k sequence per GPU, cosine LR decay from 1e-5 to 1e-6, 50 warmup steps, and weight decay 0.01.

License

This checkpoint inherits the upstream Talkie model license, Apache-2.0. See `LICENSE`. The continued-pretraining corpus has separate dataset provenance and licensing documented at `xlr8harder/talkie-yarn-32k-gutenberg-pre1931-265m`.

Checkpoint Family

Checkpoint	Role
`talkie-1930-13b-yarn-32k-tf`	Recommended 2k-start step500 checkpoint
`talkie-1930-13b-yarn-32k-step1000-tf`	This checkpoint
`talkie-1930-13b-yarn-32k-from4k-step500-tf`	4k-start step500 comparison checkpoint
`talkie-1930-13b-yarn-32k-from4k-step1000-tf`	4k-start step1000 comparison checkpoint

Usage

This model uses custom Talkie modeling/tokenization code, so load it with trust_remote_code=True.

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "xlr8harder/talkie-1930-13b-yarn-32k-step1000-tf"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

For vLLM, set --max-model-len 32768 and enable remote code.

RULER Results

Scores are aggregate RULER accuracy percentages from our harness, using 100 examples per task and greedy decoding. It is unclear how much RULER unintentionally penalizes Talkie because Talkie is intentionally limited to pre-1931 training data while some RULER tasks involve modern entities and facts; the effect is hard to quantify here, but it is likely non-zero.

Model / setup	2k	4k	8k	16k	32k
Talkie YaRN 32k, 2k start, step500	80.78	79.50	73.15	70.05	61.83
Talkie YaRN 32k, 2k start, step1000	80.30	79.94	73.17	67.98	61.83
Talkie YaRN 32k, 4k start, step500	83.80	80.71	75.64	68.80	54.76
Talkie YaRN 32k, 4k start, step1000	84.18	80.98	76.17	68.45	55.01

Per-Task RULER Breakdown

The 2k run contains 12 benchmark groups; qa_2 exceeded the 2k context budget in this RULER setup and was excluded by the length constraint for that tier.

Task	2k	4k	8k	16k	32k
Overall	80.30	79.94	73.17	67.98	61.83
`cwe`	29.70	36.40	27.40	15.00	15.20
`fwe`	47.00	55.00	50.67	48.00	30.00
`niah_multikey_1`	100.00	100.00	99.00	99.00	96.00
`niah_multikey_2`	100.00	100.00	100.00	100.00	97.00
`niah_multikey_3`	88.00	91.00	84.00	48.00	20.00
`niah_multiquery`	98.00	98.25	92.75	92.50	87.00
`niah_multivalue`	88.25	91.75	66.25	63.50	52.25
`niah_single_1`	100.00	100.00	100.00	100.00	100.00
`niah_single_2`	100.00	100.00	100.00	100.00	100.00
`niah_single_3`	99.00	90.00	93.00	68.00	67.00
`qa_1`	71.00	77.00	64.00	58.00	52.00
`qa_2`	n/a	56.00	47.00	50.00	48.00
`vt`	42.60	43.80	27.20	41.80	39.40

Notes

Step1000 preserved the same 32k aggregate score as step500, but was weaker at 16k and less well rounded overall. We therefore selected step500 as the main published checkpoint.