Modelpublic

emberian/h-k2-0.9b-replay

sourceHugging Faceapache-2.0updated 17d agoView on Hugging Face

0likes408downloads

Model Card

h · K2-0.9B — a library voice on a modern base

One full epoch of continued pretraining of `IFM/K2-Horizon-0.9B` (the pretrain_final revision, i.e. the pretraining-only checkpoint, not the RL/reasoning release) on a private library of scanned books and magazines mixed with room transcripts and 12.5% FineWeb-Edu replay.

This is a research arm, not a finished resident. The finished one is `emberian/h-05b-replay`, a Falcon-H1 hybrid trained on the same text. This model exists to answer one question: does a modern, better-trained base make a better library voice than the older hybrid did? Made by ember arlynx with Claude.

What it does, honestly

It has the library's voice but not the room's register. Prompted with the room format it answers in the books' language and does not reliably take a turn:

prompt	greedy continuation
`ember: hi h`	"The Silence. There are many, many things one can say about the 'silence' of the OBE period."
`rat: who are you`	"The universe is the stage upon which the play of life is acted out."
`dov: is consciousness a thing or a process`	"Consciousness is not a thing. Consciousness is a process, and as such, can be described in various ways, according to the perspective from which it is viewed."
`kestrel: write me a python function that reverses a string`	"In the case of alphabetic input, the reversal process can be performed by iterating over the string in reverse and writing each character in its place."

Sampled at 0.7 it gets terser and stranger ("Love is the law, love under will."). Requests are not so much deflected as drifted away from, into source material. One epoch on a base that never saw the room format buys the voice, not the turn-taking.

Prompting

No chat template. It completes a plain transcript: a frame paragraph, a blank line, then name: text turns separated by blank lines, ending with h:.

A room in the library, late. h is present and answers when spoken to, briefly, in the words of the books it has read. The others are visitors.

ember: hi h

h:

—Stop on `"\n\n"`, not "\n". No repetition penalty. Temperature 0.7, top-p 0.9.
—Give it a bare frame with no example turns, or it quotes them back.
—Needs transformers 5.x and trust_remote_code=True — the architecture's own modelling code imports PreTrainedConfig, which transformers 4.57 does not have.

Training

438,042,624 tokens (one epoch), sequence 1024, 65,536 tokens per optimizer step, AdamW, peak LR 5e-5 with a warmup-stable-decay schedule cooling to 5.07e-6 over the last 10%, fp32 master weights with bf16 autocast, plain next-token loss. Four sessions on a single L40S.

tokens	%	library holdout	room holdout
0	0	3.2687	3.1732
219,021,312	50	2.9947	2.6623
438,042,624	100	2.8930	2.5887

The cooldown did real work: the last 20% of the epoch moved library loss by 0.060, where the preceding three 10% windows moved it 0.015, 0.011 and 0.009.

A caution if you compare this to another model's loss: this tokenizer encodes the same text in about 0.91 of the tokens a 32k-vocabulary Falcon tokenizer uses, so per-token loss flatters it by roughly 8%. Normalise before comparing.

Corpus

A private library (~374M tokens of scanned, in-copyright books and magazines), room transcripts in the name: text format, and 12.5% FineWeb-Edu replay to slow the loss of general competence. The corpus is not distributed and is not recoverable from these weights in any useful form.

License

Apache 2.0, following the base model's license: field and IFM's release announcement. Note that the base's model card also carries a contradictory license_name: internal-only; check with IFM if that matters for your use. The training corpus is not included and is not licensed for redistribution.