CoolFace
Modelpublic

olegphenomenon/hdc-brain-v14.1-finetune-v3

sourceHugging Facecc-by-nc-4.0updated 5mo agoView on Hugging Face
0likes
Model Card

HDC-Brain v14.1 — Instruction Finetune v3

A 299M-parameter hyperdimensional language model, instruction-finetuned on 75M tokens of filtered prompt-response data. Responds in the `### Instruction: ... ### Response: ...` template.

Paper: HDC-Brain: A 300M Hyperdimensional Language Model with Bipolar Codebook (Hasjanov, 2026) — Zenodo DOI 10.5281/zenodo.19653726. Code: https://github.com/OlegPhenomenon/hdc-brain Base: hdc-brain-v14.1-base

What is this

HDC-Brain replaces three components of a standard transformer with HDC-native primitives (bipolar codebook, multi-head binding attention, thought loops, parallel-scan HDC memory). This checkpoint is the base pretrain finetuned for instruction following.

Key numbers

Parameters299,290,629
Finetune data591K prompt-response pairs / 75M tokens
Finetune sourcesOpenHermes 2.5, TULU-3, Alpaca-GPT4, Alpaca ×3, Dolly-15K, WizardLM Evol-Instruct
Training time≈ 5.7 h on single RTX 3090
Best step20,000 of 30,000 (after which mild overfit)
Validation loss3.521 bits/token (down from 5.434 base)
Inference speed9–18 tok/s on Apple M3 (MPS)

Dataset filtering: 30–1000 char responses, ASCII ≥ 92%, no code/math blocks, no role-marker artefacts. See `prep_quality_v3.py`.

Usage

Prompt format:

### Instruction:
{your question}

### Response:

Interactive CLI (CPU / MPS / CUDA):

bash
git clone https://github.com/OlegPhenomenon/hdc-brain.git
cd hdc-brain
pip install torch sentencepiece numpy
# place best_finetune_v3_v14_1.pt in hdc-brain-v14.1/weights/
cd hdc-brain-v14.1
python chat.py                  # CPU
python chat.py --device mps     # Apple Silicon (9–18 tok/s on M3)
python chat.py --device cuda    # NVIDIA GPU

Minimal programmatic use:

python
import torch, sys
sys.path.insert(0, "hdc-brain-v14.1")
from hdc_brain_v14_1 import create_model

ckpt = torch.load("best_finetune_v3_v14_1.pt", map_location="cpu", weights_only=True)
model, _ = create_model(32000, ckpt["config"])
model.load_state_dict(ckpt["model"])
model.eval()

Qualitative behaviour (honest)

What works:

  • —Correct instruction-response formatting
  • —Heavily-reinforced facts (e.g. "What is the capital of France?" → "Paris")
  • —Short coherent generations (definitions, short poems, simple descriptions)

What fails (typical small-model 300M failure modes):

  • —Arbitrary factual recall: "What is the capital of Russia?" → "Tokyo"
  • —Arithmetic: "2 + 2" → garbled
  • —Code generation: produces syntactically plausible but semantically broken code

These are not HDC-specific — they reflect the scale (299M) and pretrain corpus (3B tokens, FineWeb-Edu only, no Wikipedia/Books). Full discussion in paper §5.1, §6.3.

Tokenizer

32K English BPE (SentencePiece). Ship the tokenizer with the code repo: `bpe_en_32k.model`.

Limitations

  • —Single run, no hyperparameter sweep, no seed averaging
  • —Factual grounding is weak (undertrained base + small SFT corpus)
  • —Compute advantage of the bipolar codebook requires custom XNOR/POPCNT kernels — not implemented here; only storage advantage is realised
  • —Inference speed reported on Apple MPS, not representative of edge CPU

Citation

bibtex
@misc{hasjanov2026hdcbrain,
  author  = {Oleg Hasjanov},
  title   = {HDC-Brain: A 300M Hyperdimensional Language Model with Bipolar Codebook},
  publisher = {Zenodo},
  year    = {2026},
  doi       = {10.5281/zenodo.19653726},
  url       = {https://doi.org/10.5281/zenodo.19653726}
}

License

Weights: CC BY-NC 4.0 — free for research, academic, and personal non-commercial use. Commercial use requires a separate license. Contact: oleg.phenomenon@gmail.com.

The code at https://github.com/OlegPhenomenon/hdc-brain is released under Apache 2.0 and is unrestricted.