OpenTransformer/binary-transformers

1---2license: mit3tags:4- binary-neural-network5- zero-tokenization6- wire-speed-learning7- bit-level8- byte-level9language:10- en11pipeline_tag: text-generation12---13 14# Binary Transformers: Learning Language from Raw Binary15 16**Zero-tokenization transformers that learn directly from network bytes, bits, and beyond.**17 18This repository contains four novel transformer architectures exploring the limits of minimal vocabulary learning:19 20| Model | Vocab | Input | Weights | Description |21|-------|-------|-------|---------|-------------|22| **Byte-level** | 256 | bytes (0x00-0xFF) | real | One token per byte value |23| **Bit-level** | 2 | bits (0, 1) | real | Pure binary, 8 tokens per byte |24| **Dibit** | 4 | dibits (00,01,10,11) | real | 2-bit tokens, 4 per byte |25| **Pure Binary** | 2 | bits (0, 1) | **binary (-1/+1)** | BITS ALL THE WAY DOWN |26 27## Why?28 29Traditional LLMs use tokenizers (BPE, SentencePiece) with 32k-256k vocabulary. This creates:30- Tokenizer overhead and complexity31- Language/domain bias baked into vocabulary32- Preprocessing bottleneck33 34**What if we eliminated tokenization entirely?**35 36These models learn directly from raw binary data - no tokenizer, no preprocessing, just bytes flowing into neural networks. The ultimate goal: **wire-speed learning** where models absorb network traffic in real-time.37 38## Results (Live Experiments - 16 Jan 2026)39 40### Byte-Level (vocab=256)41```42Data: 350KB web crawl43BPB: 4.68 (vs 8.0 random = 41% compression)44Speed: 8.7 KB/s learning rate45Params: 0.6M46```47Learns HTML structure, XML tags, timestamps from raw bytes.48 49### Bit-Level (vocab=2)50```51Data: 550KB52Entropy: 1.008 bit/bit (vs 1.0 random = 0.8% compression)53Speed: 0.7 KB/s54Params: 85M55```56Pure binary learning - discovers byte boundaries and ASCII from 0s and 1s.57 58### Dibit (vocab=4: 00,01,10,11)59```60Data: 437KB61BPB: 7.55 (vs 8.0 random = 5.7% compression)62Speed: 0.25 KB/s63Params: 37.8M64```652-bit tokens provide 2x context efficiency vs bit-level. **Best compression so far!**66 67### Pure Binary (vocab=2, binary weights)68```69Data: 806KB70Entropy: 0.995 bit/bit (0.5% compression)71Binary params: 99.8%72Params: 4.7M73```74**BITS ALL THE WAY DOWN** - input bits, binary weights (-1/+1), output bits. 75On specialized hardware, this enables XNOR+popcount operations instead of multiply-accumulate.76 77## Architecture78 79All models use standard transformer architecture with:80- Causal self-attention81- GELU activation82- LayerNorm83- AdamW optimizer84- Straight-Through Estimator (STE) for binary weight gradients85 86### Key Innovation: Online Learning87 88Unlike traditional batch training, these models learn from streaming data:89- Micro-batches (32-512 tokens)90- Single-pass, no data curation91- Real-time network stream compatible92 93## Usage94 95### Byte-Level96```bash97# Pipe any data source98cat data.bin | python byte_trainer.py99curl -s http://example.com | python byte_trainer.py100zcat crawl.jsonl.gz | python byte_trainer.py101```102 103### Bit-Level104```bash105cat data.bin | python bit_trainer.py106```107 108### Dibit (2-bit tokens)109```bash110cat data.bin | python dibit_trainer.py111```112 113### Pure Binary (binary weights)114```bash115cat data.bin | python purebit_trainer.py116```117 118## Configuration119 120Edit the CONFIG dict in each trainer:121 122```python123CONFIG = {124    "d": 256,      # embedding dimension125    "layers": 6,   # transformer layers126    "heads": 8,    # attention heads127    "vocab": 2,    # vocabulary size128    "ctx": 2048,   # context length129}130```131 132## Files133 134```135byte_trainer.py    # Vocab=256, one token per byte136bit_trainer.py     # Vocab=2, pure bits137dibit_trainer.py   # Vocab=4, 2-bit tokens (00,01,10,11)138purebit_trainer.py # Vocab=2 + binary weights (-1/+1)139```140 141## Insights142 1431. **Byte-level is sweet spot** - 256 vocab captures ASCII structure efficiently while eliminating tokenizer overhead144 1452. **Bit-level works but slow** - 8x longer sequences mean 8x less context per forward pass146 1473. **Dibit balances** - 2-bit tokens give 2x context vs bit-level while staying "pure binary"148 1494. **Binary weights viable** - 99.8% binary params learn almost as well as real weights, enabling massive hardware speedups150 1515. **HTML is natural SFT** - Web data contains instruction-following patterns: `<h3>Question</h3><p>Answer`, `<dt>Term</dt><dd>Definition</dd>`, JSON Q&A152 153## Future Work154 155- Scale to billions of parameters156- Custom CUDA kernels for binary ops (XNOR + popcount)157- FPGA/ASIC implementation for true wire-speed learning158- Hierarchical binary models (bit → byte → word emergence)159 160## Citation161 162```bibtex163@misc{opentransformer2026binary,164  title={Binary Transformers: Learning Language from Raw Binary},165  author={OpenTransformer},166  year={2026},167  publisher={HuggingFace},168  url={https://huggingface.co/OpenTransformer/binary-transformers}169}170```171 172## License173 174MIT175 176## Acknowledgments177 178Built with PyTorch. Trained on vast.ai GPU instances. Part of the AGILLM research project.179