SurendraVB/Synaptic-Edge-Tokenizer-65K

Synaptic Edge 65,536 ByteLevel BPE Tokenizer

The official 65,536-vocabulary ByteLevel BPE Tokenizer custom-engineered for the Synaptic Edge 150M & 1B Neuromorphic Foundation Models.

Trained on over 1.6+ Billion tokens of source code, AI system kernels, formal mathematics, and multilingual texts.

Key Highlights

• —Exact $2^{16}$ Alignment: Vocabulary size is strictly 65,536, ensuring exact power-of-2 alignment with GPU warp registers and Triton megakernel tensor cores.
• —Zero OOV / Zero `<unk>`: ByteLevel fallback guarantees that every byte of UTF-8 text can be processed losslessly without out-of-vocabulary errors.
• —Code & Kernel Optimized (~70% Mixture):
• —Multi-space indentation tokens ( , , , \t) eliminate token explosion in nested code blocks.
• —Pre-seeded with Andrej Karpathy's nanoGPT/llm.c, OpenAI Triton megakernels, BitNet 1.58b STE, and Lean 4 formal proofs.
• —Delivers a ~2.2× compression ratio improvement over standard NLP tokenizers on Python, C++, Rust, and CUDA.
• —Structural Repo-Level Scoping: Built-in first-class scoping tags (<repo>, </repo>, <file>, </file>, <think>, </think>) for continuous multi-file repository pretraining.
• —1,000,000-Token Native Context: Calibrated for ultra-long context streams paired with 3D Bit-RoPE and Continual Dynamic Synaptic Plasticity.

Domain Mixture Breakdown

Quickstart

1. Using Hugging Face transformers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('SurendraVB/Synaptic-Edge-Tokenizer-65K')

code_snippet = '''@triton.jit
def fused_kernel(a_ptr, b_ptr, BLOCK_M: tl.constexpr):
    pass'''

tokens = tokenizer(code_snippet)
print('Token IDs:', tokens.input_ids)
print('Decoded:', tokenizer.decode(tokens.input_ids))

2. Using Fast Rust tokenizers

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained('SurendraVB/Synaptic-Edge-Tokenizer-65K')
output = tokenizer.encode('def forward(self, x: torch.Tensor):\n    return x * 2')
print('Subwords:', output.tokens)

Special Scoping & Indentation Tokens

Citation & Architecture

@misc{synaptic_edge_2026,
  author = {Surendra V B},
  title = {Synaptic Edge: 1-Bit Ternary Continual Learning Neuromorphic Foundation Architecture},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/SurendraVB/Synaptic-Edge-Tokenizer-65K}}
}