darioooooo0o/spark-1.7b-engram

💻 DeepSeek Engram (Coding-Specialized) on Spark-X2.5-1.7B


Requests, questions or suggestions? Message me on X: https://x.com/imdariotoo

⚠️ EXPERIMENTAL RESEARCH ARTIFACT: This repository contains an experimental, code-specialized implementation of Conditional Memory via Scalable Lookup (Engram) (arXiv:2601.07372) trained on top of a 100% frozen Spark-X2.5-1.7B model.

📥 Direct Downloads & Quick Access

📌 Overview & Multi-Language Scope

This model explores how O(1) conditional memory lookup can augment a frozen compact language model for multi-language code generation and algorithmic reasoning:

• —Polyglot Code Memory Bank: The 131k-slot coprime hash tables act as an external memory bank storing multi-token syntax patterns, recursion templates, and standard library idioms across Python, C++, Java, JavaScript, C#, SQL, Bash, and Rust.
• —100% Frozen Backbone (Zero Drift): The base 1.7B transformer weights remain completely frozen. No weights in Spark-1.7B were modified, fine-tuned, or adapted with LoRA.
• —Preserved General Reasoning: Because the base model remains untouched, the model experiences zero catastrophic forgetting of its original conversational or non-coding skills.

📚 Training Corpus (25,000,000 Multi-Language Tokens)

The Engram module was trained across 25M tokens sourced from diverse programming datasets:

1. 1.`nickrosh/Evol-Instruct-Code-80k-v1` (~12M tokens, Multilingual):
2. 2.Complex algorithmic problems, data structures, and competitive programming across Python, C++, Java, JavaScript, C#, Bash, PHP, and SQL.
3. 3.`sahil2801/CodeAlpaca-20k` + `iamtarun/python_code_instructions` (~3M tokens):
4. 4.Multi-language task completions, idiomatic one-liners, standard library manipulations, and docstring-to-code implementations.
5. 5.`codeparrot/codeparrot-clean` (~10M tokens):
6. 6.Clean real-world repository code, modular package architectures, class hierarchies, and production syntax.

🏗️ Architecture Specifications

Following DeepSeek's paper, the memory module is structured as follows:

📊 Benchmark & Training Results

1. Training & Loss Trajectory (Held-out 1M Validation Shard)

• —Step 0 Baseline Val Loss: 1.5396 (Perplexity: 4.66)
• —Final Val Loss (Step 1526, 25M tokens): `1.2275` (Perplexity: 3.41)
• —Net Improvement: -0.3121 loss reduction (-26.8% perplexity)

2. HumanEval Pass Rate (Greedy, temperature=0.0)

• —Stock Spark-X2.5-1.7B Baseline: 7 / 20 (35.0%)
• —Spark-1.7B + Native Engram (25M Tokens): 10 / 20 (50.0%, +15.0% absolute gain)

🧩 Quantization & Deployment Compatibility

• —Weight-Only Quantization (GPTQ / AWQ / INT4 / INT8):
• —Fully Compatible: The frozen base 1.7B transformer weights can be quantized (e.g. GPTQ/AWQ to compress backbone VRAM to ~1.0 GB).
• —Because inter-layer activations (hidden states) remain in 16-bit floating point (bfloat16/float16), the Engram module (enable_engram()) plugs directly into the quantized backbone without numerical degradation.
• —The Engram weights (~1.6 GB safetensors) remain unquantized in bfloat16 / float32.
• —GGUF / llama.cpp:
• —Not yet supported in standard llama.cpp: GGUF requires custom C++/GGML tensor graph kernels for Engram's multi-head XOR hashing, prime moduli grid lookup, and Causal ShortConv. Use Hugging Face / PyTorch for Engram inference.

🚀 Quickstart & Inference

1. Requirements

pip install torch transformers safetensors

2. Loading the Model with Engram

import torch
from safetensors.torch import load_file
from transformers import AutoTokenizer
# Import architecture definitions from the repository
from configuration_spark import Spark2_5Config
from modeling_spark import Spark2_5ForCausalLM

device = "cuda" if torch.cuda.is_available() else ("xpu" if hasattr(torch, "xpu") and torch.xpu.is_available() else "cpu")
dtype = torch.bfloat16

# 1. Load Base Model
model_id = "XHToken/Spark-X2.5-1.7B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = Spark2_5ForCausalLM.from_pretrained(model_id, torch_dtype=dtype).to(device)

# 2. Inject Engram Memory at Layers (2, 14)
model.enable_engram(
    target_layers=(2, 14),
    mem_dim=64,
    num_heads=8,
    slots_per_head=131072,
    orders=(2, 3, 4),
    kernel_size=4,
    dilation=2,
    device=device,
    dtype=dtype,
)

# 3. Load Trained Engram Weights
weights = load_file("engram_layer_weights_spark1.7b_25m.safetensors")
for k in weights:
    weights[k] = weights[k].to(device)
model.load_state_dict(weights, strict=False)
model.eval()

# 4. Generate Python Code
prompt = "def greatest_common_divisor(a: int, b: int) -> int:\n    \"\"\" Return a greatest common divisor of two integers a and b \"\"\"\n"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.0)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

📖 Citation & References

@article{deepseek2026engram,
  title={Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models},
  author={DeepSeek-AI and Peking University},
  journal={arXiv preprint arXiv:2601.07372},
  year={2026}
}