hakatu/fpga-whale-training

FPGA-Whale: FPGA-Friendly NanoWhale Architecture

This repository contains an FPGA-friendly reimplementation of the HuggingFaceTB/nanowhale-100m-base model, designed for deployment on Xilinx FPGAs.

Architecture Changes

Based on literature research for FPGA deployment (QFX arxiv:2401.17544, BitNet b1.58 Reloaded arxiv:2407.09527):

1. 1.Removed MoE → Dense SwiGLU FFN (no sparse routing, deterministic memory access)
2. 2.Removed Hyper-Connections → Standard residual connections (simpler dataflow)
3. 3.BitLinear layers → Ternary weights {-1, 0, +1} with INT8 activations
4. 4.Multiplication-free inference on FPGA (ternary × int8 = add/sub/nop)
5. 5.Quantization-aware training with straight-through estimator
6. 6.LayerNorm instead of RMSNorm → More FPGA-friendly normalization
7. 7.Kept MLA → Low-rank Q projection for KV-cache efficiency

Training

Knowledge distillation from the original NanoWhale teacher model with QAT on FineWeb-Edu dataset.

python train.py \
  --teacher_model HuggingFaceTB/nanowhale-100m-base \
  --hub_model_id hakatu/fpga-whale-100m \
  --distill_alpha 0.7 \
  --temperature 2.0 \
  --num_train_samples 50000 \
  --max_seq_length 512 \
  --bf16

FPGA Deployment Path

1. 1.Train this model with QAT to convergence
2. 2.Extract ternary weights and INT8 activation scales
3. 3.Export to ONNX / hls4ml format
4. 4.Synthesize with Xilinx Vitis HLS / Vivado

References

• —Dai et al., "Trainable Fixed-Point Quantization for Deep Learning Acceleration on FPGAs", arxiv:2401.17544
• —Nielsen & Schneider-Kamp, "BitNet b1.58 Reloaded", arxiv:2407.09527
• —NanoWhale: https://huggingface.co/HuggingFaceTB/nanowhale-100m-base