TaewoooPark/SpikeGPT-216M-Mixture-Finetune

SpikeGPT-216M-Mixture-Finetune

This repository contains a full-parameter fine-tuned version of `ridger/SpikeGPT-OpenWebText-216M`. Training consists of mixture SFT v0.7 followed by the lower-learning-rate v0.8 continuation. The released checkpoint is v0.8.

Model identity

Recorded same-split evaluation

The validation split is fixed across all three checkpoints: 965 examples, 54,429 supervised assistant tokens, JSONL SHA-256 23f963aa7ce2038d6c0f104e01ab37a0613de14f6f7b6f5aebb9a2f2c1c670b5. Loss is mean assistant-token cross-entropy and perplexity is exp(loss).

For base → v0.8, exp(-loss) changes from 2.248% to 4.147% (1.8448×), and the cross-entropy difference is 0.88347 bits per supervised token.

Training data

The deterministic mixture manifest SHA-256 is 7378939249535ff604e9eed66acea7414a8a35e63deb46ac649ce90a7bc32b52. It has 12,736 scheduled training occurrences, 9,934 unique source rows, and 1,166,515 supervised tokens per epoch. v0.7 processed two epochs (2,333,030 supervised-token exposures) in 3,184 optimizer steps. v0.8 processed one additional epoch (1,166,515) in 1,592 additional steps. Total exposure is 3,499,545 supervised tokens.

Validation contains 965 examples and 54,429 supervised tokens: 500 Smol test rows, 50 OASST1 validation rows, and 415 repository-authored CC0 validation rows. Complete sequences fit the 512-token context; response truncation is disabled. The combined mixture has component-scoped terms and the aggregate SPDX field NOASSERTION.

The CC0 component source has 1,778 generated training rows; the deterministic mixture selects 1,000 of them and includes all 415 validation rows.

The pinned Smol generation pipeline identifies Qwen/Qwen2.5-72B-Instruct and requires the notice Improved using Qwen. The copied pipeline source is huggingface/smollm@951394e9b214ce91e3223b2257a8eecb0a0d3d4d, path text/data/smoltalk/constraints/pipeline/pipeline.py.

Training configuration and runtime

The optimizer is AdamW (beta1=0.9, beta2=0.95, eps=1e-8). Microbatch is 1, gradient accumulation is 8, weight decay is 0.01, gradient clipping is 1.0, seed is 20260824, hardware is NVIDIA GB10, and the recorded runtime is PyTorch 2.13.0+cu130 with CUDA 13.0 and CuPy SNN backend.

Weight deltas

All 330 float32 state tensors were compared. Accumulations use float64. changed is exact float32 inequality; relative L2 is ||target - source||₂ / ||source||₂ and is displayed as a percentage.

Base → v0.8 parameter RMS changes from 0.3611604 to 0.3611292. Complete values and checkpoint hashes are in weight-delta.json.

Runtime format

The checkpoint uses the custom recurrent SpikeGPT runtime in runtime/. config.json is descriptive metadata and does not define a Transformers AutoModel implementation. Token 0 is EOS/conversation separator; token 1 is padding. Training serialization is <|endoftext|>User: ... Assistant: ...<|endoftext|>; loss is applied to assistant response and EOS tokens.

Files and integrity

• —model.safetensors: 26ce68619b45b9fbd83257630da23039188ab189161062ffbfd997888e405bff
• —model.pth: 1256a913e566bc6247c73127e7c4ae2348f8fb034b1cf5de13315713ef083471
• —20B_tokenizer.json: 56ac4821e129d2c520fdaba60abd920fa852ada51b45c0dd52bbb6bd8c985ade
• —training-lineage.json: base → v0.7 → v0.8 artifact and checkpoint links
• —weight-delta.json: full-state numerical delta report
• —eval_before.json, eval_after.json, metrics.jsonl: v0.8 continuation records
• —training/phase1/: sanitized v0.7 run, config, evaluation, and metrics records
• —dataset_manifest.json, components/: split hashes, component manifests, license/card/pipeline evidence, Qwen license, and notices
• —source-snapshot.tar.gz, source-snapshot-manifest.json, runtime/: exact v0.8 source/runtime snapshot
• —release_manifest.json: SHA-256 and byte size for all preceding release files

Licenses and attribution

• —Fine-tuned weights: Apache-2.0; see LICENSE.
• —Base weights: ridger/SpikeGPT-OpenWebText-216M@4039295cca3da1df0e5871f4bc7727b227496132. Pinned model-card metadata declares Apache-2.0; the pinned repository has no standalone LICENSE file.
• —OASST1: Apache-2.0; see components/oasst1/OASST1-LICENSE.
• —Repository-authored calibration corpus: CC0-1.0; see components/cc0_calibration/SOURCE.md.
• —Smol constraints: pinned card metadata says apache-2.0; the pinned dataset revision has no standalone dataset license file. Evidence, generation-pipeline source, QWEN-LICENSE, and SOURCE-NOTICE.md are copied under components/smol_constraints/.
• —Qwen notice: Improved using Qwen. Qwen is licensed under the Qwen LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved.
• —SpikeGPT source: BSD-2-Clause; see CODE-LICENSE-BSD-2-Clause.
• —RWKV-derived code, CUDA sources, and tokenizer provenance: Apache-2.0; see THIRD-PARTY-LICENSE-RWKV-Apache-2.0. Upstream commit: 9917078f936e1f45bff341cc58550fd03ec4e5eb.
• —SmolLM generation-pipeline code: Apache-2.0; see components/smol_constraints/SMOLLM-PIPELINE-LICENSE. This license copy applies to the pipeline code only.
• —Vendored and modified SpikingJelly 0.0.0.0.12: Open-Intelligence Open Source License V1.0; see THIRD-PARTY-LICENSE-SpikingJelly and THIRD-PARTY-LICENSE-SpikingJelly-CN. Upstream commit: 1171f5249a4ebeace6ab8d6a74d85579fafb93ed; PyPI wheel SHA-256: ffa5298947348d5928840a077dc2840c3f411b192401e1bdc4f2224883a8a51a. Section V disclosure URL: https://www.aitisa.org.cn/OI_Statement/.
• —Paper: Zhu et al., *SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks*.