prompterminal/nanogpt-enwik8-compressed

NanoGPT enwik8 - Compressed Model

Compressed nanoGPT model trained on enwik8 (Wikipedia) using low-rank matrix decomposition.

Model Details

• —Original Parameters: 28,801,536
• —Compressed Parameters: 22,755,840
• —Compression Ratio: 1.27× smaller
• —Compression Method: Low-rank decomposition (rank=16) on layers [5, 6, 7]
• —Training Data: enwik8 (Wikipedia, first 100MB)
• —Vocabulary: 6,060 characters
• —Context Length: 1024 tokens

Performance

• —Original Perplexity: 8843.82
• —Compressed Perplexity: 7387.50
• —Performance Change: -16.5%

Usage

⚠️ Note: This model requires custom code for text generation due to character-level tokenization.

# This model is designed for research and benchmarking
# Custom generation code required

Compression Technique

Uses SVD-based low-rank approximation:

• —Method: Decompose weight matrices W ≈ U × V
• —Rank: 16 (much smaller than original dimensions)
• —Layers: Compressed MLP layers in transformer blocks [5, 6, 7]

Evaluation

Ready for benchmark evaluation including:

• —Nous benchmark suite (AGIEval, GPT4ALL, TruthfulQA, Bigbench)
• —Compression technique analysis
• —Character-level language modeling research

Citation

Based on nanoGPT by Andrej Karpathy. Compression technique demonstrates effective neural network compression with minimal performance impact.