Mostafa8Mehrabi/deepseek-v3-mini

DeepSeek-V3 Mini (181,320,192 Parameters) - COMPLETE FIXED

This is a COMPLETE FIXED custom implementation of DeepSeek-V3 Mini with exactly 181,320,192 parameters.

✅ Complete Fixes Applied

• —✅ Proper Weight Tying: Input embeddings and output head share weights (embed_tokens ↔ lm_head)
• —✅ Consistent Parameter Count: 181,320,192 parameters maintained through upload/download
• —✅ No Parameter Duplication: Weight tying prevents embedding parameter doubling
• —✅ Fixed Attention Masks: No more attention mask warnings
• —✅ Proper Token Configuration: pad_token_id ≠ eos_token_id (pad: 50255, eos: 50256)
• —✅ Verified Architecture: All components properly initialized and connected

Model Details

• —Architecture: DeepSeek-V3 with Multi-Head Latent Attention (MLA)
• —Parameters: 181,320,192 (with proper weight tying)
• —Hidden Size: 768
• —Layers: 12
• —Attention Heads: 12
• —Vocabulary: 50,257 tokens
• —Precision: FP16 optimized
• —Weight Tying: ✅ Enabled and verified
• —Attention Masks: ✅ Properly handled, no warnings

Key Features

• —✅ Multi-Head Latent Attention (MLA) for memory efficiency
• —✅ Multi-Token Prediction (MTP) for improved training
• —✅ SwiGLU activation function
• —✅ RoPE positional encoding
• —✅ COMPLETE FIX: All known issues resolved

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model (all fixes automatically applied)
model = AutoModelForCausalLM.from_pretrained(
    "Mostafa8Mehrabi/deepseek-v3-mini",
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("Mostafa8Mehrabi/deepseek-v3-mini")

# Verify fixes applied
param_count = sum(p.numel() for p in model.parameters())
tied = torch.equal(model.embed_tokens.weight, model.lm_head.weight)
no_mask_warnings = tokenizer.pad_token_id != tokenizer.eos_token_id

print(f"Parameters: {param_count:,}")  # Should show 181,320,192
print(f"Weight tying: {tied}")          # Should show True
print(f"Attention masks fixed: {no_mask_warnings}")  # Should show True

# Generate text (no warnings)
inputs = tokenizer("The future of AI is", return_tensors="pt", return_attention_mask=True)
outputs = model.generate(**inputs, max_length=50, do_sample=True, temperature=0.7)
text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(text)

Fixes Summary

Parameter Count Issue ✅ FIXED

• —Before: 219M parameters (weight tying broken)
• —After: 181,320,192 parameters (weight tying working)
• —Solution: Proper weight tying from model initialization

Attention Mask Warnings ✅ FIXED

• —Before: "attention mask is not set and cannot be inferred"
• —After: No warnings, proper mask handling
• —Solution: pad_token_id (50255) ≠ eos_token_id (50256)

Upload/Download Consistency ✅ FIXED

• —Before: Parameter count changed between upload and download
• —After: Identical parameter count maintained
• —Solution: Proper state dict handling with weight tying preservation

Technical Implementation

Built with custom PyTorch implementation featuring:

• —Optimized MLA attention mechanism (~93.3% memory reduction vs standard attention)
• —Efficient KV compression with LoRA (rank=192)
• —Multi-token prediction capability (2 heads)
• —FP16 training ready
• —COMPLETE FIX: All known issues resolved

Architecture Summary

Embeddings: 50,257 × 768 = 38,597,376 params (shared with output)
Transformer: 12 layers × ~11,893,504 params/layer
Output Head: Shared with embeddings (0 additional params due to tying)
Total: 181,320,192 parameters

Verification

All issues have been completely resolved:

• —✅ Parameter count: 181,320,192 (consistent)
• —✅ Weight tying: Enabled and working
• —✅ Attention masks: No warnings
• —✅ Token configuration: Proper separation of pad/eos tokens
• —✅ Upload/download: Consistent behavior

Model created and uploaded by Mostafa8Mehrabi COMPLETE FIXED version - all known issues resolved