DexterSptizu/Mamba-7B-Reasoning

Mamba-7B-Reasoning: Instilling Chain-of-Thought (<think>) Reasoning into Selective State Space Models

This repository contains the model card, source code, evaluation benchmarks, presentation slides, and fine-tuning scripts for Mamba-7B-Reasoning.

Fine-tuned on 2x NVIDIA H200 NVL GPUs using LoRA adapter configuration ($r=16, lpha=32$) on Mamba linear projection layers (in_proj, x_proj, dt_proj) using bfloat16 precision.

🎯 Primary Project Highlights & Proof of Work

1. 1.Architecture Shift: Fine-tuned Mamba's linear projection layers using LoRA with bfloat16 precision across 2x NVIDIA H200 NVL GPUs.
2. 2.Dataset Pipeline: Processed 16,710 DeepSeek-R1 distilled reasoning samples (`BespokeLabs/Bespoke-Stratos-17k`) into structured <think>...</think> CoT conversation format.
3. 3.50-Prompt Empirical Evaluation: Evaluated Base Mamba 7B vs. Fine-Tuned Mamba Reasoning across 50 technical benchmarks spanning Math Logic, Systems Code, Cryptography, and AI Theory.
4. 4.Key Finding: Achieved 100% `<think>` CoT trigger rate with a 1.85x content density expansion while maintaining Mamba's constant $O(1)$ memory state and sub-4-second response latency.

📂 Uploaded Artifacts & Project Inventory

• —docs/50_PROMPTS_MAMBA_BASE_VS_REASONING_COMPARISON.md: 215 KB Side-by-Side 50-Prompt Evaluation Report.
• —docs/MAMBA_FINETUNING_PROOF_OF_WORK_PRESENTATION.pptx: First-Person Proof of Work PowerPoint Deck.
• —docs/MAMBA_REASONING_FINETUNING_PLAN.md: Fine-Tuning Strategy & Implementation Plan.
• —src/: Complete PyTorch, PEFT, Dataset Processing & Evaluation Source Code.
• —data/50_prompts_reasoning_results.json: Raw Evaluation Transcripts and Execution Logs.