1Amogh212/GPT-From-Scratch

GPT-2 From Scratch: Serving API & Next.js Playground

     

This project is a decoder-only transformer language model built entirely from scratch using PyTorch primitives, complete with a REST serving layer and a Next.js chat interface.

Built from scratch — no `transformers` library, no high-level wrapping. Features KV-Caching optimizations, LoRA hot-swapping, and grounded RAG search.

1. System Architecture

The project decouples the Next.js frontend from the compute-heavy FastAPI model server.

graph TD
    User([User Browser])
    subgraph Frontend [Next.js UI - Vercel]
        UI[Chat Interface<br/>Persona Management]
    end
    subgraph Backend [FastAPI - Docker]
        API[FastAPI Server<br/>SSE Streaming & LoRA Routing]
    end
    subgraph Engine [Inference Layer]
        GPT2[Custom GPT-2 Engine<br/>406M Params]
        Search[Web Search<br/>Serper.dev API]
    end

    User -->|Interacts| UI
    UI -->|REST / SSE| API
    API -->|Context & Tokens| GPT2
    API -->|Retrieval| Search

2. Model & Inference Capabilities

KV-Caching (Performance)

Standard transformers recalculate attention (Query, Key, Value) for all past tokens at every step ($O(t^2)$ latency). We implemented a Key-Value Cache that saves $K,V$ tensors for past tokens, passing only the single newest token into the model ($O(t)$ latency).

graph LR
    subgraph Pre-fill Phase
        P[Prompt] -->|Q,K,V Projection| Attn1[Compute Attention]
        Attn1 --> C1[(Store K,V in Cache)]
    end
    subgraph Decode Phase (Step t)
        T[New Token t] -->|Q,K,V Projection| Attn2[Compute Attention]
        C2[(Load K,V from Cache)] --> Attn2
        Attn2 --> C3[(Append new K,V to Cache)]
    end
    C1 -.-> C2

Honest Benchmarks (Local CPU):

• —GPT-2 Small (124M): ~21.4 tokens/second
• —GPT-2 Medium (406M): ~8.6 tokens/second

Dynamic LoRA Adapters

The backend supports hot-swapping LoRA (Low-Rank Adaptation) adapters at runtime without reloading the base model — used for the SFT instruction-tuning adapters (sft_v1_small/sft_v1_medium) and for adapters trained on-demand via Teach Mode (/finetune). Pick an adapter from the Settings panel to activate it against the live model.

Personas (Prompt-Based)

Personas (Socrates, Einstein, Shakespeare) are prompt-engineering presets, not separate fine-tuned models — there are no persona-specific LoRA adapters. Selecting one applies a style-framing instruction prepended to the prompt plus a matching sampling-parameter preset (temperature, penalties, web search on/off). Quality depends on the base/SFT model's ability to follow the framing instruction, not on dedicated persona training.

Grounded RAG Generation

When web search is enabled, the API:

1. 1.Queries Serper.dev for live snippets.
2. 2.Ranks and deduplicates snippets based on keyword overlap.
3. 3.Pre-pends the snippets as context.
4. 4.Safety Net: Computes extractive overlap on the generated answer; if overlap is near zero (hallucination), it prepends a direct quote from the sources.

3. Project Structure & Testing

The system is covered by a test suite (pytest) comprising 44 passing integration and unit tests.

GPT-PRODUCTION-LEVEL/
├── app/                  # FastAPI server, inference engine, RAG search
├── model/                # PyTorch primitives (attention, layers, lora, gpt)
├── frontend/             # Next.js App Router (React)
├── data/                 # Datasets & tokenization utilities
├── training/             # Pre-training and LoRA fine-tuning scripts
├── tests/                # 44 unit & integration tests
└── checkpoints/          # Base models and adapter states

4. Setup & Running

Prerequisites: Python 3.10+, Node.js 18+

Backend (FastAPI)

# Setup and activate virtual environment
python -m venv venv
venv\Scripts\activate  # Windows

# Install requirements
pip install -r requirements-dev.txt

# Start FastAPI serving backend
python -m uvicorn app.api:app --reload --port 8000

Frontend (Next.js)

cd frontend
npm install
npm run dev

5. Limitations & Reality Check

While this is a robust system, it is built for educational/portfolio purposes and is not a replacement for commercial LLMs:

• —CPU Bottleneck: The backend currently targets CPU deployment (e.g. Hugging Face free tier). Real-world systems run on GPUs via Triton/vLLM.
• —Model Size: 406M parameters is very small. It struggles with complex logical reasoning without RAG grounding.
• —Batching: The FastAPI implementation handles requests sequentially or via threads. It lacks Continuous Batching (iteration-level scheduling) required for FAANG-scale throughput.
• —Generation Quality: The custom LoRA finetuning on Cosmopedia text introduces style shifts but does not eliminate hallucinations entirely.

👨‍💻 About the Author

Amogh Samadhiya — Backend & MLOps Engineer

Final-year B.Tech student specializing in ML Systems, Distributed Systems, and Production MLOps.

Connect & Collaborate:

• —📧 Email: amoghsamadhiya779@gmail.com
• —🔗 LinkedIn: amogh-samadhiya
• —💼 Availability: Open to remote Backend / ML Engineering internships and opportunities.