kussssh/IPO-Analyzer
0
1# IPO Analysis Platform — Full Architecture & Interview Guide2 3---4 5## WHAT IS THIS PROJECT? (Simple One-Liner)6 7> An AI-powered platform that reads 400+ page IPO documents (called DRHP/RHP) and gives you a clear investment analysis — like having a professional analyst in your pocket.8 9---10 11## THE BIG PICTURE — HOW IT WORKS END TO END12 13```14User uploads DRHP PDF15 ↓16[Step 1] PDF Parser (PyMuPDF4LLM) → extracts text page by page17 ↓18[Step 2] Chunker → splits text into smart, overlapping pieces19 ↓20[Step 3] Embeddings (sentence-transformers) → converts text to numbers (vectors)21 ↓22[Step 4] ChromaDB → stores those vectors on disk (Vector Database)23 ↓24[Step 5] 8 Analysis Modules run in parallel25 Each module:26 → sends queries to ChromaDB (dense search) + BM25 (keyword search)27 → combines results (Ensemble Retriever)28 → reranks them (CrossEncoder)29 → sends top chunks to GPT-4o-mini with a structured prompt30 → gets structured JSON output via Pydantic schema31 ↓32[Step 6] Final Scorecard module combines all 8 signals33 ↓34[Step 7] JSON result saved to DB + shown on Next.js frontend35```36 37---38 39## THE FILE-BY-FILE EXECUTION PIPELINE (For Interviews)40 41*If an interviewer asks: "Walk me through the exact code execution when a user clicks 'Analyze'. Which file runs first, and what happens next?" — This is your answer.*42 43### 1. The Trigger (Frontend)44- **File:** `frontend-next/app/analytics/[company]/page.tsx`45- **What happens:** The user clicks the "Analyze" button. The React frontend opens a **WebSocket** connection to the backend to get real-time progress updates (so the UI doesn't freeze during the long analysis). While waiting, the frontend shows the 3D Brain Loader (`components/BrainLoader.tsx`).46 47### 2. The API Gateway (Backend Entry Point)48- **File:** `backend/api.py`49- **What happens:** The FastAPI server receives the WebSocket connection at the `@app.websocket("/analyze/ws/{company_id}")` endpoint. It immediately hands the job over to the central orchestrator file.50 51### 3. The Orchestrator (The Master Controller)52- **File:** `backend/company_pipeline.py`53- **What happens:** This is the brain of the backend. It controls the entire flow:54 1. **Check Cache:** It checks SQLite (`database.py`) if this company has already been analyzed. If yes, it returns it instantly.55 2. **Prepare Document:** If not, it calls `_prepare_document()`.56 57### 4. Document Preparation (Parsing & Chunking)58- **Files:** `backend/parser.py` and `backend/chunker.py`59- **What happens:** 60 - `parser.py` downloads the DRHP PDF and uses PyMuPDF4LLM to extract text from all 400+ pages.61 - `chunker.py` takes that massive text and slices it into smart, overlapping chunks (separating tables, financials, and narrative text).62 63### 5. Vector Database Setup (Embedding)64- **Files:** `backend/retriever.py` and `backend/embeddings.py`65- **What happens:** 66 - `retriever.py` takes all the chunks and sends them to `embeddings.py`.67 - `embeddings.py` uses the local HuggingFace model (`all-MiniLM-L6-v2`) to convert text into numbers (vectors) in **parallel batches of 100**.68 - `retriever.py` then saves these vectors into **ChromaDB** on the local disk so they can be searched quickly later.69 70### 6. Parallel AI Analysis (The Core AI Engine)71- **Files:** `backend/company_pipeline.py` → `backend/modules/*.py`72- **What happens:** Once the database is ready, `company_pipeline.py` runs the `_run_modules_parallel()` function. 73 - It fires off **7 asynchronous tasks at the exact same time** (using `asyncio.gather`).74 - Each task triggers a specific module file: `business.py`, `financials.py`, `risks.py`, `valuation.py`, etc.75 76### 7. Inside a Single Module (Retrieval + LLM)77- **Files:** `backend/modules/base.py`, `backend/retriever.py`, `backend/llm.py`, `backend/schema.py`78- **What happens inside every module:**79 1. The module asks `retriever.py` to search ChromaDB for specific topics (e.g., "Find all text about revenue").80 2. `retriever.py` returns the top 25 chunks using Hybrid Search (Vector + BM25 keyword) and reranks them.81 3. The module passes those chunks + a specific prompt to `llm.py` (GPT-4o-mini).82 4. `llm.py` returns a structured JSON response strictly matching the Pydantic data models defined in `schema.py`.83 84### 8. The Final Verdict85- **Files:** `backend/modules/scorecard.py` and `backend/database.py`86- **What happens:** Once all parallel modules finish, `company_pipeline.py` sends their results to `scorecard.py`. It tallies the positive/negative signals and generates a final "BUY / AVOID" verdict. The massive final JSON is saved to SQLite via `database.py`.87 88### 9. Back to the User89- **Files:** `backend/api.py` → `frontend-next/app/analytics/[company]/page.tsx`90- **What happens:** `api.py` sends the final JSON back through the WebSocket. The frontend React components receive the data, the 3D Brain Loader disappears, and the beautiful analysis dashboard renders on the user's screen.91 92---93 94## PART 1: TECHNOLOGY STACK95 96### Frontend97| Technology | Why Used |98|---|---|99| **Next.js 14** (App Router) | React framework for the UI, server-side rendering, routing |100| **TypeScript** | Type safety in frontend code |101| **Tailwind CSS** | Utility-first CSS for styling |102| **Three.js / WebGL** | 3D brain loader animation shown while analysis runs |103| **Vercel** | Deployment platform for the frontend |104 105### Backend106| Technology | Why Used |107|---|---|108| **Python (FastAPI)** | REST API server, handles all business logic |109| **LangChain** | Framework to connect LLM + retrievers + prompts in a pipeline |110| **ChromaDB** | Vector database to store and search document embeddings |111| **sentence-transformers** | Local HuggingFace model to create embeddings |112| **BM25** | Keyword-based sparse retrieval algorithm |113| **CrossEncoder** | Reranker model to re-sort retrieved chunks by relevance |114| **GPT-4o-mini (OpenAI)** | LLM used for final structured extraction |115| **PyMuPDF / PyMuPDF4LLM** | PDF parsing library |116| **Pydantic** | Data validation and structured output schemas |117| **SQLite** | Local database for caching, users, companies, embeddings |118| **Supabase** | Cloud PostgreSQL + file storage for production |119 120### Integrations121| Service | Purpose |122|---|---|123| **Supabase** | Auth, cloud DB, PDF storage in production |124| **Razorpay** | Payment gateway for subscriptions |125| **Airtable** | CRM — syncs user signups/logins |126| **Google OAuth** | Social login |127| **Google Analytics** | Page tracking |128 129---130 131## PART 2: DETAILED WORKFLOW (Step-by-Step)132 133### STEP 1 — PDF PARSING (parser.py)134 135The user uploads a DRHP PDF. The parser reads it using **PyMuPDF4LLM**.136 137**What PyMuPDF4LLM does:**138- Reads every page of the PDF139- Converts tables to Markdown format (pipes `|` and dashes `---`)140- Returns a list of `Document` objects — one per page141- Each Document has `page_content` (the text) and `metadata` (page number, source file)142 143**Company name extraction:**144- The parser scans the first 5 pages using regex patterns145- Looks for patterns like `"NAME OF THE COMPANY: XYZ LIMITED"`146- Uses a scoring system — labeled patterns score 100, generic ALL CAPS patterns score 60147- Falls back to "Unknown" if no match found148 149**Two parser modes:**150- `full` mode: Uses pymupdf4llm with ONNX layout analysis (better table extraction, local dev)151- `fast` mode: Uses plain PyMuPDF text extraction (faster, used in production/Supabase)152 153---154 155### STEP 2 — CHUNKING (chunker.py)156 157A 400-page PDF can have thousands of lines. You can't send the whole thing to an LLM — it's too large and expensive. So you **chunk** it into smaller pieces.158 159**What chunking does:**160Splits each page's text into overlapping windows of text.161 162**Smart chunking — 3 types detected:**163 1641. **Tables** (`chunk_type = "table"`)165 - Detected if text has `>10 pipe characters` and `---`166 - Uses chunk size: 2000 chars, overlap: 100 chars167 1682. **Financial Data** (`chunk_type = "financial_data"`)169 - Detected if number density > 15% OR contains keywords like "crore", "ebitda", "revenue"170 - Uses chunk size: 2000 chars171 1723. **Narrative Text** (`chunk_type = "narrative"`)173 - Regular paragraphs174 - Adaptive sizing: 1200–1600 chars depending on paragraph density175 - Small chunks (<900 chars) get merged to avoid tiny useless pieces176 177**Section Detection:**178Each chunk is tagged with the DRHP section it came from:179`Risk Factors`, `Business Overview`, `Financial Statements`, `MD&A`, `Valuation`, etc.180 181This uses regex patterns on the first 500 characters of each page.182 183**Chunk Metadata stored per chunk:**184```json185{186 "page": 42,187 "section": "Risk Factors",188 "chunk_type": "narrative",189 "chunk_index": 3,190 "chunk_hash": "sha256_of_content",191 "char_count": 1150192}193```194 195The `chunk_hash` is a SHA-256 hash of the normalized text — used for deduplication and embedding cache lookups.196 197---198 199### STEP 3 — EMBEDDINGS (embeddings.py)200 201**What is an Embedding?**202An embedding is a list of numbers (a vector) that represents the *meaning* of a piece of text. Similar texts have vectors that are mathematically close to each other.203 204**Example:**205- "Revenue grew by 40%" → [0.12, -0.45, 0.78, ...] (384 numbers)206- "Sales increased 40 percent" → [0.11, -0.44, 0.77, ...] (very similar!)207- "The promoter sold shares" → [0.90, 0.23, -0.55, ...] (very different)208 209**Model used:**210`sentence-transformers/all-MiniLM-L6-v2`211- A HuggingFace local model (runs on CPU, no API cost)212- Produces **384-dimensional vectors**213- Normalized vectors (length = 1) — important for cosine similarity214 215**How embeddings are created:**216```python217from langchain_community.embeddings import HuggingFaceEmbeddings218 219embedder = HuggingFaceEmbeddings(220 model_name="sentence-transformers/all-MiniLM-L6-v2",221 model_kwargs={'device': 'cpu'},222 encode_kwargs={'normalize_embeddings': True, 'batch_size': 100}223)224```225 226**Parallel Embedding (the optimization I built):**227Instead of embedding chunks one-by-one (slow), I batch them and run batches in parallel using `ThreadPoolExecutor`:228- Split all chunks into batches of 100229- Run up to 8 batches simultaneously230- 3-4x faster than sequential embedding for large documents231 232**Embedding Cache:**233Embeddings are expensive to compute. I store them in SQLite:234- Key: `(model_name, chunk_hash)`235- Value: the vector (as JSON)236- On re-analysis of the same document, cache is hit → **zero re-computation**237 238---239 240### STEP 4 — VECTOR DATABASE — ChromaDB (retriever.py)241 242**What is a Vector Database?**243A specialized database that stores vectors and can find the most similar vectors to a query vector very fast.244 245**Vector DB I use: ChromaDB**246- Open-source, runs locally (no cloud needed)247- Persists to disk (in `chroma_stores/` folder)248- Each company's DRHP gets its own Chroma "collection"249 250**Search Algorithm ChromaDB uses: HNSW**251 252HNSW = **Hierarchical Navigable Small World**253 254Simple explanation:255- Imagine a graph where each vector is a node256- HNSW builds multiple layers of this graph (like a highway system)257- Top layers have few nodes with long connections (highways)258- Bottom layers have all nodes with short connections (local roads)259- To find similar vectors, you start at the top (few candidates) and zoom in260 261**Why HNSW?**262- Approximate Nearest Neighbor (ANN) — not exact but >99% accurate263- Extremely fast — O(log n) search time264- Much faster than brute-force cosine similarity at scale265 266**MMR Search (Maximal Marginal Relevance):**267I use MMR instead of plain similarity search:268```python269vectorstore.as_retriever(270 search_type="mmr",271 search_kwargs={"k": 25, "fetch_k": 50}272)273```274MMR fetches 50 candidates, then picks 25 that are:275- Relevant to the query276- But also *diverse* from each other (avoids returning 25 nearly identical chunks)277 278---279 280### STEP 5 — HYBRID RETRIEVAL (The RAG Engine)281 282**What is RAG?**283RAG = **Retrieval-Augmented Generation**284 285Instead of asking GPT-4o-mini to memorize all financial data (impossible), we:2861. **Retrieve** the most relevant text chunks from the document2872. **Augment** the LLM prompt with those chunks as context2883. **Generate** a structured answer based only on that context289 290**My Hybrid Retrieval = Dense + Sparse:**291 292**1. Dense Retrieval (ChromaDB + HNSW)**293- Converts query to a vector using the same embedding model294- Finds the top-25 most similar chunks by cosine similarity295- Good at semantic matches ("profit increased" matches "revenue grew")296 297**2. Sparse Retrieval (BM25)**298- BM25 = Best Match 25 (a keyword-based algorithm)299- Scores documents by term frequency and inverse document frequency300- Good at exact keyword matches ("EBITDA margin", "₹450 crore")301- I built a custom `SimpleSparseRetriever` with full BM25 math (k1=1.5, b=0.75)302 303**3. Ensemble Retriever (RRF Fusion)**304Combines both with **Reciprocal Rank Fusion**:305```306score = 0.6 × (1 / (dense_rank + 60)) + 0.4 × (1 / (sparse_rank + 60))307```308Dense gets weight 0.6, sparse gets 0.4. The +60 is a smoothing constant.309 310**4. CrossEncoder Reranker**311After getting ~25 merged results, I pass them through a CrossEncoder:312- Model: `cross-encoder/ms-marco-MiniLM-L-6-v2`313- Unlike the embedding model (which encodes query and doc separately), CrossEncoder reads them **together**314- Much more accurate relevance scoring315- Reduces 25 chunks → top 14316 317**Why separate embedding model and CrossEncoder?**318- Embedding model is fast but approximate (good for initial retrieval from thousands)319- CrossEncoder is slow but accurate (good for re-scoring 25 candidates)320- This two-stage approach is called **bi-encoder + cross-encoder** pipeline321 322---323 324### STEP 6 — 8 ANALYSIS MODULES (modules/)325 326Each module is a focused analyst. They all run using the same pattern defined in `base.py`:327 328```329Module → retrieval_queries → retrieve+rerank → build context → LLM prompt → Pydantic output330```331 332| Module | What it Analyzes | Output Schema |333|---|---|---|334| M1 Business Quality | Business model, segments, strengths, risks | `BusinessQuality` |335| M2 Revenue & Profitability | Revenue, EBITDA, PAT for 3 years | `RevenueProfitability` |336| M3 Growth Quality | Revenue trend vs margin trend | `GrowthProfitabilityCheck` |337| M4 Valuation | P/E, EV/EBITDA vs peers | `ValuationAnalysis` |338| M5 Promoter & OFS | Who owns shares, who is selling | `PromoterOFS` |339| M6 Use of Proceeds | Where IPO money goes | `UseOfProceeds` |340| M7 Risk Analysis | High/Medium/Low risks categorized | `RiskAnalysis` |341| M8 Institutional | Anchor investors, QIB subscription | `InstitutionalParticipation` |342 343**How a module runs (M1 example):**344```python345# 7 different retrieval queries sent in parallel346retrieval_queries = [347 "company business model revenue segments operations overview",348 "competitive strengths market position industry leadership",349 "revenue generated from sale of products percentage",350 ...351]352# All queries fire concurrently using ThreadPoolExecutor353# Results merged, reranked, top-22 chunks kept354# Then sent to GPT-4o-mini with structured prompt355```356 357**Structured Output with Pydantic:**358```python359structured_llm = llm.with_structured_output(BusinessQuality)360chain = prompt | structured_llm361result = chain.invoke({"context": context_text, "extraction_task": prompt})362# result is a validated Python object, not raw text363```364 365LangChain's `with_structured_output` forces the LLM to return JSON matching the Pydantic schema. If it doesn't, LangChain retries automatically.366 367**Each module returns a SIGNAL: POSITIVE / NEUTRAL / NEGATIVE**368 369---370 371### STEP 7 — FINAL SCORECARD (modules/scorecard.py)372 373Takes the 8 signals, counts positives/neutrals/negatives, and produces:374- Overall verdict: `STRONG BUY` / `BUY` / `NEUTRAL` / `AVOID`375- Investment thesis (one paragraph)376- Key positives list377- Key concerns list378 379---380 381### STEP 8 — AI CHAT (retriever.py — build_transient_hybrid_retriever)382 383Users can ask questions about the IPO after analysis:384 385```386User: "What is the revenue CAGR?"387 ↓388Build transient in-memory retriever (no disk write)389 ↓390Hybrid retrieve → rerank391 ↓392GPT-4o-mini answers using retrieved chunks393 ↓394Answer shown in chat UI395```396 397The chat uses a **transient retriever** — same hybrid RAG but in-memory (not persisted to Chroma disk). This is faster for interactive Q&A.398 399**LangChain is used here** to:400- Build the `ChatPromptTemplate` (system message + human message)401- Connect prompt → LLM in a chain: `chain = prompt | llm`402- Use `ChatOpenAI` client with retry logic403 404> **Note:** LangGraph is NOT used in this project. LangChain is used for prompt templates, LLM chains, and the retriever abstractions.405 406---407 408## PART 3: DATABASE DESIGN (database.py)409 410**SQLite** is used as the primary local database with these tables:411 412| Table | What it stores |413|---|---|414| `companies` | Company catalog from SEBI — name, DRHP/RHP/Prospectus URLs |415| `documents` | PDF metadata — download status, SHA256 hash, page count |416| `document_chunks` | All chunks for each company's document |417| `analyses` | Cached analysis results (JSON) keyed by company+doctype+sha256 |418| `embedding_cache` | Vector embeddings cached by (model, chunk_hash) |419| `users` | User accounts — email, password hash, subscription status |420| `telemetry` | User activity events |421| `diffs` | DRHP→RHP comparison results |422| `notifications` | In-app notifications per user |423| `referrals` | Referral tracking |424 425**Smart caching:** If the same PDF is re-analyzed, SHA-256 hash is checked. If unchanged, cached result is returned instantly — no re-embedding, no LLM calls.426 427---428 429## PART 4: FRONTEND PAGES (Next.js)430 431| Page | What it shows |432|---|---|433| `/` (Home) | Landing page with hero, features, how it works |434| `/companies` | Browse all IPOs from SEBI catalog |435| `/analytics` | Full analysis dashboard — 8 module cards, scorecard, AI chat |436| `/watchlist` | User's saved IPOs |437| `/settings` | Profile, subscription management |438| `/login` `/signup` | Auth pages |439 440---441 442## PART 5: HOW DATA FLOWS — SEBI INTEGRATION (sebi.py)443 444The backend has a SEBI scraper:445- Scrapes SEBI's website to get all current DRHP listings446- Matches each DRHP with its corresponding RHP and Prospectus447- Uses fuzzy string matching to link documents across filing stages448- Stores everything in the `companies` SQLite table449- Runs periodically to keep the catalog fresh450 451---452 453## PART 6: DIFF ENGINE (diff_engine.py)454 455A unique feature — comparing DRHP vs RHP:456- Chunks from both documents are compared using their `chunk_hash`457- New chunks in RHP = `added`458- Chunks only in DRHP = `removed`459- Modified content = `modified`460- A markdown diff report is generated highlighting what changed between filings461 462---463 464## PART 7: KEY DESIGN DECISIONS & WHY465 466**1. Why local embeddings (HuggingFace) instead of OpenAI embeddings?**467- No API cost per embedding (free)468- No rate limits — can embed 400-page documents without throttling469- all-MiniLM-L6-v2 is fast and good enough for document retrieval470- Parallelizable across threads471 472**2. Why ChromaDB instead of Pinecone/Weaviate?**473- Runs locally — no cloud dependency474- Persistent to disk — survives server restarts475- Free — no per-query cost476- Simple Python API477 478**3. Why Hybrid (BM25 + Dense) instead of just one?**479- Dense alone misses exact keywords (e.g., specific financial terms)480- BM25 alone misses semantic meaning481- Hybrid catches both — consistently better recall482 483**4. Why CrossEncoder reranking?**484- Bi-encoder embeddings compare query and document *separately* — good for speed485- CrossEncoder reads them *together* — much better at judging true relevance486- The cost is only over 25 candidates (not thousands), so it's fast enough487 488**5. Why Pydantic schemas?**489- Forces LLM to return structured, validated data490- No parsing of raw text needed491- If LLM hallucinates a wrong type, Pydantic raises validation error → retry492 493**6. Why SQLite + not just ChromaDB for everything?**494- ChromaDB stores vectors (for search)495- SQLite stores everything else: user data, analysis cache, company catalog496- They serve different purposes497 498---499 500## PART 8: INTERVIEW QUESTIONS & ANSWERS501 502### Q1: What is RAG? How did you implement it?503**Answer:** RAG is Retrieval-Augmented Generation. Instead of asking the LLM to memorize document content (impossible for 400 pages), I retrieve the most relevant chunks at query time and give them to the LLM as context. I implemented a Hybrid RAG: ChromaDB does dense vector search, BM25 does keyword search, an Ensemble Retriever fuses both using Reciprocal Rank Fusion, and a CrossEncoder reranks the results. Finally, the top chunks go into the LLM prompt via a LangChain chain.504 505### Q2: What is an embedding? How does it work?506**Answer:** An embedding is a fixed-size vector of numbers (like coordinates) that represents the semantic meaning of text. I use the `all-MiniLM-L6-v2` model from HuggingFace. It runs a transformer neural network on the input text and outputs a 384-dimensional vector. Similar texts produce vectors that are geometrically close (high cosine similarity). This lets us search by meaning, not just keywords.507 508### Q3: What vector database did you use and what search algorithm does it use?509**Answer:** I used **ChromaDB**. It uses **HNSW (Hierarchical Navigable Small World)** for approximate nearest neighbor search. HNSW builds a multi-layer graph where top layers have fewer, far-reaching connections and bottom layers are dense and local. Search starts at the top layer with few candidates and refines downward — achieving O(log n) complexity. I also use MMR (Maximal Marginal Relevance) to get diverse, non-redundant results.510 511### Q4: What is BM25? How is it different from vector search?512**Answer:** BM25 (Best Match 25) is a keyword-based ranking algorithm. It scores documents by how often a query term appears in the document (term frequency) weighted by how rare that term is across all documents (inverse document frequency). It's good for exact keyword matches. Vector search is semantic — it finds text with similar meaning even if words differ. Hybrid combines both for better overall recall.513 514### Q5: What is LangChain and where did you use it?515**Answer:** LangChain is a framework for building LLM applications. I used it for: (1) `ChatPromptTemplate` to build system + human message prompts, (2) `llm.with_structured_output(PydanticSchema)` to force structured JSON output, (3) `ChatOpenAI` as the LLM client with retry logic, (4) `HuggingFaceEmbeddings` wrapper for the local embedding model, (5) `Chroma` vectorstore integration, (6) `Document` class as the standard chunk object. The chain pattern `prompt | llm` is the LangChain Expression Language (LCEL).516 517### Q6: Did you use LangGraph?518**Answer:** No, I did not use LangGraph in this project. LangGraph is for building stateful, multi-agent workflows with loops and branching. My pipeline is linear (parse → chunk → embed → retrieve → generate), so I didn't need it. I used standard LangChain with ThreadPoolExecutor for parallelism.519 520### Q7: How do you handle the case where the same document is re-uploaded?521**Answer:** I compute a SHA-256 hash of the PDF file. This hash is stored with the analysis result in SQLite. On re-upload, I recompute the hash and check if an analysis already exists for that hash. If yes, I return the cached result immediately — no re-parsing, re-embedding, or LLM calls. For ChromaDB, I store the hash in a `.pdf_hash` file inside the collection directory and skip rebuilding the index if the hash matches.522 523### Q8: How do you handle parallel module execution?524**Answer:** The 8 analysis modules each need to retrieve documents and call the LLM. For retrieval, I use a `ThreadPoolExecutor` inside `AnalysisContext.build_context_docs()` — all retrieval queries for a module fire concurrently. The `AnalysisContext` class also memoizes retrieval results, so if two modules search for the same query, the second one gets results from cache. The modules themselves are called sequentially in `pipeline.py`, but each module's internal retrieval is parallelized.525 526### Q9: What is a CrossEncoder and why is it different from a bi-encoder?527**Answer:** A bi-encoder (like my embedding model) encodes the query and each document chunk *independently* into vectors, then computes similarity. It's fast but approximate. A CrossEncoder takes the query and document *together* as one input and outputs a single relevance score. Because it sees both at once, it can capture cross-attention between query and document terms — much more accurate. The trade-off is speed: it's too slow to run over thousands of documents, so I run it only over the 25 candidates already retrieved.528 529### Q10: How does your chunking strategy work?530**Answer:** I do section-aware, adaptive chunking. First, I detect which DRHP section each page belongs to (Risk Factors, Financial Statements, etc.) using regex patterns. Then for each page, I detect the content type: table, financial data, or narrative. Tables and financial data use smaller chunks (2000 chars, 100 overlap) to keep numbers together. Narrative text uses adaptive sizing — long prose gets bigger chunks (1600 chars), dense sections get smaller. I also merge tiny chunks (<900 chars) together to avoid having useless small pieces. Every chunk gets a SHA-256 hash for caching.531 532### Q11: How does the AI chat work?533**Answer:** After analysis, users can chat about the IPO. When a user sends a message, the backend builds a transient (in-memory) hybrid retriever from the pre-processed chunks for that company. It retrieves the most relevant chunks using the same BM25 + ChromaDB + CrossEncoder pipeline, then constructs a LangChain prompt with those chunks as context and sends it to GPT-4o-mini. The response is streamed back to the frontend. "Transient" means I don't persist this retriever to disk — it's built fresh in memory for the session.534 535### Q12: How does the SEBI scraper work?536**Answer:** I built a scraper in `sebi.py` that fetches the SEBI DRHP listing page, parses company names and filing dates, and upserts them into the SQLite `companies` table. For each DRHP, I run a fuzzy string matching algorithm to find the corresponding RHP and Prospectus filings (if they exist). This matching uses normalized company name comparison — stripping "Limited", "Private", special chars — and a match score threshold. This keeps the platform's company catalog automatically up to date.537 538### Q13: How do you store and retrieve embeddings?539**Answer:** Embeddings are stored in the SQLite `embedding_cache` table with columns: `model_name`, `chunk_hash`, `vector_json`. When embedding a batch of chunks, I first check which chunk hashes are already in the cache. Only the cache-miss chunks get sent to the embedding model. After computing, new embeddings are saved back. For a 400-page document with 800 chunks, the second analysis of the same document takes near-zero time for embeddings.540 541### Q14: What is Pydantic and why did you use it for LLM output?542**Answer:** Pydantic is a Python data validation library. I defined schemas (like `BusinessQuality`, `ValuationAnalysis`) as Pydantic `BaseModel` subclasses. LangChain's `llm.with_structured_output(MySchema)` instructs the LLM to return JSON matching that schema, then parses and validates it automatically. If a field has the wrong type or is missing, Pydantic raises a validation error. This eliminates the need to manually parse LLM text output and guarantees consistent, typed data structures from every module.543 544### Q15: What would you improve next?545**Answer:** (1) Add streaming progress updates to the frontend during analysis using SSE (Server-Sent Events). (2) Switch to a faster embedding model like `bge-small-en` for production. (3) Add a feedback mechanism where users can rate analysis accuracy, creating a training dataset. (4) Implement automatic RHP vs DRHP delta highlighting on the UI — the diff engine is built but not fully surfaced. (5) Scale to concurrent multi-user analysis using a job queue like Celery + Redis instead of in-process threading.546 547---548 549## PART 9: QUICK REVISION CHEAT SHEET550 551```552PDF → PyMuPDF4LLM (parser) → pages as Documents553 ↓554 chunker.py → smart chunks (narrative/table/financial)555 ↓556 embeddings.py → all-MiniLM-L6-v2 → 384-dim vectors557 ↓558 ChromaDB (HNSW) + SQLite (embedding_cache)559 ↓560 Retrieval: ChromaDB (dense MMR) + BM25 (keyword)561 ↓562 EnsembleRetriever (RRF fusion 0.6/0.4)563 ↓564 CrossEncoder reranker (ms-marco-MiniLM-L-6-v2)565 ↓566 LangChain chain: ChatPromptTemplate | ChatOpenAI (gpt-4o-mini)567 ↓568 Pydantic structured output → 8 module results + scorecard569 ↓570 Next.js frontend displays results + AI Chat571```572 573**Key Numbers to Remember:**574- Embedding dimensions: **384**575- Embedding model: **all-MiniLM-L6-v2**576- Reranker model: **cross-encoder/ms-marco-MiniLM-L-6-v2**577- LLM: **GPT-4o-mini**578- Top-K retrieved: **25**579- Top-K after rerank: **14**580- Embedding batch size: **100**581- Parallel embedding workers: **8**582- Analysis modules: **8 + 1 scorecard**583 584---585 586## PART 10: PROMPTING TECHNIQUES USED587 588This is a very common interview question. Here is exactly what techniques are used and why.589 590### Technique 1 — Role Prompting (Persona Assignment)591**What it is:** Give the LLM a specific role/identity before asking it to do something.592 593**Where used:** Every module's system prompt starts with:594```595"You are a sell-side equity research analyst at a global investment bank analyzing an Indian IPO DRHP."596```597 598**Why it works:** LLMs produce more domain-specific, accurate output when given a clear expert persona. The model activates "finance analyst" knowledge rather than giving generic answers.599 600---601 602### Technique 2 — Structured Output Prompting (Schema-Constrained Generation)603**What it is:** Instead of asking the LLM to write free text, you force it to fill in a predefined JSON schema using Pydantic.604 605**Where used:** Every module uses `llm.with_structured_output(PydanticSchema)`.606 607**Example:**608```python609class BusinessQuality(BaseModel):610 company_name: str611 revenue_segments: List[RevenueSegment]612 signal: str # must be POSITIVE / NEUTRAL / NEGATIVE613 signal_reasoning: str614```615 616**Why it works:** The LLM cannot deviate from the schema. If it tries to return wrong types, LangChain catches the validation error and retries. This eliminates the need to parse free text.617 618---619 620### Technique 3 — Chain-of-Thought (Step-by-Step Instructions)621**What it is:** Instead of asking "extract the revenue", you break it into explicit steps the LLM must follow.622 623**Where used:** Module 2 (Revenue) prompt uses explicit STEP instructions:624```625STEP 1 — Find the unit: Look for "Amount in ₹ Lakhs" or "₹ Crores" at top of table.626STEP 2 — Strip ALL commas from numbers before arithmetic ("20,000" → 20000)627STEP 3 — Convert to Crores using the unit from STEP 1628STEP 4 — Extract per fiscal year...629STEP 5 — Write trend_summary630```631 632**Why it works:** LLMs make fewer errors when they follow a chain of reasoning steps rather than jumping to the answer. It also makes the output auditable — you can check which step failed.633 634---635 636### Technique 4 — Few-Shot Examples (Good vs Bad)637**What it is:** Show the LLM examples of what you DO want and what you DON'T want.638 639**Where used:** Risk module (M7) and Business module (M1) prompts:640```641Bad: "The company has customer concentration risk."642Good: "Top 3 clients contribute 65% of revenue (₹450 Cr of ₹690 Cr total).643 Loss of any single major client would materially impact profitability."644```645Also in M1:646```647Example good strength: "High entry barrier — Sole provider of X component in India"648Example bad: "Experienced management team" (too generic)649```650 651**Why it works:** Concrete examples anchor the LLM to the expected output quality and format without verbose instructions.652 653---654 655### Technique 5 — Negative Constraints (DO NOT Rules)656**What it is:** Explicitly tell the model what NOT to do.657 658**Where used:** Throughout all modules:659```660- Extract ONLY what is explicitly stated. NEVER invent or assume numbers.661- Do NOT mix figures from different reporting periods.662- NEVER extrapolate, forecast, or estimate future years.663- Do NOT extract raw material purchases unless they are explicitly sold as a product.664- NEVER return 0 if you cannot find the value — return null instead.665```666 667**Why it works:** LLMs tend to hallucinate when uncertain. Explicit "never do X" rules reduce hallucination significantly because the model is trained to follow instructions.668 669---670 671### Technique 6 — Grounding (Context-First Prompting)672**What it is:** Always give the LLM the source document context BEFORE asking it to extract anything.673 674**Where used:** Every module builds a `context_text` from retrieved chunks and passes it as:675```python676("human", "{extraction_task}\n\nDRHP CONTEXT:\n{context}")677```678 679**Why it works:** This is the "RAG" grounding — the LLM cannot hallucinate facts because every claim must come from the provided context. The system prompt reinforces this:680```681"Extract ONLY what is explicitly stated in the context."682```683 684---685 686### Technique 7 — Signal Rules (Classification Rubric)687**What it is:** Give the LLM a decision rubric so it classifies consistently.688 689**Where used:** Every module has explicit signal rules:690```691POSITIVE if: Revenue CAGR >15% AND EBITDA margin stable/improving692NEUTRAL if: Revenue CAGR 8-15% OR margins volatile but improving693NEGATIVE if: Revenue CAGR <8% OR margins consistently declining OR losses694```695 696**Why it works:** Without a rubric, different analysis runs would classify the same company differently. With explicit thresholds, the output is reproducible and consistent.697 698---699 700### Technique 8 — Document-Type Aware Prompting701**What it is:** The prompt changes slightly depending on whether the document is a DRHP, RHP, or Prospectus.702 703**Where used:** The `doc_type` parameter is injected throughout prompts:704```705"You are analyzing a {doc_type.upper()} — this is different from a DRHP in that706 it may contain updated pricing, subscription figures, and final allocation data."707```708 709**Why it works:** Prevents the model from confusing document types and applying wrong extraction rules.710 711---712 713## PART 11: HOW HALLUCINATION IS HANDLED714 715### What is Hallucination?716Hallucination = LLM makes up facts, numbers, or names that don't exist in the source document.717 718### My 6-Layer Anti-Hallucination Defense719 720**Layer 1 — RAG Grounding (Most Important)**721The LLM never works from memory. It ALWAYS gets specific text chunks from the actual DRHP as context. The system prompt says:722```723"Extract ONLY what is explicitly stated in the context. Never invent or assume numbers."724```725This is the #1 defense. If the text isn't in the context window, the LLM is told to return `null`, not guess.726 727---728 729**Layer 2 — Null over Zero Rule**730```731"If you cannot find EBITDA, return null — NEVER return 0."732```733Returning 0 when a value is missing is a silent hallucination. Returning `null` is honest — the code then handles missing values correctly instead of using a wrong number.734 735---736 737**Layer 3 — Pydantic Validation**738Every LLM output is validated against a Pydantic schema. If the model returns:739- A string where a float is expected → validation error → LangChain retries740- A signal value other than "POSITIVE/NEUTRAL/NEGATIVE" → caught741- Missing required fields → caught742 743---744 745**Layer 4 — Python Post-Processing (M2 Revenue Module)**746Instead of trusting the LLM to calculate percentages:747```python748# LLM extracts raw numbers. Python calculates margins:749fy.ebitda_margin_pct = round((fy.ebitda_cr / fy.revenue_cr) * 100, 2)750fy.pat_margin_pct = round((fy.pat_cr / fy.revenue_cr) * 100, 2)751fy.revenue_yoy_pct = round(((curr - prev) / prev) * 100, 2)752```753LLMs make arithmetic errors. Offloading math to Python eliminates a whole class of hallucination.754 755---756 757**Layer 5 — Future Year Hallucination Filter**758LLMs sometimes extrapolate future years (e.g., "FY2026" when the doc only shows FY2024):759```python760current_year = datetime.datetime.now().year761financials = [f for f in financials762 if not any(str(y) in f.year for y in range(current_year+1, current_year+4))]763```764Any year beyond current year is stripped out.765 766---767 768**Layer 6 — Lakhs Auto-Detection Safety Net**769Indian financial documents use Lakhs or Crores. LLMs sometimes miss the unit header and return Lakh values as Crore values (100x error). I detect this:770```python771# If most revenue values > 500, they're probably in Lakhs, not Crores772lakhs_count = sum(1 for v in revenue_values if v > 500)773if lakhs_count >= len(revenue_values) * 0.6:774 # Auto-convert: divide all by 100775 fy.revenue_cr = round(fy.revenue_cr / 100, 2)776```777 778---779 780**Layer 7 — Retry with Raw Pages Fallback**781If the LLM returns ALL null values for financial data (total failure):782```python783all_null = all(f.revenue_cr is None and f.pat_cr is None for f in result.financials)784if all_null and raw_context_chunks:785 result = _retry_with_raw_pages(llm, raw_context_chunks)786```787The system retries the LLM call with a different, simpler prompt using raw page text directly.788 789---790 791**Layer 8 — Known Company Name Injection**792To prevent the LLM from mis-labeling the company name (e.g., calling the company by a competitor's name):793```python794if known_company_name:795 extraction_prompt = (796 f"KNOWN COMPANY NAME: {known_company_name}\n"797 f"Use EXACTLY this name unless the DRHP explicitly contradicts it.\n\n"798 + extraction_prompt799 )800```801 802---803 804## PART 12: CHALLENGING INTERVIEW QUESTIONS & CROSS-CHALLENGES805 806### CROSS-CHALLENGE Q1: "You said you use Pydantic for structured output. But what if the LLM returns a valid Pydantic object with wrong numbers — like a hallucinated revenue of ₹9999 Cr? Pydantic won't catch that."807 808**Answer:** You're absolutely right — Pydantic validates *structure* (types, field names), not *semantic correctness* (whether the number is plausible). That's why I have additional layers:8091. **Python post-processing** — I recalculate margins from raw numbers, so an impossible margin would be mathematically obvious8102. **The Lakhs auto-detect heuristic** — catches unit conversion errors (the most common source of wrong numbers)8113. **The RAG grounding** — the LLM is given only chunks from the actual document. If it returns ₹9999 Cr, it must have read that in the doc, or the chunk retrieval failed to find the right table8124. **The null-is-better-than-wrong rule** — I explicitly instruct the LLM to return null if uncertain rather than guessing813 814For a production system, I would add a plausibility check: compare extracted revenue against SEBI filing metadata or cross-check across two retrieval passes.815 816---817 818### CROSS-CHALLENGE Q2: "Why did you use all-MiniLM-L6-v2 for embeddings? It's only 384 dimensions and a small model. Wouldn't a larger model like text-embedding-3-large (3072 dims) be more accurate?"819 820**Answer:** That's a fair question. My choice was driven by these constraints:8211. **No API cost** — all-MiniLM-L6-v2 runs locally for free. For a 400-page DRHP with 800+ chunks, OpenAI's embedding API would cost money per analysis. With local embeddings, it's free and unlimited.8222. **Speed** — the small model embeds 100 chunks in ~2 seconds on CPU. A larger model would be much slower or require GPU.8233. **For retrieval tasks** (not similarity scoring), small models are competitive. MTEB benchmarks show all-MiniLM-L6-v2 at ~57% on retrieval tasks vs text-embedding-3-large at ~64% — not a massive gap for domain-specific structured document retrieval.8244. **The CrossEncoder compensates** — the bi-encoder's approximate ranking is corrected by the CrossEncoder reranker. So even if the initial retrieval is 80% accurate, the reranker refines the final top-14.825 826In production at scale, I'd migrate to a better embedding model. The embedding cache in SQLite makes this easy — swap the model, invalidate cache, re-embed.827 828---829 830### CROSS-CHALLENGE Q3: "Hybrid retrieval with BM25 + Dense — how do you actually combine their scores? If BM25 scores are in the range 0-20 and cosine similarity is 0-1, they're on completely different scales. How do you normalize?"831 832**Answer:** I use **Reciprocal Rank Fusion (RRF)** — NOT score fusion. This is the key insight.833 834Instead of combining raw scores (which are incomparable across methods), RRF uses only the **rank position**:835```836score = weight × (1 / (rank + 60))837```838- A document ranked #1 by BM25 gets: `0.4 × (1/61) = 0.00656`839- A document ranked #1 by ChromaDB gets: `0.6 × (1/61) = 0.00984`840- A document ranked #5 by both gets: `0.4×(1/65) + 0.6×(1/65) = 0.01538`841 842The 60 is a smoothing constant that reduces the impact of rank differences at the top. Documents that rank high in BOTH retrievers float to the top, regardless of their raw scores. This is scale-invariant — BM25 scores vs cosine similarities don't matter at all.843 844---845 846### CROSS-CHALLENGE Q4: "You said ChromaDB uses HNSW. But HNSW gives approximate results, not exact nearest neighbors. How do you know you're not missing critical chunks?"847 848**Answer:** Great question. A few things mitigate this:8491. **I retrieve top-25**, not top-1. Even if HNSW misses the single closest vector, it'll still find the correct chunk within the top 25 candidates with very high probability. HNSW's recall@25 is typically >99%.8502. **BM25 acts as a safety net** — HNSW might miss a chunk where the query and document use different words (semantic gap). BM25 will still find it via exact keyword match. Together, hybrid retrieval has higher recall than either alone.8513. **Forced keyword injection** — for critical patterns I know must appear (like "revenue from operations percentage"), I have `forced_keyword_patterns` that hard-inserts matching chunks regardless of retrieval ranking:852```python853forced_keyword_patterns=["revenue generated from sale of our products", "percentage of our revenue"]854```8554. **The CrossEncoder reranker** operates on the retrieved set — it won't find chunks that weren't retrieved, but the combination of hybrid + forced patterns makes missing critical chunks very unlikely.856 857---858 859### CROSS-CHALLENGE Q5: "What's the difference between your SimpleEnsembleRetriever and LangChain's built-in EnsembleRetriever? Why did you write your own?"860 861**Answer:** LangChain's built-in `EnsembleRetriever` uses the same RRF algorithm. I wrote `SimpleEnsembleRetriever` because:8621. **Dependency control** — LangChain's EnsembleRetriever requires `langchain_community` with BM25Retriever, which sometimes has version conflicts8632. **Fallback resilience** — my implementation catches errors per-retriever and continues with remaining retrievers. If BM25 crashes, dense retrieval still works8643. **Custom doc ID logic** — I use `chunk_hash` metadata as the deduplication key, which is more reliable than LangChain's default content hashing8654. **In-memory caching** — `AnalysisContext` memoizes raw search results so the same query never hits the retriever twice across parallel module calls866 867---868 869### CROSS-CHALLENGE Q6: "You use GPT-4o-mini. It's a smaller model. For financial analysis with very specific numbers from complex documents, wouldn't GPT-4 or Claude be more accurate?"870 871**Answer:** This is a deliberate trade-off:8721. **GPT-4o-mini is fast** — large models like GPT-4 take 30-60s per call. With 8 modules, that's 4-8 minutes. GPT-4o-mini cuts this to ~1 minute total.8732. **Context quality matters more than model size** — since I use RAG, the LLM is only answering questions about text directly in its context window. It doesn't need to "know" finance — it needs to accurately read and extract from the chunks I give it. For this extraction task, GPT-4o-mini is very competitive.8743. **Structured output reduces model dependency** — with Pydantic schemas, the model just needs to fill fields correctly, not reason from scratch.8754. **Cost** — GPT-4 is ~30x more expensive than GPT-4o-mini. For a multi-user platform, this matters a lot.8765. **The base_url parameter** — my config supports swapping the LLM via environment variable. I can switch to Claude or GPT-4 with zero code changes.877 878---879 880### ADVANCED Q: "What is the AnalysisContext class and why does it exist?"881 882**Answer:** `AnalysisContext` is a per-analysis memoization layer. The problem it solves: 8 modules run in the same analysis. Modules M1 and M7 both might search for "risk factors". Without caching, that's two identical ChromaDB + BM25 + CrossEncoder calls — wasted work.883 884`AnalysisContext` has three caches:8851. `_raw_query_cache` — maps query string → raw retrieved docs (no reranking)8862. `_reranked_query_cache` — maps (query, section_filter, top_k) → reranked docs8873. `_final_rerank_cache` — maps (combined_query, doc_ids_tuple, top_k) → final reranked set888 889So when two modules use the same query, the second module gets cached results instantly — no retriever call, no CrossEncoder call. This saves significant time on large documents.890 891---892 893## PART 13: CODING QUESTIONS894 895### Coding Q1: "Write the BM25 scoring formula in Python from scratch."896 897```python898import math899import re900from typing import List, Dict901 902class BM25:903 def __init__(self, documents: List[str], k1: float = 1.5, b: float = 0.75):904 self.k1 = k1905 self.b = b906 self.documents = documents907 self.tokenized = [self._tokenize(d) for d in documents]908 self.N = len(documents)909 self.avgdl = sum(len(d) for d in self.tokenized) / max(self.N, 1)910 self.idf = self._compute_idf()911 912 def _tokenize(self, text: str) -> List[str]:913 return re.findall(r'[a-z0-9]+', text.lower())914 915 def _compute_idf(self) -> Dict[str, float]:916 df: Dict[str, int] = {}917 for tokens in self.tokenized:918 for term in set(tokens):919 df[term] = df.get(term, 0) + 1920 idf = {}921 for term, count in df.items():922 # BM25 IDF formula with smoothing923 idf[term] = math.log(1 + (self.N - count + 0.5) / (count + 0.5))924 return idf925 926 def score(self, query: str, doc_index: int) -> float:927 query_terms = self._tokenize(query)928 doc_tokens = self.tokenized[doc_index]929 doc_len = len(doc_tokens)930 931 # Term frequency in document932 tf: Dict[str, int] = {}933 for term in doc_tokens:934 tf[term] = tf.get(term, 0) + 1935 936 score = 0.0937 for term in query_terms:938 if term not in self.idf:939 continue940 freq = tf.get(term, 0)941 # BM25 term score formula942 numerator = freq * (self.k1 + 1)943 denominator = freq + self.k1 * (1 - self.b + self.b * (doc_len / self.avgdl))944 score += self.idf[term] * (numerator / denominator)945 return score946 947 def rank(self, query: str) -> List[tuple]:948 scores = [(i, self.score(query, i)) for i in range(self.N)]949 return sorted(scores, key=lambda x: x[1], reverse=True)950 951# Usage:952bm25 = BM25(["revenue grew by 40 percent", "promoter sold shares", "EBITDA margin improved"])953print(bm25.rank("revenue growth")) # [(0, high_score), (2, medium), (1, low)]954```955 956**Explain:** k1 controls term frequency saturation (1.2-2.0). b controls document length normalization (0=none, 1=full). IDF gives rare terms higher weight. avgdl normalizes for longer documents.957 958---959 960### Coding Q2: "Write cosine similarity between two vectors in Python without numpy."961 962```python963import math964from typing import List965 966def cosine_similarity(vec_a: List[float], vec_b: List[float]) -> float:967 """968 cosine_similarity = dot_product(a, b) / (magnitude(a) * magnitude(b))969 Range: -1 (opposite) to 1 (identical), 0 = orthogonal (unrelated)970 """971 if len(vec_a) != len(vec_b):972 raise ValueError("Vectors must be same length")973 974 dot_product = sum(a * b for a, b in zip(vec_a, vec_b))975 magnitude_a = math.sqrt(sum(a * a for a in vec_a))976 magnitude_b = math.sqrt(sum(b * b for b in vec_b))977 978 if magnitude_a == 0 or magnitude_b == 0:979 return 0.0980 981 return dot_product / (magnitude_a * magnitude_b)982 983# With normalized vectors (all-MiniLM-L6-v2 normalizes by default):984# magnitude is already 1, so cosine_similarity = dot_product985# This is why normalized embeddings make search faster!986```987 988---989 990### Coding Q3: "Write Reciprocal Rank Fusion to merge two ranked lists."991 992```python993from typing import List, Dict994 995def reciprocal_rank_fusion(996 ranked_lists: List[List[str]],997 weights: List[float],998 k: int = 60999) -> List[str]:1000 """1001 Merge multiple ranked lists using RRF.1002 ranked_lists: each list is doc_ids in ranked order (best first)1003 weights: importance weight per list (must sum to 1)1004 k: smoothing constant (default 60, standard in literature)1005 """1006 scores: Dict[str, float] = {}1007 1008 for ranked_list, weight in zip(ranked_lists, weights):1009 for rank, doc_id in enumerate(ranked_list, start=1):1010 rrf_score = weight * (1.0 / (rank + k))1011 scores[doc_id] = scores.get(doc_id, 0.0) + rrf_score1012 1013 # Sort by combined RRF score descending1014 return sorted(scores.keys(), key=lambda d: scores[d], reverse=True)1015 1016# Example:1017dense_results = ["doc_A", "doc_C", "doc_B", "doc_D"] # ChromaDB ranking1018sparse_results = ["doc_B", "doc_A", "doc_E", "doc_C"] # BM25 ranking1019 1020fused = reciprocal_rank_fusion(1021 ranked_lists=[dense_results, sparse_results],1022 weights=[0.6, 0.4]1023)1024print(fused) # doc_A and doc_B likely at top (appeared high in both)1025```1026 1027---1028 1029### Coding Q4: "Write chunking logic that detects if a text is a table or narrative."1030 1031```python1032import re1033 1034def detect_chunk_type(text: str) -> str:1035 """1036 Detect whether a text chunk is a table, financial data, or narrative.1037 """1038 # Table detection: markdown pipes + dashes1039 if text.count("|") > 10 and "---" in text:1040 return "table"1041 1042 # Financial data: high number density or finance keywords1043 words = text.split()1044 numbers = re.findall(r'\b\d[\d,.]*\b', text)1045 number_density = len(numbers) / max(len(words), 1)1046 1047 financial_keywords = ["crore", "lakh", "inr", "ebitda", "revenue",1048 "pat", "profit", "margin", "turnover"]1049 has_finance_keywords = any(kw in text.lower() for kw in financial_keywords)1050 1051 if number_density > 0.15 or has_finance_keywords:1052 return "financial_data"1053 1054 return "narrative"1055 1056# Test:1057table_text = "| Year | Revenue | Profit |\n|---|---|---|\n| FY24 | 500 | 80 |"1058finance_text = "The EBITDA for FY2024 was ₹120 crore representing a margin of 24%."1059narrative_text = "The company was founded in 2010 and operates across three verticals."1060 1061print(detect_chunk_type(table_text)) # "table"1062print(detect_chunk_type(finance_text)) # "financial_data"1063print(detect_chunk_type(narrative_text)) # "narrative"1064```1065 1066---1067 1068### Coding Q5: "How would you implement a simple embedding cache using a dictionary? What are the trade-offs vs SQLite?"1069 1070```python1071import hashlib1072import json1073from typing import Dict, List, Optional1074 1075class InMemoryEmbeddingCache:1076 """Simple in-memory embedding cache. Lost on server restart."""1077 1078 def __init__(self):1079 self._cache: Dict[str, List[float]] = {} # chunk_hash -> vector1080 1081 def get(self, chunk_hash: str) -> Optional[List[float]]:1082 return self._cache.get(chunk_hash)1083 1084 def set(self, chunk_hash: str, vector: List[float]) -> None:1085 self._cache[chunk_hash] = vector1086 1087 def get_batch(self, hashes: List[str]) -> Dict[str, List[float]]:1088 return {h: self._cache[h] for h in hashes if h in self._cache}1089 1090 def compute_hash(self, text: str) -> str:1091 normalized = ' '.join(text.lower().split()).encode('utf-8')1092 return hashlib.sha256(normalized).hexdigest()1093 1094 1095# --- Trade-offs ---1096# In-Memory Cache:1097# PROS: Microsecond reads, zero overhead, simple code1098# CONS: Lost on restart, uses RAM (384 floats × 4 bytes × 10,000 chunks = ~15MB)1099 1100# SQLite Cache (what I actually use):1101# PROS: Persists across restarts, survives crashes, can cache millions of chunks1102# CONS: Millisecond reads (disk I/O), concurrent write locking, JSON serialization overhead1103 1104# My actual implementation uses SQLite:1105# Table: embedding_cache(model_name, chunk_hash, vector_json)1106# Key insight: vector stored as JSON string, loaded back as list on read1107# Batch reads use: SELECT chunk_hash, vector_json WHERE chunk_hash IN (?,?,?)1108```1109 1110---1111 1112## PART 14: FINAL IMPORTANT TIPS FOR INTERVIEW1113 1114### Things to Say Confidently1115- "I implemented a custom BM25 retriever from scratch — I understand the IDF formula and why k1=1.5, b=0.75 are the standard defaults"1116- "I use Reciprocal Rank Fusion, not score normalization, because RRF is scale-invariant across different retrieval methods"1117- "The CrossEncoder and bi-encoder serve different purposes — bi-encoder for fast initial retrieval, CrossEncoder for accurate final reranking"1118- "I deliberately offload all math to Python and tell the LLM to return nulls rather than calculated values — this removes a whole class of hallucination"1119- "I built AnalysisContext to memoize retrieval calls across 8 parallel modules — this saved significant processing time"1120 1121### Things NOT to Say1122- Don't say "LangGraph" — you don't use it1123- Don't say "I just used LangChain" without explaining what you did with it1124- Don't say "the LLM handles hallucination" — explain your 7-layer defense system1125 1126### If Asked "Did you write all this code yourself?"1127Say: "Yes. I designed the architecture from scratch — the hybrid retrieval strategy, the section-aware chunking, the parallel embedding pipeline, the AnalysisContext memoization, the Lakhs/Crores auto-detection heuristic, and the Python post-processing for financial math. LangChain gave me the glue (prompt templates, LLM client, Chroma wrapper), but all the domain logic and performance optimizations are my own design decisions."1128 1129---1130 1131## PART 15: NEXT.JS INTERVIEW QUESTIONS (Frontend)1132 1133### Q1: What is Next.js and why did you use it instead of plain React?1134 1135**Answer:**1136Next.js is a React framework built on top of React that adds server-side rendering, file-based routing, API routes, and image/font optimization.1137 1138**Why I used it:**1139- **App Router** (Next.js 14) — folder-based routing. Each folder under `app/` is a route. Example: `app/analytics/page.tsx` → `/analytics` URL.1140- **Server Components by default** — components render on the server, reducing JS sent to browser = faster load1141- **`use client` directive** — I mark only interactive components as client-side (like the Companies page with `useState`, `useEffect`)1142- **Built-in font optimization** — Google Fonts loaded via `<link>` in `layout.tsx` with `preconnect` headers, zero layout shift1143- **`next/script`** — I used it for Google Analytics script with `strategy="afterInteractive"` so it doesn't block page load1144- **Metadata API** — I define `export const metadata` in `layout.tsx` for SEO: title, description, OpenGraph, Twitter card1145 1146---1147 1148### Q2: What is the difference between Server Components and Client Components in Next.js 14?1149 1150| | Server Component | Client Component |1151|---|---|---|1152| Renders on | Server | Browser |1153| Can use | `async/await`, DB calls, fs | `useState`, `useEffect`, browser APIs |1154| JS sent to browser | None | Yes |1155| How to mark | Default (no directive) | `'use client'` at top |1156 1157**In my project:**1158- `layout.tsx` — Server Component (renders HTML shell, metadata, fonts)1159- `companies/page.tsx` — Client Component (`'use client'`) because it uses `useState`, `useEffect`, `useRouter`1160- `AuthContext` — Client Component (uses React Context which needs browser)1161 1162---1163 1164### Q3: What is the App Router? How does routing work in your project?1165 1166**Answer:**1167In Next.js 14 App Router, every folder under `app/` that contains a `page.tsx` becomes a URL route automatically.1168 1169**My routes:**1170```1171app/1172 page.tsx → / (landing page)1173 companies/page.tsx → /companies1174 analytics/page.tsx → /analytics1175 watchlist/page.tsx → /watchlist1176 settings/page.tsx → /settings1177 login/page.tsx → /login1178 signup/page.tsx → /signup1179 payment/page.tsx → /payment1180```1181 1182**`layout.tsx`** wraps every page with shared UI (Navbar, AuthProvider, Analytics script). This is the "root layout".1183 1184**Navigation:** I use `useRouter()` from `next/navigation`:1185```typescript1186const router = useRouter();1187router.push(`/analytics/${encodeURIComponent(company.normalized_name)}`);1188```1189 1190---1191 1192### Q4: How does AuthContext work in your project? Explain React Context.1193 1194**Answer:**1195React Context is a way to share state across many components without passing props down every level (prop drilling).1196 1197**My AuthContext:**1198```typescript1199const { user, updateUser } = useAuth();1200```