omkarkudalkar23/citationEdge

1# Visual Parser Agent2 3Extracts and semantically understands figures, flowcharts, architecture diagrams,4and charts from research paper PDFs. Connects visual evidence to the claim5verification pipeline.6 7---8 9## Why it exists10 11Text-only parsing misses critical information locked inside images:12 13- A claim says _"as shown in Figure 3, accuracy improves by 40%"_ — the figure14  data must validate this, not just the surrounding text.15- Citations like `[1]` or `[Smith 2020]` sometimes appear only inside diagrams.16- Flowcharts and architecture diagrams describe methodology that may never be17  fully written out in prose.18 19---20 21## How it works22 23```24PDF25 │26 ├─ Step 1: pymupdf drawing detection27 │          Count vector paths per page.28 │          Pages with ≥ 8 paths = diagram candidate.29 │          (Flowcharts, architecture diagrams, charts are all vector graphics.)30 │31 ├─ Step 2: Page render at 1.8× resolution32 │          Each candidate page rendered to a PIL image (~724×1024 px).33 │34 ├─ Step 3: Groq llama-4-scout-17b vision35 │          Image + structured prompt → JSON with 7 fields.36 │37 └─ Step 4: Store Figure node in Neo4j38            (d:Document)-[:HAS_FIGURE]->(f:Figure)39```40 41### Why pymupdf drawing detection over alternatives42 43| Method | What it finds | Problem |44|---|---|---|45| pdfplumber embedded | Rasterized bitmap images | Misses all vector graphics (most research diagrams) |46| pymupdf all pages | Everything | Too broad — 38 pages for a 10-page paper |47| **pymupdf drawing detection** | Pages with ≥8 vector paths | Targets actual diagrams precisely |48 49Tested on `ML4H.pdf` (10 pages):50- pdfplumber: **0 images** found51- pymupdf all pages: **38 items** (all full-page renders, no filtering)52- Drawing detection: **20 candidate pages** — only pages that actually contain diagrams53 54---55 56## Output schema57 58Each detected figure produces a `Figure` node in Neo4j:59 60```json61{62  "figure_id": "fig_a3b2c1",63  "page": 4,64  "visual_type": "flowchart | architecture | graph_chart | table | neural_network | equation | other",65  "title": "Figure caption or inferred title",66  "description": "1-2 sentence description of what the visual shows",67  "components": ["step 1", "step 2", "..."],68  "supported_claims": ["claim this figure proves or shows"],69  "visible_citations": ["[1]", "[Smith 2020]"],70  "visible_text": "all readable text inside the figure"71}72```73 74---75 76## Real examples (tested on `flowchart.png` and `ML4H.pdf`)77 78### Example 1 — Flowchart79 80**Input:** `flowchart.png` — Research Methodology flowchart81 82**Output:**83```json84{85  "has_visual": true,86  "visual_type": "flowchart",87  "title": "Flowchart for Research Methodology with Design and Development",88  "description": "A flowchart illustrating the steps involved in research methodology with design and development.",89  "components": [90    "Identify Problem",91    "Define Objectives of Solution",92    "Design & Development",93    "Exhibition",94    "Evaluation",95    "Communication",96    "Problem Centered Initiation",97    "Objective Centered Solution",98    "Design Centered Solution",99    "Context Initiation"100  ],101  "supported_claims": [102    "The flowchart presents a structured approach to research methodology.",103    "It highlights the importance of identifying problems and defining objectives."104  ],105  "visible_citations": [],106  "visible_text": [107    "IDENTIFY PROBLEM",108    "DEFINE OBJECTIVES OF SOLUTION",109    "DESIGN & DEVELOPMENT",110    "EXHIBITION",111    "EVALUATION",112    "COMMUNICATION",113    "Problem Centered Initiation Text Here",114    "Objective Centered Solution Text Here",115    "Design Centered Solution Text Here",116    "Context Initiation Text Here"117  ]118}119```120 121### Example 2 — Neural Network Architecture122 123**Input:** Page 4 of `ML4H.pdf` (112 vector paths detected)124 125**Output:**126```json127{128  "has_visual": true,129  "visual_type": "neural_network",130  "title": "The general structure of the most basic type of artificial neuron, called a perceptron",131  "description": "The general structure of a perceptron is shown, including the weighted sum, activation function, and output.",132  "components": ["bias b", "weighted sum", "activation function", "inputs x1..xn", "output y"],133  "supported_claims": [134    "Single perceptrons are limited to learning linearly separable functions"135  ],136  "visible_citations": [],137  "visible_text": "output (output to next layer), bias b, activation function, y = { 1 if ∑..."138}139```140 141### Example 3 — Graph/Chart142 143**Input:** Page 5 of `ML4H.pdf` (31 vector paths detected)144 145**Output:**146```json147{148  "has_visual": true,149  "visual_type": "graph_chart",150  "title": "Representations of the Boolean functions OR and XOR",151  "description": "The figure shows two graphs representing the Boolean functions OR and XOR, and their corresponding truth tables.",152  "components": [153    "logical OR graph",154    "logical XOR graph",155    "truth table for OR",156    "truth table for XOR"157  ],158  "supported_claims": [159    "The OR function is linearly separable, whereas the XOR function is not."160  ],161  "visible_citations": [],162  "visible_text": "logical OR (linearly separable), logical XOR (not linearly separable), Input..."163}164```165 166---167 168## How it connects to claim verification169 170The `EvidenceGroundingAgent` (Wave 3) queries Figure nodes when grounding claims:171 172```cypher173MATCH (d:Document {doc_id: $doc_id})-[:HAS_FIGURE]->(f:Figure)174WHERE any(kw IN $keywords WHERE175      toLower(f.description) CONTAINS kw176   OR toLower(f.visible_text) CONTAINS kw177   OR toLower(f.supported_claims) CONTAINS kw)178RETURN f179```180 181When a figure's `supported_claims` strongly overlaps a text claim (≥2 keywords),182a direct relationship is created:183 184```185(f:Figure)-[:VISUALLY_SUPPORTS]->(c:Claim)186```187 188This means:189- _"The OR function is linearly separable"_ in text → linked to the XOR/OR chart190- _"As shown in Figure 3, sigmoid units handle non-linearity"_ → linked to the191  sigmoid diagram on page 7192- Any `[1]` citation found in `visible_citations` → surfaced in citation gap analysis193 194---195 196## Running the agent197 198```python199from agents.visual_parser_agent import VisualParserAgent200 201agent = VisualParserAgent()202result = await agent.execute(ctx)203# result = {204#   "figures": [...],205#   "candidates_checked": 20,206#   "figures_found": 14207# }208```209 210The agent runs as **Wave 1** (parallel with `ParserAgent`) and is **non-critical** —211if Groq vision fails or the PDF has no visuals, the text pipeline continues unaffected.212 213### Test standalone on any image214 215```python216from PIL import Image217from agents.visual_parser_agent import _img_to_b64, _call_groq_vision, _resize218import os219 220img = _resize(Image.open("your_figure.png"), 1024)221result = _call_groq_vision(_img_to_b64(img), os.getenv("GROQ_API_KEY"))222print(result)223```224 225---226 227## Configuration228 229| Variable | Default | Description |230|---|---|---|231| `GROQ_API_KEY` | required | Groq API key (set in `.env`) |232| `MIN_VECTOR_PATHS` | `8` | Min vector paths to consider a page a diagram candidate |233| `MAX_IMAGES_PER_PDF` | `20` | Max pages analyzed per PDF (cost control) |234| Vision model | `meta-llama/llama-4-scout-17b-16e-instruct` | Groq vision model used |235