Akshanshsensei/PDF-Constrained-Conversational-Agent

1# Evaluator Instructions — PDF-Constrained Conversational Agent2 3## Overview4 5This document provides structured test cases for evaluating the PDF-Constrained Conversational Agent.6Upload the included **`sample.pdf`** ("Global Climate Change Report 2024") to the agent interface before running any of the queries below.7 8**Sample PDF:** `sample.pdf` — A 5-page technical report by the International Climate Research Consortium (ICRC), covering causes, impacts, and mitigation strategies for global climate change.9 10---11 12## How to Run the Agent13 141. Launch the agent: `python app.py`152. Open your browser at `http://127.0.0.1:7860`163. Upload `sample.pdf` using the "Upload PDF" panel on the left174. Wait for the ✅ confirmation message185. Begin asking the queries below in the chat box19 20---21 22## ✅ Valid Queries (Agent Must Answer with Citation)23 24These queries are **fully answerable** from the PDF. The agent must produce an accurate response ending with a `[Source: Page X]` citation.25 26---27 28### Query 129**Input:** `What is the current concentration of CO2 in the atmosphere?`30 31**Expected behavior:** The agent identifies the specific statistic from Page 1 of the document.32 33**Expected response (approximate):**34> The atmospheric CO2 concentration has reached **421 parts per million (ppm)**, which is the highest level in over 800,000 years. [Source: Page 1]35 36**Grading criteria:** Exact figure (421 ppm), correct citation.37 38---39 40### Query 241**Input:** `What are the two main factors driving rising sea levels?`42 43**Expected behavior:** The agent retrieves the precise breakdown from Section 2.1 (Page 3).44 45**Expected response (approximate):**46> Rising sea levels are driven by two main factors: **thermal expansion** of warming ocean waters (accounting for 42% of the observed rise), and the **melting of land-based ice sheets and glaciers** (accounting for 58%). [Source: Page 3]47 48**Grading criteria:** Both factors named, percentages accurate, correct citation.49 50---51 52### Query 353**Input:** `By how much has the cost of solar PV fallen over the last decade?`54 55**Expected behavior:** The agent retrieves the clean statistic from Section 3.1 (Page 4).56 57**Expected response (approximate):**58> The cost of solar photovoltaic (PV) energy has fallen by **89%** over the past decade, making it one of the cheapest sources of new electricity generation in most of the world. [Source: Page 4]59 60**Grading criteria:** Correct percentage (89%), correct citation.61 62---63 64### Query 465**Input:** `How many countries have carbon pricing initiatives in place, and what percentage of global emissions do they cover?`66 67**Expected behavior:** The agent retrieves the multi-part statistic from Section 3.2 (Page 4).68 69**Expected response (approximate):**70> As of 2024, **73 carbon pricing initiatives** are in place globally, covering approximately **23% of global greenhouse gas emissions**. [Source: Page 4]71 72**Grading criteria:** Both figures correct (73 initiatives, 23%), correct citation.73 74---75 76### Query 5 (Follow-up / Conversational Memory Test)77*Ask Query 4 first, then immediately ask this follow-up:*78 79**Input:** `What are the two forms of carbon pricing mentioned?`80 81**Expected behavior:** The agent uses its conversational memory to understand this is a follow-up question about the carbon pricing topic, and retrieves the two specific forms from the same section.82 83**Expected response (approximate):**84> There are two main forms of carbon pricing: a **carbon tax**, which sets a direct price on emissions, and an **emissions trading system (ETS)** (also known as cap-and-trade), which sets a cap on total emissions and allows entities to buy and sell allowances. [Source: Page 4]85 86**Grading criteria:** Both forms named correctly, demonstrates memory of prior conversational context.87 88### Query 6 (Agentic Tool-Use)89**Input:** `Exactly how many times does the word 'climate' appear in this document?`90 91**Expected behavior:** The agent invokes the `count_word` Python tool. The UI briefly shows `🔧 Running tool: count_word...` before the LLM returns the exact mathematical count.92 93**Grading criteria:** Exact word count derived from Python execution, not hallucinated by token prediction.94 95---96 97## ❌ Invalid / Out-of-Scope Queries (Agent Must Refuse)98 99These queries **cannot be answered** from the PDF. The agent must explicitly refuse and not hallucinate an answer.100 101---102 103### Invalid Query 1104**Input:** `Who is the current President of the United States?`105 106**Expected behavior:** The agent refuses, as this is a general knowledge question with no relation to the document's content.107 108**Expected response (approximate):**109> I cannot answer that based on the provided document.110 111**Anti-hallucination check:** The agent must NOT name any politician or make any inference. A failure here constitutes a critical hallucination.112 113---114 115### Invalid Query 2116**Input:** `What is the GDP of India in 2024?`117 118**Expected behavior:** The agent refuses. The document discusses climate statistics, not economic indicators for individual countries.119 120**Expected response (approximate):**121> I cannot answer that based on the provided document.122 123**Anti-hallucination check:** The agent must not confuse climate-related economic data (carbon pricing coverage) with national GDP figures.124 125---126 127### Invalid Query 3128**Input:** `Write me a Python script to calculate carbon emissions.`129 130**Expected behavior:** The agent refuses to generate code. This is a request for creative/technical generation, not an information retrieval task from the document.131 132**Expected response (approximate):**133> I cannot answer that based on the provided document.134 135**Anti-hallucination check:** The agent must NOT produce any code. This tests whether the LLM can resist instruction-following requests that bypass its grounding constraints.136 137---138 139## 🌐 Bonus: Multilingual Test Queries140 141The agent's primary embedding model (Gemini) and reranker (BAAI/bge-reranker-m3) are multilingual. These queries test cross-lingual retrieval — asking a question in a different language about content that exists in both English and the respective language on Page 5.142 143---144 145### Multilingual Query 1 (Hindi)146**Input:** `CO2 की वायुमंडलीय सांद्रता कितनी है?`147*(Translation: "What is the atmospheric concentration of CO2?")*148 149**Expected behavior:** The agent retrieves the relevant chunk (Page 1 has the figure in English; Page 5 has a Hindi summary), and responds with the correct figure.150 151**Expected response (approximate):**152> वायुमंडल में CO2 की सांद्रता **421 ppm** तक पहुँच गई है। [Source: Page 1]153> *(or an equivalent English response with the correct data)*154 155---156 157### Multilingual Query 2 (French)158**Input:** `Quel est l'objectif de l'Accord de Paris concernant la température?`159*(Translation: "What is the Paris Agreement's goal regarding temperature?")*160 161**Expected behavior:** The agent retrieves the relevant information from Section 4.1 / Page 5 and responds correctly.162 163**Expected response (approximate):**164> L'Accord de Paris vise à limiter le réchauffement climatique à **bien en dessous de 2 degrés Celsius** et à poursuivre des efforts pour le limiter à **1,5 degré Celsius**. [Source: Page 5]165> *(or an accurate English translation of the same)*166 167---168 169## Evaluation Rubric Summary170 171| Criterion | Test Cases | What to Look For |172|---|---|---|173| **Accuracy** | Valid Q1–Q4 | Exact figures match the PDF |174| **Conversational Memory** | Valid Q5 | Correct follow-up without re-specifying topic |175| **Hallucination Robustness** | Invalid Q1–Q3 | Clean refusal, no fabricated data |176| **Citation Quality** | All valid queries | Every answer ends with `[Source: Page X]` |177| **Multilingual Grounding** | Bonus Q1–Q2 | Correct answer from non-English query |178