rolloraq/frantic_ruleriot
0
1{2 "cells": [3 {4 "cell_type": "markdown",5 "id": "a08fede0",6 "metadata": {},7 "source": [8 "# Query Expansion & Re-Ranking"9 ]10 },11 {12 "cell_type": "markdown",13 "id": "08e903b6",14 "metadata": {},15 "source": [16 "In diesem Notebook erweitern und optimieren wir die RAG-Pipeline durch:\n",17 " - Query Expansion (HyDE)\n",18 " - Bulk-Retrieval aller Expansions\n",19 " - Cross-Encoder Re-Ranking\n"20 ]21 },22 {23 "cell_type": "markdown",24 "id": "1d6d0b5a",25 "metadata": {},26 "source": [27 "#### Setup"28 ]29 },30 {31 "cell_type": "code",32 "execution_count": 1,33 "id": "ccb8598a",34 "metadata": {},35 "outputs": [36 {37 "name": "stdout",38 "output_type": "stream",39 "text": [40 "Requirement already satisfied: sentence-transformers in /usr/local/python/3.12.1/lib/python3.12/site-packages (4.1.0)\n",41 "Requirement already satisfied: openai in /usr/local/python/3.12.1/lib/python3.12/site-packages (1.82.0)\n",42 "Requirement already satisfied: faiss-cpu in /usr/local/python/3.12.1/lib/python3.12/site-packages (1.11.0)\n",43 "Requirement already satisfied: transformers<5.0.0,>=4.41.0 in /usr/local/python/3.12.1/lib/python3.12/site-packages (from sentence-transformers) (4.52.3)\n",44 "Requirement already satisfied: tqdm in /usr/local/python/3.12.1/lib/python3.12/site-packages (from sentence-transformers) (4.67.1)\n",45 "Requirement already satisfied: torch>=1.11.0 in /home/codespace/.local/lib/python3.12/site-packages (from sentence-transformers) (2.6.0+cpu)\n",46 "Requirement already satisfied: scikit-learn in /home/codespace/.local/lib/python3.12/site-packages (from sentence-transformers) (1.6.1)\n",47 "Requirement already satisfied: scipy in /home/codespace/.local/lib/python3.12/site-packages (from sentence-transformers) (1.15.2)\n",48 "Requirement already satisfied: huggingface-hub>=0.20.0 in /usr/local/python/3.12.1/lib/python3.12/site-packages (from sentence-transformers) (0.32.2)\n",49 "Requirement already satisfied: Pillow in /home/codespace/.local/lib/python3.12/site-packages (from sentence-transformers) (11.1.0)\n",50 "Requirement already satisfied: typing_extensions>=4.5.0 in /home/codespace/.local/lib/python3.12/site-packages (from sentence-transformers) (4.12.2)\n",51 "Requirement already satisfied: filelock in /home/codespace/.local/lib/python3.12/site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (3.13.1)\n",52 "Requirement already satisfied: numpy>=1.17 in /home/codespace/.local/lib/python3.12/site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (2.2.4)\n",53 "Requirement already satisfied: packaging>=20.0 in /home/codespace/.local/lib/python3.12/site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (24.2)\n",54 "Requirement already satisfied: pyyaml>=5.1 in /home/codespace/.local/lib/python3.12/site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (6.0.2)\n",55 "Requirement already satisfied: regex!=2019.12.17 in /usr/local/python/3.12.1/lib/python3.12/site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (2024.11.6)\n",56 "Requirement already satisfied: requests in /home/codespace/.local/lib/python3.12/site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (2.32.3)\n",57 "Requirement already satisfied: tokenizers<0.22,>=0.21 in /usr/local/python/3.12.1/lib/python3.12/site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (0.21.1)\n",58 "Requirement already satisfied: safetensors>=0.4.3 in /usr/local/python/3.12.1/lib/python3.12/site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (0.5.3)\n",59 "Requirement already satisfied: fsspec>=2023.5.0 in /home/codespace/.local/lib/python3.12/site-packages (from huggingface-hub>=0.20.0->sentence-transformers) (2024.6.1)\n",60 "Requirement already satisfied: hf-xet<2.0.0,>=1.1.2 in /usr/local/python/3.12.1/lib/python3.12/site-packages (from huggingface-hub>=0.20.0->sentence-transformers) (1.1.2)\n",61 "Requirement already satisfied: anyio<5,>=3.5.0 in /home/codespace/.local/lib/python3.12/site-packages (from openai) (4.9.0)\n",62 "Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/python/3.12.1/lib/python3.12/site-packages (from openai) (1.9.0)\n",63 "Requirement already satisfied: httpx<1,>=0.23.0 in /home/codespace/.local/lib/python3.12/site-packages (from openai) (0.28.1)\n",64 "Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/python/3.12.1/lib/python3.12/site-packages (from openai) (0.10.0)\n",65 "Requirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/python/3.12.1/lib/python3.12/site-packages (from openai) (2.11.5)\n",66 "Requirement already satisfied: sniffio in /home/codespace/.local/lib/python3.12/site-packages (from openai) (1.3.1)\n",67 "Requirement already satisfied: idna>=2.8 in /home/codespace/.local/lib/python3.12/site-packages (from anyio<5,>=3.5.0->openai) (3.10)\n",68 "Requirement already satisfied: certifi in /home/codespace/.local/lib/python3.12/site-packages (from httpx<1,>=0.23.0->openai) (2025.1.31)\n",69 "Requirement already satisfied: httpcore==1.* in /home/codespace/.local/lib/python3.12/site-packages (from httpx<1,>=0.23.0->openai) (1.0.7)\n",70 "Requirement already satisfied: h11<0.15,>=0.13 in /home/codespace/.local/lib/python3.12/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai) (0.14.0)\n",71 "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/python/3.12.1/lib/python3.12/site-packages (from pydantic<3,>=1.9.0->openai) (0.7.0)\n",72 "Requirement already satisfied: pydantic-core==2.33.2 in /usr/local/python/3.12.1/lib/python3.12/site-packages (from pydantic<3,>=1.9.0->openai) (2.33.2)\n",73 "Requirement already satisfied: typing-inspection>=0.4.0 in /usr/local/python/3.12.1/lib/python3.12/site-packages (from pydantic<3,>=1.9.0->openai) (0.4.1)\n",74 "Requirement already satisfied: networkx in /home/codespace/.local/lib/python3.12/site-packages (from torch>=1.11.0->sentence-transformers) (3.3)\n",75 "Requirement already satisfied: jinja2 in /home/codespace/.local/lib/python3.12/site-packages (from torch>=1.11.0->sentence-transformers) (3.1.6)\n",76 "Requirement already satisfied: setuptools in /home/codespace/.local/lib/python3.12/site-packages (from torch>=1.11.0->sentence-transformers) (76.0.0)\n",77 "Requirement already satisfied: sympy==1.13.1 in /home/codespace/.local/lib/python3.12/site-packages (from torch>=1.11.0->sentence-transformers) (1.13.1)\n",78 "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/codespace/.local/lib/python3.12/site-packages (from sympy==1.13.1->torch>=1.11.0->sentence-transformers) (1.3.0)\n",79 "Requirement already satisfied: MarkupSafe>=2.0 in /home/codespace/.local/lib/python3.12/site-packages (from jinja2->torch>=1.11.0->sentence-transformers) (3.0.2)\n",80 "Requirement already satisfied: charset-normalizer<4,>=2 in /home/codespace/.local/lib/python3.12/site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers) (3.4.1)\n",81 "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/codespace/.local/lib/python3.12/site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers) (2.3.0)\n",82 "Requirement already satisfied: joblib>=1.2.0 in /home/codespace/.local/lib/python3.12/site-packages (from scikit-learn->sentence-transformers) (1.4.2)\n",83 "Requirement already satisfied: threadpoolctl>=3.1.0 in /home/codespace/.local/lib/python3.12/site-packages (from scikit-learn->sentence-transformers) (3.6.0)\n",84 "Note: you may need to restart the kernel to use updated packages.\n"85 ]86 },87 {88 "name": "stderr",89 "output_type": "stream",90 "text": [91 "/usr/local/python/3.12.1/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",92 " from .autonotebook import tqdm as notebook_tqdm\n"93 ]94 }95 ],96 "source": [97 "%pip install sentence-transformers openai faiss-cpu\n",98 "\n",99 "import os\n",100 "from dotenv import load_dotenv\n",101 "import faiss\n",102 "import pandas as pd\n",103 "import numpy as np\n",104 "from openai import OpenAI\n",105 "from sentence_transformers import CrossEncoder\n"106 ]107 },108 {109 "cell_type": "code",110 "execution_count": 2,111 "id": "747bb1dd",112 "metadata": {},113 "outputs": [],114 "source": [115 "# .env laden und API-Key prüfen\n",116 "load_dotenv()\n",117 "api_key = os.getenv(\"OPENAI_API_KEY\")\n",118 "if not api_key:\n",119 " raise ValueError(\"OPENAI_API_KEY fehlt in der .env!\")\n",120 "client = OpenAI(api_key=api_key)"121 ]122 },123 {124 "cell_type": "code",125 "execution_count": 3,126 "id": "566a5ffa",127 "metadata": {},128 "outputs": [129 {130 "name": "stdout",131 "output_type": "stream",132 "text": [133 "models-Verzeichnis: ['finetuned-frantic-encoder']\n",134 "feintuned-frantic-encoder: ['special_tokens_map.json', 'model.safetensors', 'tokenizer_config.json', 'config.json', 'eval', 'tokenizer.json', 'README.md', 'vocab.txt']\n"135 ]136 }137 ],138 "source": [139 "import os\n",140 "print(\"models-Verzeichnis:\", os.listdir(\"models\"))\n",141 "print(\"feintuned-frantic-encoder:\", os.listdir(\"models/finetuned-frantic-encoder\"))"142 ]143 },144 {145 "cell_type": "code",146 "execution_count": 4,147 "id": "66dca880",148 "metadata": {},149 "outputs": [],150 "source": [151 "# Cross-Encoder für Re-Ranking // wird asukommentiert, da Finetuning aus Notebook 6 geladen wir. Dient zur Dokumentation und Nachvollziehbarkeit\n",152 "#cross_encoder = CrossEncoder(\"cross-encoder/ms-marco-MiniLM-L-6-v2\")\n",153 "\n",154 "# Lade lokal gespeichertes Modell\n",155 "cross_encoder = CrossEncoder(\"models/finetuned-frantic-encoder\", local_files_only=True)"156 ]157 },158 {159 "cell_type": "markdown",160 "id": "0faf0e46",161 "metadata": {},162 "source": [163 "#### 3.1 FAISS-Index & Chunks laden"164 ]165 },166 {167 "cell_type": "code",168 "execution_count": 5,169 "id": "9b4aa4ad",170 "metadata": {},171 "outputs": [172 {173 "name": "stdout",174 "output_type": "stream",175 "text": [176 "Index size: 363, Chunks loaded: 363\n"177 ]178 }179 ],180 "source": [181 "index = faiss.read_index(\"faiss_store/faiss_index.index\")\n",182 "chunks_df = pd.read_pickle(\"faiss_store/chunks_mapping.pkl\")\n",183 "print(f\"Index size: {index.ntotal}, Chunks loaded: {len(chunks_df)}\")"184 ]185 },186 {187 "cell_type": "markdown",188 "id": "83333fbb",189 "metadata": {},190 "source": [191 "#### 3.2 Chunk-Refinement"192 ]193 },194 {195 "cell_type": "code",196 "execution_count": 6,197 "id": "358aa964",198 "metadata": {},199 "outputs": [],200 "source": [201 "# Wir hängen chapter_title/card_title an den Text an, um mehr Signal für den Re-Ranker zu haben\n",202 "\n",203 "chunks_df[\"aug_text\"] = (\n",204 " chunks_df.get(\"chapter_title\", pd.Series(\"\", index=chunks_df.index))\n",205 " + \" – \"\n",206 " + chunks_df[\"text\"]\n",207 ")"208 ]209 },210 {211 "cell_type": "markdown",212 "id": "b2c4fe5e",213 "metadata": {},214 "source": [215 "#### 3.3 HyDE-basierte Query Expansion"216 ]217 },218 {219 "cell_type": "code",220 "execution_count": 7,221 "id": "a6f1b443",222 "metadata": {},223 "outputs": [224 {225 "name": "stdout",226 "output_type": "stream",227 "text": [228 "['Wie viele Karten sind im Basisspiel enthalten?', 'Kannst du mir sagen, wie viele Karten das Basisspiel umfasst?', 'Wie viele Karten beinhaltet das Basisspiel im Ganzen?', 'Könnte ich erfahren, aus wie vielen Karten das Basisspiel besteht?', 'Ich würde gerne wissen, wie viele Karten das Basisspiel hat?', 'Wie ist die genaue Anzahl der Karten im Basisspiel?', 'Kannst du die Anzahl der Karten, die das Basisspiel enthält, präzisieren?']\n"229 ]230 }231 ],232 "source": [233 "def expand_query_hyde(query: str, num_variations: int = 7) -> list[str]:\n",234 " \"\"\"\n",235 " Generiert hypothetische, erweiterte Versionen der Query über GPT-4 (HyDE).\n",236 " \"\"\"\n",237 " prompt = (\n",238 " f\"Du bist ein Such-Query-Generator. Gib mir {num_variations} präzise, \"\n",239 " \"erweiterte Versionen der folgenden Frage, getrennt durch '---'.\\n\\n\"\n",240 " f\"Frage: {query}\\n\\nAntwort:\"\n",241 " )\n",242 " resp = client.chat.completions.create(\n",243 " model=\"gpt-4\",\n",244 " messages=[{\"role\": \"user\", \"content\": prompt}],\n",245 " temperature=0.8,\n",246 " max_tokens=200\n",247 " )\n",248 " text = resp.choices[0].message.content\n",249 " parts = [p.strip() for p in text.split('---') if p.strip()]\n",250 " return parts[:num_variations] or [query]\n",251 "\n",252 "# Test der Expansion\n",253 "print(expand_query_hyde(\"Wie viele Karten hat das Basisspiel?\"))"254 ]255 },256 {257 "cell_type": "markdown",258 "id": "ccf1ecc0",259 "metadata": {},260 "source": [261 "#### 3.4 Test-Set & Ground-Truth laden"262 ]263 },264 {265 "cell_type": "code",266 "execution_count": 8,267 "id": "6606e96c",268 "metadata": {},269 "outputs": [],270 "source": [271 "# Testset laden\n",272 "df_test = pd.read_csv(\"data/testset/test_set.csv\")\n",273 "\n",274 "# Ground Truth korrekt extrahieren – keine Liste, sondern direkt Integer\n",275 "ground_truth = {\n",276 " row.query: int(row.true_ids) if pd.notnull(row.true_ids) else None\n",277 " for row in df_test.itertuples(index=False)\n",278 "}\n",279 "\n",280 "# Metric-Funktionen (einmal definieren)\n",281 "def precision_at_k(pred_ids, true_id, k):\n",282 " return 1.0 if true_id in pred_ids[:k] else 0.0\n",283 "\n",284 "def reciprocal_rank(pred_ids, true_id):\n",285 " if true_id in pred_ids:\n",286 " return 1.0 / (pred_ids.index(true_id) + 1)\n",287 " return 0.0"288 ]289 },290 {291 "cell_type": "markdown",292 "id": "66873274",293 "metadata": {},294 "source": [295 "#### 3.5 Retrieval und Re-Ranking Pipeline"296 ]297 },298 {299 "cell_type": "code",300 "execution_count": 9,301 "id": "27729cb5",302 "metadata": {},303 "outputs": [304 {305 "name": "stdout",306 "output_type": "stream",307 "text": [308 " idx text combined_score\n",309 "0 104 – Die oberste Karte des Kartenstapels wird au... 0.991203\n",310 "1 355 – Vor jeder Runde wird der Regelkarten-Stapel... 0.813337\n",311 "2 276 – Deckt für alle Mitspielenden je eine Karte\\... 0.793418\n",312 "3 193 – Du kannst diese Karten nur auf eine der\\nzw... 0.777171\n",313 "4 44 – oberste Karte vom Ablagestapel bis alle tot... 0.764544\n",314 "\n",315 "Simple Top-5 Treffer (IDs & Text):\n",316 "- ID 359: – FRANTIC GRUNDSPIEL\n",317 "125 Spielkarten\n",318 "(schwarze Rückseite)\n",319 "81 Zahlenkarten\n",320 "· 18 blaue Karten (1 bis 9)\n",321 "· 18 rote Karten (1 bis 9)\n",322 "· 18 grüne Karten (1…\n",323 "- ID 159: – 125 SPIELKARTEN (SCHWARZE RÜCKSEITE)\n",324 "18 blaue Karten (1 bis 9)\n",325 "18 rote Karten (1 bis 9)\n",326 "18 grüne Karten (1 bis 9)\n",327 "18 gelbe Karten (1 bis 9)\n",328 "9 Schwa…\n",329 "- ID 103: – Zu Beginn des Spiels werden die Spielkarten gemischt und\n",330 "alle Spieler erhalten sieben Karten. Die restlichen Karten\n",331 "werden verdeckt in die Mitte ge…\n",332 "- ID 101: – wir diese Erweiterung nur mit dem Basisspiel oder mit\n",333 "Basisspiel & Troublemaker zu spielen.…\n",334 "- ID 280: – der Reihe nach drei neue Karten vom\n",335 "Stapel.…\n"336 ]337 }338 ],339 "source": [340 "def retrieve_rerank(\n",341 " query: str,\n",342 " top_n: int = 20,\n",343 " final_k: int = 5,\n",344 " alpha: float = 0.85\n",345 ") -> pd.DataFrame:\n",346 " # 1. Expansion\n",347 " expansions = expand_query_hyde(query)\n",348 "\n",349 " # 2. Bulk-Retrieval mit Hard-Negatives\n",350 " hard_negs = [51, 44, 243]\n",351 " records = [{\"idx\": hn, \"faiss_dist\": 1e6} for hn in hard_negs]\n",352 "\n",353 " # 3. Bulk-Retrieval für alle Expansions\n",354 " for q in expansions:\n",355 " emb = client.embeddings.create(\n",356 " model=\"text-embedding-ada-002\",\n",357 " input=[q]\n",358 " ).data[0].embedding\n",359 " dists, idxs = index.search(\n",360 " np.array([emb], dtype=np.float32),\n",361 " top_n\n",362 " )\n",363 " for dist, idx in zip(dists[0], idxs[0]):\n",364 " records.append({\"idx\": int(idx), \"faiss_dist\": float(dist)})\n",365 "\n",366 " # 4. Kandidaten-Pool deduplizieren\n",367 " df = pd.DataFrame(records).drop_duplicates(\"idx\").reset_index(drop=True)\n",368 " df[\"text\"] = df[\"idx\"].apply(lambda i: chunks_df.iloc[i][\"aug_text\"])\n",369 "\n",370 " # 5. Cross-Encoder-Scoring\n",371 " pairs = [[query, txt] for txt in df[\"text\"]]\n",372 " df[\"ce_score\"] = cross_encoder.predict(pairs)\n",373 "\n",374 " # 6. FAISS-Similarity\n",375 " df[\"faiss_sim\"] = 1.0 / (1.0 + df[\"faiss_dist\"])\n",376 "\n",377 " # 7. Min-Max-Normierung\n",378 " df[\"faiss_norm\"] = (\n",379 " df[\"faiss_sim\"] - df[\"faiss_sim\"].min()\n",380 " ) / (df[\"faiss_sim\"].max() - df[\"faiss_sim\"].min() + 1e-9)\n",381 " df[\"ce_norm\"] = (\n",382 " df[\"ce_score\"] - df[\"ce_score\"].min()\n",383 " ) / (df[\"ce_score\"].max() - df[\"ce_score\"].min() + 1e-9)\n",384 "\n",385 " # 8. Kombination & Top-k Auswahl\n",386 " df[\"combined_score\"] = (\n",387 " alpha * df[\"ce_norm\"] + (1 - alpha) * df[\"faiss_norm\"]\n",388 " )\n",389 " topk = df.sort_values(\"combined_score\", ascending=False).head(final_k)\n",390 " return topk[[\"idx\", \"text\", \"combined_score\"]].reset_index(drop=True)\n",391 "\n",392 "\n",393 "def retrieve_simple_ids(query: str, k: int = 5) -> list[int]:\n",394 " emb = client.embeddings.create(\n",395 " model=\"text-embedding-ada-002\",\n",396 " input=[query]\n",397 " ).data[0].embedding\n",398 " dists, idxs = index.search(\n",399 " np.array([emb], dtype=np.float32),\n",400 " k\n",401 " )\n",402 " return idxs[0].tolist()\n",403 "\n",404 "\n",405 "# Test der neuen Pipeline\n",406 "print(retrieve_rerank(\n",407 " \"Wie viele Karten hat das Basisspiel?\",\n",408 " top_n=15,\n",409 " final_k=5,\n",410 " alpha=0.85\n",411 "))\n",412 "\n",413 "print(\"\\nSimple Top-5 Treffer (IDs & Text):\")\n",414 "for idx in retrieve_simple_ids(\"Wie viele Karten hat das Basisspiel?\", k=5):\n",415 " print(f\"- ID {idx}: {chunks_df.iloc[idx]['aug_text'][:150]}…\")\n"416 ]417 },418 {419 "cell_type": "markdown",420 "id": "a4e9f2d5",421 "metadata": {},422 "source": [423 "#### 3.6 α-Sweep: besten α-Wert finden"424 ]425 },426 {427 "cell_type": "code",428 "execution_count": 10,429 "id": "d889ab4f",430 "metadata": {},431 "outputs": [432 {433 "name": "stdout",434 "output_type": "stream",435 "text": [436 " P@1_baseline P@1_finetuned MRR_baseline MRR_finetuned\n",437 "alpha \n",438 "0.6 1.0 0.2 1.0 0.2\n",439 "0.7 1.0 0.2 1.0 0.2\n",440 "0.8 1.0 0.2 1.0 0.2\n",441 "0.9 1.0 0.2 1.0 0.2\n"442 ]443 }444 ],445 "source": [446 "alphas = [0.6, 0.7, 0.8, 0.9]\n",447 "sweep_results = []\n",448 "\n",449 "for alpha in alphas:\n",450 " metrics = {\"alpha\": alpha, \"P@1_baseline\":0, \"P@1_finetuned\":0,\n",451 " \"MRR_baseline\":0, \"MRR_finetuned\":0}\n",452 " count = 0\n",453 "\n",454 " for q in df_test[\"query\"].head(5):\n",455 " true_id = ground_truth[q]\n",456 " if true_id is None:\n",457 " continue\n",458 " count += 1\n",459 "\n",460 " base_ids = retrieve_simple_ids(q, k=5)\n",461 " ft_ids = retrieve_rerank(q, top_n=20, final_k=5, alpha=alpha)[\"idx\"].tolist()\n",462 "\n",463 " # P@1 und RR pro Frage\n",464 " p1_base = precision_at_k(base_ids, true_id, 1)\n",465 " p1_ft = precision_at_k(ft_ids, true_id, 1)\n",466 " rr_base = reciprocal_rank(base_ids, true_id)\n",467 " rr_ft = reciprocal_rank(ft_ids, true_id)\n",468 "\n",469 " metrics[\"P@1_baseline\"] += p1_base\n",470 " metrics[\"P@1_finetuned\"] += p1_ft\n",471 " metrics[\"MRR_baseline\"] += rr_base\n",472 " metrics[\"MRR_finetuned\"] += rr_ft\n",473 "\n",474 " # Durchschnitt über alle Fragen\n",475 " for k in [\"P@1_baseline\",\"P@1_finetuned\",\"MRR_baseline\",\"MRR_finetuned\"]:\n",476 " metrics[k] /= count\n",477 "\n",478 " sweep_results.append(metrics)\n",479 "\n",480 "sweep_df = pd.DataFrame(sweep_results)\n",481 "print(pd.DataFrame(sweep_results).set_index(\"alpha\"))"482 ]483 },484 {485 "cell_type": "markdown",486 "id": "1c4a460d",487 "metadata": {},488 "source": [489 "### 3.7 Evaluation: Precision@k & MRR"490 ]491 },492 {493 "cell_type": "code",494 "execution_count": 11,495 "id": "0ca08f87",496 "metadata": {},497 "outputs": [498 {499 "data": {500 "text/html": [501 "<div>\n",502 "<style scoped>\n",503 " .dataframe tbody tr th:only-of-type {\n",504 " vertical-align: middle;\n",505 " }\n",506 "\n",507 " .dataframe tbody tr th {\n",508 " vertical-align: top;\n",509 " }\n",510 "\n",511 " .dataframe thead tr th {\n",512 " text-align: left;\n",513 " }\n",514 "</style>\n",515 "<table border=\"1\" class=\"dataframe\">\n",516 " <thead>\n",517 " <tr>\n",518 " <th></th>\n",519 " <th>query</th>\n",520 " <th>k</th>\n",521 " <th colspan=\"2\" halign=\"left\">mrr</th>\n",522 " <th colspan=\"2\" halign=\"left\">precision</th>\n",523 " </tr>\n",524 " <tr>\n",525 " <th>method</th>\n",526 " <th></th>\n",527 " <th></th>\n",528 " <th>baseline</th>\n",529 " <th>fine-tuned</th>\n",530 " <th>baseline</th>\n",531 " <th>fine-tuned</th>\n",532 " </tr>\n",533 " </thead>\n",534 " <tbody>\n",535 " <tr>\n",536 " <th>0</th>\n",537 " <td>Wann endet eine Spielrunde?</td>\n",538 " <td>1</td>\n",539 " <td>1.0</td>\n",540 " <td>0.00</td>\n",541 " <td>1.0</td>\n",542 " <td>0.0</td>\n",543 " </tr>\n",544 " <tr>\n",545 " <th>1</th>\n",546 " <td>Wann endet eine Spielrunde?</td>\n",547 " <td>3</td>\n",548 " <td>1.0</td>\n",549 " <td>0.00</td>\n",550 " <td>1.0</td>\n",551 " <td>0.0</td>\n",552 " </tr>\n",553 " <tr>\n",554 " <th>2</th>\n",555 " <td>Wann endet eine Spielrunde?</td>\n",556 " <td>5</td>\n",557 " <td>1.0</td>\n",558 " <td>0.00</td>\n",559 " <td>1.0</td>\n",560 " <td>0.0</td>\n",561 " </tr>\n",562 " <tr>\n",563 " <th>3</th>\n",564 " <td>Was bewirkt die Karte DOUBLEMAKER?</td>\n",565 " <td>1</td>\n",566 " <td>1.0</td>\n",567 " <td>0.00</td>\n",568 " <td>1.0</td>\n",569 " <td>0.0</td>\n",570 " </tr>\n",571 " <tr>\n",572 " <th>4</th>\n",573 " <td>Was bewirkt die Karte DOUBLEMAKER?</td>\n",574 " <td>3</td>\n",575 " <td>1.0</td>\n",576 " <td>0.00</td>\n",577 " <td>1.0</td>\n",578 " <td>0.0</td>\n",579 " </tr>\n",580 " <tr>\n",581 " <th>...</th>\n",582 " <td>...</td>\n",583 " <td>...</td>\n",584 " <td>...</td>\n",585 " <td>...</td>\n",586 " <td>...</td>\n",587 " <td>...</td>\n",588 " </tr>\n",589 " <tr>\n",590 " <th>115</th>\n",591 " <td>Wie werden Powerkarten aktiviert?</td>\n",592 " <td>3</td>\n",593 " <td>1.0</td>\n",594 " <td>0.25</td>\n",595 " <td>1.0</td>\n",596 " <td>0.0</td>\n",597 " </tr>\n",598 " <tr>\n",599 " <th>116</th>\n",600 " <td>Wie werden Powerkarten aktiviert?</td>\n",601 " <td>5</td>\n",602 " <td>1.0</td>\n",603 " <td>0.25</td>\n",604 " <td>1.0</td>\n",605 " <td>1.0</td>\n",606 " </tr>\n",607 " <tr>\n",608 " <th>117</th>\n",609 " <td>Wie wird das Basisspiel vorbereitet?</td>\n",610 " <td>1</td>\n",611 " <td>1.0</td>\n",612 " <td>0.00</td>\n",613 " <td>1.0</td>\n",614 " <td>0.0</td>\n",615 " </tr>\n",616 " <tr>\n",617 " <th>118</th>\n",618 " <td>Wie wird das Basisspiel vorbereitet?</td>\n",619 " <td>3</td>\n",620 " <td>1.0</td>\n",621 " <td>0.00</td>\n",622 " <td>1.0</td>\n",623 " <td>0.0</td>\n",624 " </tr>\n",625 " <tr>\n",626 " <th>119</th>\n",627 " <td>Wie wird das Basisspiel vorbereitet?</td>\n",628 " <td>5</td>\n",629 " <td>1.0</td>\n",630 " <td>0.00</td>\n",631 " <td>1.0</td>\n",632 " <td>0.0</td>\n",633 " </tr>\n",634 " </tbody>\n",635 "</table>\n",636 "<p>120 rows × 6 columns</p>\n",637 "</div>"638 ],639 "text/plain": [640 " query k mrr precision \\\n",641 "method baseline fine-tuned baseline \n",642 "0 Wann endet eine Spielrunde? 1 1.0 0.00 1.0 \n",643 "1 Wann endet eine Spielrunde? 3 1.0 0.00 1.0 \n",644 "2 Wann endet eine Spielrunde? 5 1.0 0.00 1.0 \n",645 "3 Was bewirkt die Karte DOUBLEMAKER? 1 1.0 0.00 1.0 \n",646 "4 Was bewirkt die Karte DOUBLEMAKER? 3 1.0 0.00 1.0 \n",647 ".. ... .. ... ... ... \n",648 "115 Wie werden Powerkarten aktiviert? 3 1.0 0.25 1.0 \n",649 "116 Wie werden Powerkarten aktiviert? 5 1.0 0.25 1.0 \n",650 "117 Wie wird das Basisspiel vorbereitet? 1 1.0 0.00 1.0 \n",651 "118 Wie wird das Basisspiel vorbereitet? 3 1.0 0.00 1.0 \n",652 "119 Wie wird das Basisspiel vorbereitet? 5 1.0 0.00 1.0 \n",653 "\n",654 " \n",655 "method fine-tuned \n",656 "0 0.0 \n",657 "1 0.0 \n",658 "2 0.0 \n",659 "3 0.0 \n",660 "4 0.0 \n",661 ".. ... \n",662 "115 0.0 \n",663 "116 1.0 \n",664 "117 0.0 \n",665 "118 0.0 \n",666 "119 0.0 \n",667 "\n",668 "[120 rows x 6 columns]"669 ]670 },671 "metadata": {},672 "output_type": "display_data"673 }674 ],675 "source": [676 "results = []\n",677 "for q in df_test[\"query\"]:\n",678 " true_id = ground_truth[q]\n",679 " if true_id is None:\n",680 " continue\n",681 "\n",682 " baseline_ids = retrieve_simple_ids(q, k=10)\n",683 " finetuned_ids = retrieve_rerank(q, top_n=20, final_k=10, alpha=0.85)[\"idx\"].tolist()\n",684 "\n",685 " for k in [1, 3, 5]:\n",686 " results.append({\n",687 " \"query\": q,\n",688 " \"method\": \"baseline\",\n",689 " \"k\": k,\n",690 " \"precision\": precision_at_k(baseline_ids, true_id, k),\n",691 " \"mrr\": reciprocal_rank(baseline_ids, true_id)\n",692 " })\n",693 " results.append({\n",694 " \"query\": q,\n",695 " \"method\": \"fine-tuned\",\n",696 " \"k\": k,\n",697 " \"precision\": precision_at_k(finetuned_ids, true_id, k),\n",698 " \"mrr\": reciprocal_rank(finetuned_ids, true_id)\n",699 " })\n",700 "\n",701 "eval_df = pd.DataFrame(results)\n",702 "pivot = eval_df.pivot_table(\n",703 " index=[\"query\", \"k\"],\n",704 " columns=\"method\",\n",705 " values=[\"precision\", \"mrr\"]\n",706 ").reset_index()\n",707 "\n",708 "display(pivot)"709 ]710 },711 {712 "cell_type": "markdown",713 "id": "f0100e27",714 "metadata": {},715 "source": [716 "#### 3.8 Baseline-Vergleich"717 ]718 },719 {720 "cell_type": "code",721 "execution_count": 12,722 "id": "b5ac3649",723 "metadata": {},724 "outputs": [725 {726 "name": "stdout",727 "output_type": "stream",728 "text": [729 "\n",730 "Simple Top-5 Treffer:\n",731 "- FRANTIC GRUNDSPIEL\n",732 "125 Spielkarten\n",733 "(schwarze Rückseite)\n",734 "81 Zahlenkarten\n",735 "· 18 blaue Karten (1 bis 9)\n",736 "· 18 rote Karten (1 bis 9)\n",737 "· 18 grüne Karten (1 bi …\n",738 "- 125 SPIELKARTEN (SCHWARZE RÜCKSEITE)\n",739 "18 blaue Karten (1 bis 9)\n",740 "18 rote Karten (1 bis 9)\n",741 "18 grüne Karten (1 bis 9)\n",742 "18 gelbe Karten (1 bis 9)\n",743 "9 Schwarze …\n",744 "- Zu Beginn des Spiels werden die Spielkarten gemischt und\n",745 "alle Spieler erhalten sieben Karten. Die restlichen Karten\n",746 "werden verdeckt in die Mitte geleg …\n",747 "- wir diese Erweiterung nur mit dem Basisspiel oder mit\n",748 "Basisspiel & Troublemaker zu spielen. …\n",749 "- der Reihe nach drei neue Karten vom\n",750 "Stapel. …\n"751 ]752 }753 ],754 "source": [755 "def retrieve_simple(query: str, k: int = 5) -> list[str]:\n",756 " emb = client.embeddings.create(model=\"text-embedding-ada-002\", input=[query]).data[0].embedding\n",757 " dists, idxs = index.search(np.array([emb], dtype=np.float32), k)\n",758 " return [chunks_df.iloc[i][\"text\"] for i in idxs[0]]\n",759 "\n",760 "print(\"\\nSimple Top-5 Treffer:\")\n",761 "for t in retrieve_simple(\"Wie viele Karten hat das Basisspiel?\"):\n",762 " print(\"-\", t[:150], \"…\")"763 ]764 }765 ],766 "metadata": {767 "kernelspec": {768 "display_name": "Python 3",769 "language": "python",770 "name": "python3"771 },772 "language_info": {773 "codemirror_mode": {774 "name": "ipython",775 "version": 3776 },777 "file_extension": ".py",778 "mimetype": "text/x-python",779 "name": "python",780 "nbconvert_exporter": "python",781 "pygments_lexer": "ipython3",782 "version": "3.12.1"783 }784 },785 "nbformat": 4,786 "nbformat_minor": 5787}788