PerturbReason/PerturbReason_dataset_code

1# eval_v3 — PerturbReason Evaluation Pipeline2 3## Overview4 5Four-tier evaluation pipeline for PerturbQA reasoning tasks.  6Tier 4 (LLM Rescue) uses a **split workflow**: export hard cases locally → run batch vLLM inference on HPC → merge results back.7 8```9Tier 1  Answer Accuracy      (local, fast)10Tier 2  Symbolic Reasoning   (local, needs --external-kg)11Tier 3  GO Functional Sim.   (local, needs --go-gmt)   ← replaces BLEUrT12Tier 4  LLM Rescue           (server: export → sbatch → merge)13```14 15---16 17## Quick Start — Tiers 1+218 19```bash20python -m eval_v3 eval \21    --gt-dir /path/to/gt_dir \22    --pred-dir /path/to/pred_dir \23    --external-kg omnipath_gene_graph_allinteraction_1229.pkl \24    --tiers 1 225```26 27## Quick Start — Tiers 1+2+3 (with GO Similarity)28 29```bash30python -m eval_v3 eval \31    --gt-dir /path/to/gt_dir/ \32    --pred-dir /path/to/pred_dir/ \33    --external-kg omnipath_gene_graph_allinteraction_1229.pkl \34    --go-gmt eval_v3/c5.go.bp.v2023.2.Hs.symbols.gmt \35    --tiers 1 2 336```37 38GMT file location: `eval_v3/c5.go.bp.v2023.2.Hs.symbols.gmt` (MSigDB v2023.2 Biological Process, 7,647 gene-sets).  39No GPU required — pure CPU, ~7 s total for tier 3 on 40k samples.40 41Results are written to `eval_v3/eval_v3_output/` (timestamped JSON + CSVs).42 43---44 45## Tier-4 Rescue Workflow (3 steps)46 47### Step 1 — Export hard cases48```bash49python -m eval_v3 export-rescue \50    --samples-csv eval_v3/eval_v3_output/<eval_run>/samples_<ts>.csv \51    --pred-dir /path/to/pred_dir52```53 54Output filename is auto-timestamped, e.g. `rescue_prompts_20260324_142501.jsonl`.55The console output prints the exact paths to use in Step 2 and Step 3.56 57Hard cases are samples with error labels in `{CORRECT_HALLUCINATED_EDGE, CORRECT_RIGHT_FOR_WRONG_REASON, WRONG_CORRECT_CHAIN}` plus any parse failures (~42% of total samples).58 59The rationale was:60 61`CORRECT_HALLUCINATED_EDGE` — answer right but reasoning uses hallucinated edges; Tier 1+2 signals conflict → LLM needed to judge whether the reasoning is acceptable62`CORRECT_RIGHT_FOR_WRONG_REASON` — answer right but reasoning doesn't cover the GT path; structurally ambiguous63`WRONG_CORRECT_CHAIN` — answer wrong but the causal chain is valid and grounded in KG; most interesting failure case — LLM might determine the GT label is wrong or the prediction is acceptable64 65 66### Step 2 — Server inference (SLURM)67Copy the timestamped `rescue_prompts_<ts>.jsonl` to the server, update `INPUT_FILE` / `OUTPUT_FILE`68in `qwen_rescue_vllm.sh` (the export step prints the exact names), then:69```bash70sbatch eval_v3/qwen_rescue_vllm.sh71# or directly:72$PYTHON_BIN eval_v3/qwen_rescue_vllm.py \73    --model_path /path/to/Qwen3-4B-Instruct \74    --input_file rescue_prompts_<ts>.jsonl \75    --output_file rescue_responses_<ts>.jsonl \76    --tp_size 277```78 79### Step 3 — Merge responses back80```bash81python -m eval_v3 merge-rescue \82    --samples-csv eval_v3/eval_v3_output/<eval_run>/samples_<ts>.csv \83    --rescue-responses eval_v3/eval_v3_output/<eval_run>/rescue_responses_<ts>.jsonl84```85 86Output CSV and JSON are also auto-timestamped: `samples_with_rescue_<ts>.csv` / `.json`.87 88---89 90## Input Structure91 92Both the GT directory and the prediction directory must share the same **5 split sub-folders**.93Within each split folder there must be exactly **one `*chemical*` JSONL** and **one `*genetic*` JSONL**.94Pairing is done by split + perturbation type — filenames can differ freely between the two directories.95 96```97noisy_input/              ← GT directory (--gt-dir)98  id_test/99    *chemical*.jsonl      ← one chemical file (any name)100    *genetic*.jsonl       ← one genetic file (any name)101  cell_ood/102  pert_ood/103  double_ood/104  other_test/105 106any_pred_folder/          ← Predictions directory (--pred-dir)107  id_test/108    *chemical*.jsonl      ← paired to GT chemical by pert type keyword109    *genetic*.jsonl       ← paired to GT genetic by pert type keyword110  cell_ood/111  pert_ood/112  double_ood/113  other_test/114```115 116Each JSONL line in a prediction file must contain structured output blocks:117 118```119<thinking>...</thinking>120<answer>up / down / unchanged</answer>121<triplet>[[gene_a, sign, gene_b], ...]</triplet>122```123 124---125 126## Tiers127 128| Tier | Name | Key metrics | Requires |129|------|------|-------------|----------|130| 1 | Answer Accuracy | accuracy, balanced accuracy, F1 (macro/weighted) | — |131| 2 | Symbolic Reasoning (M1–M5) | edge F1, path connectivity, sign match, hallucination rate, 10-class error label | `--external-kg` |132| 3 | GO Functional Similarity | IDF-weighted cosine sim of GO-BP profiles (mean/median) | `--go-gmt` |133| 4 | LLM Rescue | rescue verdict, answer correction, label reclassification | offline: `qwen_rescue_vllm.sh` |134 135### M5 Error Taxonomy (Tier 2 output)136 137| Label | Meaning |138|-------|---------|139| `CORRECT` | Answer correct, chain valid, no hallucinations |140| `CORRECT_HALLUCINATED_EDGE` | Correct answer but ≥50% edges not in KG — **hard case** |141| `CORRECT_RIGHT_FOR_WRONG_REASON` | Correct answer but chain coverage too low — **hard case** |142| `CORRECT_SIGN_FLIP` | Correct answer, isolated sign error |143| `WRONG_CORRECT_CHAIN` | Wrong answer despite valid-looking chain — **hard case** |144| `WRONG_HALLUCINATED_EDGE` | Wrong answer + hallucinated edges |145| `WRONG_NO_PATH` | Wrong answer, no valid path found |146| `WRONG_SIGN_FLIP` | Wrong answer due to sign error |147| `PARSE_FAIL` | Could not extract model answer |148| `UNCLASSIFIED` | Wrong answer, no symbolic diagnosis |149 150---151 152## CLI Reference153 154### `eval` subcommand155```156--gt-dir PATH            GT JSONL directory (split structure)157--pred-dir PATH          Prediction JSONL directory158--gt-file PATH           Single GT JSONL file (alternative to --gt-dir)159--pred-file PATH         Single prediction JSONL file160--external-kg PATH       OmniPath KG pickle (for Tier 2)161--tiers {1,2,3}          Which tiers to run (default: 1 2)162--go-gmt PATH            MSigDB GMT file for Tier 3 GO similarity163                         (e.g. eval_v3/c5.go.bp.v2023.2.Hs.symbols.gmt)164-o / --output-dir PATH   Output directory (default: eval_v3/eval_v3_output/)165```166 167### `export-rescue` subcommand168```169--samples-csv PATH       Samples CSV from 'eval' step (required)170--pred-dir PATH          Prediction directory for raw model outputs (required)171-o PATH                  Output JSONL (default: eval_v3/eval_v3_output/rescue_prompts_<ts>.jsonl)172--max-samples INT        Cap number of rescue prompts exported173```174 175### `merge-rescue` subcommand176```177--samples-csv PATH       Original samples CSV from 'eval' step (required)178--rescue-responses PATH  Rescue responses JSONL from server (required)179-o / --output-csv PATH   Updated CSV (default: eval_v3/eval_v3_output/samples_with_rescue.csv)180--summary-json PATH      Updated summary JSON path181```182 183---184 185## Examples186 187**Tiers 1+2 on a single file pair:**188```bash189python -m eval_v3 eval \190    --gt-file noisy_input/id_test/pqa_..._chemical_....jsonl \191    --pred-file noisy_context_output_0322/id_test/qwen_pred_..._chemical_....jsonl \192    --external-kg eval_v3/omnipath_gene_graph_allinteraction_1229.pkl \193    --tiers 1 2194```195 196**Tiers 1+2+3 (with GO Similarity):**197```bash198python -m eval_v3 eval \199    --gt-dir /path/to/noisy_input \200    --pred-dir /path/to/noisy_context_output \201    --external-kg eval_v3/omnipath_gene_graph_allinteraction_1229.pkl \202    --go-gmt eval_v3/c5.go.bp.v2023.2.Hs.symbols.gmt \203    --tiers 1 2 3204```205 206---207 208## Output Files209 210| File | Description |211|------|-------------|212| `summary_<ts>.json` | Per-file aggregated metrics (Tiers 1–3) |213| `samples_<ts>.csv` | Per-sample results (answer, error label, edge F1, go_sim_score, …) |214| `file_summary_<ts>.csv` | One row per file — accuracy, F1, edge metrics, go_sim_mean |215| `rescue_prompts_<ts>.jsonl` | Hard-case prompts for server inference (Step 1 output) |216| `rescue_responses_<ts>.jsonl` | LLM outputs from server (Step 2 output) |217| `samples_with_rescue.csv` | Updated samples after merge (Step 3 output) |218| `samples_with_rescue.json` | Updated summary after merge |219 220---221 222## Understanding the Results223 224### Without Rescue (Tiers 1+2 only)225 226The primary metric from `summary_<ts>.json` is **`accuracy`** — the fraction of samples where the model's extracted `<answer>` (up/down/unchanged) exactly matches the ground truth.227 228Key fields in the summary JSON:229 230```json231{232  "accuracy": 0.413,          // fraction with correct answer box233  "balanced_accuracy": ...,   // macro-averaged over 3 classes234  "f1_macro": ...,            // unweighted mean of per-class F1235  "error_distribution": {236    "CORRECT": 14513,         // answer right, chain valid237    "WRONG_CORRECT_CHAIN": 22155,  // answer wrong, but chain looks valid — the interesting case238    "WRONG_NO_PATH": 1262,    // answer wrong, no valid path found in KG239    "CORRECT_HALLUCINATED_EDGE": ...,240    "CORRECT_RIGHT_FOR_WRONG_REASON": ...241  }242}243```244 245The **M5 error taxonomy** (Tier 2) classifies every sample into one of 10 labels based on the relationship between answer correctness and chain quality.  See the Tiers table above for definitions.246 247`WRONG_CORRECT_CHAIN` is the most important error class: it means the model's causal chain is topologically valid and grounded in the KG, but the final answer label is wrong.  This is ambiguous — it could be a transcription error, an annotation disagreement, or a genuine reasoning failure. LLM rescue (Tier 4) resolves this ambiguity.248 249---250 251### With Rescue (Tier 4 applied)252 253After running `merge-rescue`, the updated `samples_with_rescue.json` carries two distinct accuracy figures:254 255| Metric | Field | What it means |256|--------|-------|---------------|257| **Hard correct rate** | `hard_correct_rate` | Exact answer match (unchanged by rescue). This is the strict accuracy. |258| **Soft correct count** | `soft_correct_count` | Samples where the LLM examiner found the reasoning was actually right despite the wrong answer label. |259| **Effective correct rate** | `effective_correct_rate` | `(hard_correct + soft_correct) / total`. Upper-bound on model quality — how often the reasoning process was sound. |260 261Example from a baseline run:262 263```264Hard correct rate:      41.3%   (16,436 / 39,830)265Soft correct count:     21,469266Effective correct rate: 95.2%   (37,905 / 39,830)267```268 269The large gap between hard and effective rates means the model frequently reasons correctly but writes the wrong label in the `<answer>` box (ANSWER_BUG).  This is a known artifact of instruction-following models on structured output tasks — the chain of thought reaches the right conclusion but the summary label is wrong.270 271New columns added to `samples_with_rescue.csv` after merge:272 273| Column | Values | Meaning |274|--------|--------|---------|275| `llm_rescue_label` | rescue type string | Which rescue type was applied to this sample |276| `llm_rescue_verdict` | see below | Examiner verdict |277| `llm_rescue_soft_correct` | `True` / absent | Set to True when verdict is ANSWER_BUG or ALTERNATE |278| `llm_rescue_explanation` | free text | Examiner's one-sentence justification |279 280---281 282### LLM Rescue — What Happens Inside283 284The rescue LLM (Qwen3-4B) acts as an independent examiner.  It receives the original question, the GT answer, the model's causal chain, and the KG evidence, then returns a structured verdict.  Three rescue types are used:285 286#### `wrong_correct_chain` (the main case, ~92% of rescues)287Applies to: samples labeled `WRONG_CORRECT_CHAIN` by Tier 2 — the model's chain is valid but its answer label is wrong.288 289The examiner compares the model's reasoning path against the GT path and returns one of three verdicts:290 291| Verdict | Meaning | Effect on metrics |292|---------|---------|-------------------|293| **`ANSWER_BUG`** | The causal chain is biologically sound and its logic leads to the **correct** conclusion — the model simply wrote the wrong label in `<answer>`. Example: chain correctly shows inhibition → downregulation, but the answer box says "up". This is a transcription-style error, not a reasoning failure. | Counted as **soft correct**. Sample flagged `llm_rescue_soft_correct = True`. |294| **`CHAIN_BUG`** | The chain contains a logical or biological error that makes the wrong answer follow from flawed premises. The model genuinely reasoned incorrectly. | Sample remains wrong. No change to accuracy. |295| **`ALTERNATE`** | The model identified a **valid alternative pathway** not present in the GT path. Both the GT and the model's chain are defensible given current knowledge. The disagreement is a dataset annotation issue, not a model error. | Counted as **soft correct**. Error label updated to `WRONG_CORRECT_CHAIN_ALTERNATE`. |296 297#### `correct_wrong_reason` (~8% of rescues)298Applies to: samples labeled `CORRECT_RIGHT_FOR_WRONG_REASON` — right answer, but the reasoning chain seems to miss the GT path.299 300The examiner assesses whether the chain is still biologically plausible.301 302| Verdict | Meaning | Effect |303|---------|---------|--------|304| **`YES`** | The chain is plausible despite low KG coverage. The model may be using implicit domain knowledge. | Label updated to `CORRECT_REASONING_PLAUSIBLE`. |305| **`NO`** | The reasoning is weak or coincidental — the model got lucky. | Label stays `CORRECT_RIGHT_FOR_WRONG_REASON`. |306 307#### `parse_fail` (<1% of rescues)308Applies to: samples where Tier 1 could not extract an answer from the model output.309 310The examiner reads the raw response and extracts the intended answer (up/down/unchanged).  If successful, the sample is re-evaluated and can flip from wrong to correct.311 312---313 314### Rescue Verdict Distribution (example run)315 316```317wrong_correct_chain verdicts (22,174 samples):318  ANSWER_BUG:  18,058  (81.4%)  → reasoning right, answer label wrong319  ALTERNATE:    3,411  (15.4%)  → valid alternative path320  CHAIN_BUG:      705  ( 3.2%)  → genuine reasoning error321 322correct_wrong_reason verdicts (1,879 samples):323  NO:   1,620  (86.2%)  → weak/coincidental reasoning324  YES:    259  (13.8%)  → plausible despite low coverage325```326 327The high `ANSWER_BUG` rate (81%) is the key finding: the baseline model almost always reasons correctly through the causal chain but has a systematic tendency to misstate the direction in its final answer box.  This points to a training/instruction-following issue rather than a domain understanding failure.328 329---330 331### Which Number to Report332 333- **For strict benchmarking** (comparison against other models): use `hard_correct_rate`.  This is reproducible without running the LLM rescue.334- **For assessing reasoning quality** (did the model understand the biology?): use `effective_correct_rate` and discuss the ANSWER_BUG breakdown.335- **For error analysis**: use the `error_distribution` in the rescue JSON — specifically the ratio of `WRONG_CORRECT_CHAIN` (chain OK, label wrong) to `WRONG_NO_PATH` (no valid path at all).336 337---338 339## Observed Results (Qwen3 baseline, 39,830 samples, OmniPath KG)340 341> **KG**: `omnipath_gene_graph_allinteraction_1229.pkl` — 19,840 nodes, 284,347 edges.  342> Chemical perturbations have partial KG coverage (In-KG 66–71%); genetic perturbations are almost fully covered (In-KG 98–99%).  343> **GO GMT**: `eval_v3/c5.go.bp.v2023.2.Hs.symbols.gmt` — 17,959 genes, 7,647 BP terms.  344> GO score excludes the perturbation gene and effect gene (shared anchors) to compare only intermediate reasoning nodes.345 346| Split | Pert type | Accuracy | Edge F1 (strict) | In-KG rate | Hallucination rate | GO sim mean |347|-------|-----------|----------|------------------|------------|--------------------|-----------  |348| id_test | chemical | 0.672 | 0.770 | 0.685 | 0.314 | 0.854 |349| id_test | genetic | 0.574 | 0.877 | 0.999 | 0.001 | 0.933 |350| cell_ood | chemical | 0.642 | 0.776 | 0.686 | 0.313 | 0.860 |351| cell_ood | genetic | 0.570 | 0.868 | 1.000 | 0.000 | 0.929 |352| pert_ood | chemical | 0.773 | 0.799 | 0.789 | 0.211 | 0.891 |353| pert_ood | genetic | 0.501 | 0.855 | 0.999 | 0.001 | 0.927 |354| double_ood | chemical | 0.669 | 0.806 | 0.785 | 0.216 | 0.894 |355| double_ood | genetic | 0.545 | 0.904 | 1.000 | 0.000 | 0.942 |356| other_test | chemical | 0.451 | 0.785 | 0.998 | 0.002 | 0.860 |357| other_test | genetic | 0.760 | 0.857 | 0.994 | 0.006 | 0.935 |358 359Hard cases (Tier-4 candidates): **20,377 / 39,830** (51.2%)360 361### Tier 3 — GO Score Interpretation362 363| Error label | GO mean | Interpretation |364|-------------|---------|----------------|365| `CORRECT` | 0.948 | Model uses same intermediate genes as GT |366| `WRONG_CORRECT_CHAIN` | 0.911 | Wrong answer but genes are functionally nearby |367| `CORRECT_HALLUCINATED_EDGE` | 0.926 | Correct answer; hallucinated edges stay on-pathway |368| `WRONG_HALLUCINATED_EDGE` | 0.846 | Hallucinated genes slightly off-pathway |369| `WRONG_NO_PATH` | 0.781 | Structural failure — genuinely different pathway |370| `CORRECT_RIGHT_FOR_WRONG_REASON` | 0.728 | Correct answer but wrong intermediate genes |371 372The 51% of samples scoring exactly 1.0 reflects the model's high edge F1 (≈0.88 genetic) — when the model predicts the same intermediate genes as GT, the GO vectors are identical.  The metric is most informative for the ~49% of samples where chains differ: it captures whether the alternative genes are functionally related to the GT genes.  A score of 0.0–0.5 strongly indicates the model is reasoning in a completely wrong biological neighbourhood.  Pearson r with edge F1 is ~0.86 — correlation is expected and does not indicate redundancy for the structurally-disagreeing cases.373