Datasetpublic

CHML-real/CHML-real-ecp-local-llm-audit-benchmark

Links GitHub: https://github.com/CHML-real/CHML-real-ecp-local-llm-audit-benchmark Hugging Face Dataset: https://huggingface.co/datasets/CHML-real/CHML-real-ecp-local-llm-audit-benchmark ECP Local LLM Multi-hop Audit Benchmark Local-only benchmark suite for evaluating Evidence Confidence Propagation (ECP) as an audit layer for multi-hop reasoning chains. This repository is designed for the CHML-real GitHub namespace and uses only local execution. The LLM… See the full description on the dataset page: https://huggingface.co/datasets/CHML-real/CHML-real-ecp-local-llm-audit-benchmark.

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes260downloads

Dataset Card

ECP Local LLM Multi-hop Audit Benchmark

Local-only benchmark suite for evaluating Evidence Confidence Propagation (ECP) as an audit layer for multi-hop reasoning chains.

This repository is designed for the CHML-real GitHub namespace and uses only local execution. The LLM experiments use Ollama; no external API keys, no remote datasets, and no prompt/raw output persistence are required by default.

What this is

—A reproducible benchmark suite for evidence-confidence-propagation.
—A local Ollama-based multi-hop QA and evidence-chain audit pipeline.
—A weak/zero bridge gating study for suppressing false confidence in multi-hop chains.
—A research prototype for local LLM evidence auditing.

What this is not

—It is not a universal hallucination detector.
—It is not a replacement for retrieval or factual source checking.
—It does not claim that ECP is the best general-purpose QA confidence score.
—It is primarily an audit layer for cases where intermediate evidence is weak or missing.

Core conclusion

ECP is not a universal replacement for local LLM self-confidence or simple QA confidence baselines. In easy QA settings, self-confidence or mean evidence can outperform ECP. However, under adversarial multi-hop bridge conditions where intermediate evidence is weak or missing, ECP-style weak-bridge gating suppresses false confidence more effectively than simple baselines.

Main experiments

Experiment	Purpose
Exp2	Local LLM multi-hop QA baseline
Exp6	Adversarial unsupported bridge test
Exp10	Explicit weak/zero bridge gating
Exp11	Partial weak bridge threshold sweep
Exp13	Single-pass LLM-generated evidence-chain QA
Exp17	Claim decomposition + separate verification
Exp18	Self-consistency bridge audit
Exp19	Counterfactual bridge challenge
Exp20	External-free answerer/verifier mode

Diagnostics and mechanism ablations from Exp1, Exp3, Exp4, Exp5, Exp7, Exp8, Exp9, and Exp12 are retained for reproducibility.

Quick start

bash

cd ~/Desktop/CHMLabs

git clone https://github.com/CHML-real/ecp-local-llm-audit-benchmark.git
cd ecp-local-llm-audit-benchmark

source ../venv/bin/activate
pip install -r requirements.txt

Check Ollama:

bash

ollama list
curl http://localhost:11434/api/tags

Run a fast local LLM audit:

bash

export OLLAMA_MODEL="qwen25-coder-14b-local:latest"
export ECP_OLLAMA_TIMEOUT=120
export ECP_NUM_PREDICT=96

python run_all.py \
  --profile quick \
  --exp2-limit 4 \
  --exp13-limit 4 \
  --exp17-limit 4 \
  --exp18-limit 3 \
  --exp18-samples 3 \
  --exp19-limit 4 \
  --exp20-limit 3

Role-split verifier mode:

bash

export ECP_ANSWERER_MODEL="qwen25-coder-14b-local:latest"
export ECP_VERIFIER_MODEL="gemma4-coding-12b-local:latest"

python run_all.py --profile quick --exp17-limit 4 --exp20-limit 3

Deterministic-only run:

bash

python run_all.py --profile quick \
  --skip-exp2 --skip-exp13 --skip-exp15 \
  --skip-exp17 --skip-exp18 --skip-exp19 --skip-exp20 \
  --skip-3d

Open the report:

bash

google-chrome \
  --ignore-gpu-blocklist \
  --use-gl=swiftshader \
  --enable-webgl \
  --enable-unsafe-swiftshader \
  results/advanced_report.html

Recommended final public framing

Use this repository as a benchmark and audit prototype, not as an overclaimed proof of universal hallucination detection.

ECP-style weak-bridge gating is useful as a local LLM audit layer for multi-hop reasoning when intermediate evidence is weak or missing.

Repository status

—Target GitHub owner: CHML-real
—Recommended repo name: ecp-local-llm-audit-benchmark
—Suggested first tag: v0.1.0-local-audit
—Hugging Face release: planned after the GitHub benchmark is stable

Reports

A quick v8 report example is included in reports/v8_quick_report_2026-07-05.md. Treat it as a smoke-test result, not the final paper-scale result.

Visual Results Gallery

This repository includes reproducible visual outputs under results/figures/. These figures summarize the main benchmark finding: ECP is not a universal replacement for local LLM self-confidence, but ECP-style weak-bridge gating is useful as an audit layer when intermediate evidence is weak or missing.

Main Bridge-Audit Results

Exp6 — Adversarial Unsupported Bridge

ECP-style propagation is most useful when an otherwise high-confidence multi-hop chain contains a weak or unsupported internal bridge.

[image]

Interactive HTML:

—Exp6 adversarial objective
—Exp6 suppression gain
—Exp6 clean retention
—Exp6 score separation

Exp10 — Explicit Weak/Zero Bridge Gating

Explicit bridge gates improve confidence suppression compared with convex no-gate scoring.

[image]

Exp11 — Partial Weak-Bridge Thresholds

The benchmark repeatedly shows that weak-bridge thresholding is stronger than zero-evidence-only gating. In the current runs, thresholds around 0.20–0.30 are the most effective.

[image]

Exp12 — Mask / Gate Comparison

Soft and hard bridge gates are compared against convex no-gate scoring and oracle-style masks.

[image]

Local LLM QA and Calibration Figures

Exp2 — Local LLM Multi-hop QA

Exp2 evaluates local Ollama QA outputs and compares self-confidence, simple baselines, and ECP-derived confidence scores.

[image]

Diagnostic Figures

Exp1 — Synthetic Chain Confidence Behavior

[image]

Interactive HTML:

—Exp1 3D confidence trajectories
—Exp1 3D confidence surface
—Exp1 3D component surface

Exp3 — Unsupported Hop Detection

The benchmark uses zero_evidence_flag or pole_sources containing zero_evidence for unsupported-hop bookkeeping. The broader is_singular flag is useful diagnostically but includes root-boundary poles and should not be used as the clean unsupported-hop detector.

[image]

Interactive HTML:

—Exp3 unsupported hop separation
—Exp3 confidence valleys

Exp4 / Exp5 — Parameter and Pole-Source Diagnostics

[image]

Interactive HTML:

—Exp4 calibrated confidence landscape
—Exp4 floor-collapse landscape
—Exp4 ECP minus baseline landscape
—Exp4 penalty sensitivity
—Exp5 pole-source ablation

Exp8 / Exp9 — Bridge Difficulty and Selective Prediction

[image]

Interactive HTML:

—Exp8 ECP bridge difficulty
—Exp8 suppression landscape

Full Report

The complete generated report is included here:

—Advanced HTML report
—v8 quick report markdown

The HTML report contains the full experiment tables, generated figures, and local-run metadata.