LiamDuero/telco-analysis

telco-analysis

Aggregate (not per-question) evaluation results for my MSc thesis (GSMA Open Telco AI Initiative) — RAG configuration comparison across 6 generator models, built for supervisor review. Source data lives in `LiamDuero/telco-eval`; this repo is a curated, flattened extract of just the final numbers (accuracy, cost, latency, retrieval metrics, etc.) with the large per-question arrays stripped out.

Built by telcolens/liam/build_telco_analysis_repo.py, re-runnable at any time to refresh these files from the latest results on telco-eval.

Scope

Question set: final_3gpp_105.json (105 questions: 55 mc / 40 tf / 10 open, difficulty-first + series-stratified) — the one set all 6 models below share, so results are directly comparable.

Models (6):

Deliberately excluded: otel-2.0-local — confirmed broken (degenerate output: walls of tab/newline characters with a stray JSON fragment, same root cause as an earlier documented chat-template/tokenizer mismatch on that server, just a different garbage pattern) via a live content audit, not a real result.

Per model, up to 19 evaluation runs: 6 (chunking/enrichment/embedding) combos × 2 rerankers (generic cross-encoder + telecom-specific) + the same 6 combos with no reranker + 1 no-RAG baseline. Self-consistency: every RAG answer is 3 independent generations, majority-voted (mc/tf) or averaged (open).

Known gaps (present in the underlying data, not an error in this repo): Gemma has no baseline specifically against final_3gpp_105.json (only against the earlier small/medium sets); the 4 OpenRouter models were never run through the no-reranker leg (only Gemma and TelecomGPT-R1 were). all_results.json simply omits what doesn't exist — nothing here is fabricated or interpolated.

Files

• —`all_results.json` — every (model, phase, combo) result as one flat list. Each entry is a full aggregate result (accuracy, abstention/wrong rate, retrieval metrics, latency, tokens, real $ cost, by_format breakdown) tagged with model_id, phase (grid/no_reranker/baseline), reranker_label/reranker_key (null for no-reranker/baseline), and run_name.
• —`mc_results.json` / `tf_results.json` / `open_results.json` — the same runs, but only each one's per-format breakdown (by_format["mc"/"tf"/"open"]) pulled out into its own row, so a single format can be compared across every model/combo without digging through the nested by_format dict in all_results.json. mc/tf rows carry accuracy/abstention_rate/wrong_rate; open rows carry faithfulness/answer_relevancy/context_recall/answer_correctness. All rows also carry retrieval metrics (mean_similarity, canonical_hit_rate, etc.), since those apply regardless of format.

None of these files contain question_results (the per-question array with individual answers/ scores) — for that level of detail, pull the specific result.json from telco-eval directly.

Considerations

Personal working repo for thesis progress, not a curated public benchmark. Cost figures are real (OpenRouter's own billed usage.cost, or $0.00 for genuinely-free local vLLM), not estimates.