argyrotsipi/ParliaBench

🏛️ ParliaBench — UK Parliamentary Speech Generation

Interactive inference demo for ParliaBench, a benchmark framework for evaluating LLM-generated UK parliamentary speeches.

Paper: ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech Authors: Marios Koniaris, Argyro Tsipi, Panayiotis Tsanakas · NTUA arXiv: 2511.08247

What This Space Does

1. 1.Generate synthetic parliamentary speeches conditioned on party, EuroVoc topic, parliamentary section, house, and political orientation
2. 2.Inspect the exact prompt sent to each model (chat template tokens included)
3. 3.Browse curated samples — synthetic vs real ParlaMint-GB speeches side by side
4. 4.Tune generation parameters (temperature, top-p, repetition penalty)

Models

Five LLMs fine-tuned on ParlaMint-GB via QLoRA (Unsloth framework):

Baseline (non-fine-tuned) versions are also selectable for direct comparison.

Datasets

LoRA Training Configuration

Prompt Structure

System prompt (generation):

You are a seasoned UK parliamentary member. Generate a coherent speech of
{min_words}-{max_words} words in standard English (no Unicode artifacts, no special characters).
Use proper British parliamentary language appropriate for the specified House.
The speech should reflect the political orientation and typical positions of the
specified party on the given topic.

Context string (pipe-separated, matches generation code exactly):

EUROVOC TOPIC: {topic} | SECTION: {section} | PARTY: {party} | POLITICAL ORIENTATION: {orientation} | HOUSE: {house}

Each model wraps these in its own chat template (Mistral [INST], Llama header tokens, Gemma <start_of_turn>, Qwen/Yi ChatML).

Generation Parameters (thesis defaults)

Evaluation Framework

27 560 speeches evaluated across three dimensions:

Linguistic Quality

Perplexity · Self-BLEU · Distinct-n · GRUEN Score · BERTScore · MoverScore

Semantic Coherence

LLM-as-a-Judge (coherence, conciseness, relevance) via FlowJudge-v0.1 (3.8B)

Political Authenticity ← novel metrics

• —Political Spectrum Alignment (PSA) — embedding cosine similarity to spectrum axis
• —Party Alignment — cosine similarity to real party speech embeddings
• —LLM-as-a-Judge (authenticity, political appropriateness, overall quality)

Statistical analysis: paired t-tests, independent t-tests, one-way ANOVA, Bonferroni correction.

Space File Structure

argyrotsipi/ParliaBench/
├── app.py               # Gradio UI + inference pipeline
├── utils.py             # Party data, topic lists, validator (from SpeechValidator)
├── prompt_templates.py  # Model-specific chat templates (exact from speech_generator.py)
├── sample_data.json     # 7 curated demo speeches
├── requirements.txt
└── README.md

Training code → GitHub Datasets → `argyrotsipi/train-dataset` · `argyrotsipi/generated-dataset`

Citation

@article{koniaris2025parliabench,
  title   = {ParliaBench: An Evaluation and Benchmarking Framework for
             LLM-Generated Parliamentary Speech},
  author  = {Koniaris, Marios and Tsipi, Argyro and Tsanakas, Panayiotis},
  journal = {arXiv preprint arXiv:2511.08247},
  year    = {2025},
  url     = {https://arxiv.org/abs/2511.08247}
}