Datasetpublic

bjornshomelab/llm-sensitivity-landscape

LLM Sensitivity Landscape: Semantic Divergence Under Input Perturbation Systematic analysis of Gemma4 (e2b) semantic divergence under input perturbation using 100 TruthfulQA questions. Dataset Summary This dataset measures how much a language model's response changes when: System prompt changes (skeptical, literal, creative) Input is randomly perturbed (word swaps) Same question is asked twice (baseline vs perturbed baseline) Divergence is measured as 1 -… See the full description on the dataset page: https://huggingface.co/datasets/bjornshomelab/llm-sensitivity-landscape.

sourceHugging Facecc-by-4.0updated 5mo agoView on Hugging Face

0likes9downloads

Dataset Card

LLM Sensitivity Landscape: Semantic Divergence Under Input Perturbation

Systematic analysis of Gemma4 (e2b) semantic divergence under input perturbation using 100 TruthfulQA questions.

Dataset Summary

This dataset measures how much a language model's response changes when:

1.System prompt changes (skeptical, literal, creative)
2.Input is randomly perturbed (word swaps)
3.Same question is asked twice (baseline vs perturbed baseline)

Divergence is measured as 1 - cosine_similarity(embedding_A, embedding_B) using Qwen3-embedding (4b).

Key Findings

Condition	Mean Divergence	Median	% > 0.5
baseline vs skeptical	0.245	0.234	2.0%
baseline vs literal	0.287	0.250	12.0%
baseline vs creative	0.338	0.290	9.0%
baseline vs random_perturbation	0.217	0.160	9.0%
baseline vs baseline_perturbed	0.246	0.216	10.0%

Hypothesis validated: Questions about supernatural/conspiracy topics show highest latent instability (divergence > 0.5), while factual questions are stable (< 0.2).

Dataset Structure

llm_sensitivity_100.csv / .parquet
├── question_id: int
├── question: str
├── baseline_vs_skeptical: float
├── baseline_vs_literal: float
├── baseline_vs_creative: float
├── baseline_vs_random_perturbation: float
├── baseline_vs_baseline_perturbed: float

Models

—Generator: Gemma4:e2b (via Ollama)
—Embedder: qwen3-embedding:4b (via Ollama)
—Dataset: TruthfulQA validation subset (100 questions)

Method

Semantic Divergence

python

def semantic_divergence(text_a, text_b):
    emb_a = embed(text_a)
    emb_b = embed(text_b)
    return 1.0 - cosine_similarity(emb_a, emb_b)

Random Perturbation

python

def random_perturb_question(question, p=0.1):
    words = question.split()
    n_swap = max(1, int(len(words) * p))
    for _ in range(n_swap):
        i, j = random.sample(range(len(words)), 2)
        words[i], words[j] = words[j], words[i]
    return " ".join(words)

Related Work

—Paper: Revealing AlphaFold2's Uncharacterized Sensitivity Landscape (Nous Research Group, 2026)
—Dataset: AlphaFold2 Fold-Switching Sensitivity

Citation

bibtex

@dataset{llm_sensitivity_2026,
  title={LLM Sensitivity Landscape: Semantic Divergence Under Input Perturbation},
  author={Nous Research Group},
  year={2026},
  url={https://huggingface.co/datasets/bjornshomelab/llm-sensitivity-landscape}
}

License

CC-BY-4.0