CoolFace
Apppublic

obversarystudios/failure-geometry-demo

sourceHugging Facemitupdated 5mo agoView on Hugging Face
0likes
App README

Failure Geometry Demo

Self-contained research demo for failure structure analysis on compositional reasoning. No API key required. Runs entirely with scikit-learn.

text
CARB dataset → weak baselines → failure extraction → TF-IDF + SVD embeddings → KMeans → MI comparison

What this demonstrates

Two deliberately weak baselines expose different failure geometries on the same dataset:

BaselineFailure pattern
always_1Systematic bias — fails on every false-labeled item
keyword_heuristicNegation-sensitive — fails on affirmative-false and negated-true items

Pooling failures from both lets the demo ask:

Are failure clusters organised by reasoning category, by which baseline failed, or both?

Mutual information over cluster assignments answers this. The accuracy-by-type chart shows per-baseline slice performance. The 2-D SVD scatter shows whether clusters are visually separable.

Dataset

50 controlled reasoning examples across four types:

  • —Transitivity — chained relational premises (All A → B; All B → C; therefore All A → C)
  • —Negation — rule-state-conclusion triples where the state negates the conclusion
  • —Syllogism — classical universal / particular forms with valid and invalid conclusions
  • —Distractor logic — valid causal rules with irrelevant sentences injected as distractors

Dataset schema matches failure-induced-benchmarks CARB conventions: x, y, reasoning_type.

What this does not claim

  • —The baselines are deliberately weak to surface failures; they are not representative of production models.
  • —MI scores on 50 items are indicative, not statistically conclusive.
  • —This is a scaffold for the pipeline, not a benchmark result.

Related

  • —[obversarystudios.org](https://obversarystudios.org) — research engineering narrative.
  • —Failure discovery on binary reasoning — research framing.
  • —Failure clusters as interventions — what clusters imply for system changes.
  • —[carb-observability-space](https://huggingface.co/spaces/obversarystudios/carb-observability-space) — live HF Inference API version (requires HF_TOKEN).
  • —[agent-threat-map](https://huggingface.co/spaces/obversarystudios/agent-threat-map) — agent fragility benchmark (metrics + embedding/cluster/MI on scored probes).
  • —[github.com/architectfromthefuture/failure-induced-benchmarks](https://github.com/architectfromthefuture/failure-induced-benchmarks) — CARB generator and failure geometry library.

Honest scope

  • —Verified here: pipeline runs end-to-end on the seed dataset; MI scores are computed correctly.
  • —Described but not verified here: statistical significance of MI gaps; generalization beyond this seed set.