youssefhassan13/exp03-l23-hardening

Exp 03 / 03b — Klüver L2/3 hardening study (SDXL + SD 3.5)

Pre-registered study from the Operating System Hypothesis project. Sweep classifier-free guidance across two architectures and score every output blind, on two independent rubrics, for how far object structure has come apart.

The prediction was written down and committed before the run. The commit dates in the GitHub repo are the proof.

What is here that is not on GitHub

The 860 generated PNGs. Every text artifact in this dataset is also in the public repo; the images are gitignored there because of their size, so this dataset is the reproduction path for anything that needs to look at the actual pictures.

Design

• —Guidance grid: 1.0, 2.0, 3.0, 5.0, 7.0, 11.0, 15.0
• —Prompts: 6, including a designed control
• —Seeds: 10 per cell, held fixed across guidance and model
• —Unconditional baseline: empty prompt, same seeds
• —Models: SDXL (convolutional UNet) and Stable Diffusion 3.5 (MMDiT)

Two rubrics on the same images

• —Exp 03, local scale. Klüver Level 2/3: reduplication, fragmentation, condensation, distortion, each 0–3, plus a binary tiling flag.
• —Exp 03b, global scale. Suzuki-style continuous axes: veridicality, spontaneity, complexity. Pre-registered separately, before scoring.

Headline result, including the miss

The overall pre-registered claim required both models to clear every gate, and it did not confirm.

Post-hoc, the two models differ in shape rather than magnitude: SDXL is a gradient that survives dropping the lowest guidance setting, SD 3.5 is a cliff that does not.

The judges are part of the data

Two of the three originally budgeted judges were inert and failed silently: well-formed JSON, fluent captions, and zero variance across the whole scale. judgements_llama.json and archive/judgements_qwen_Qwen2.5-VL-7B_*.json are kept deliberately so that failure is inspectable rather than described. The screening probes in probes/ show the capacity threshold between 8B and 32B.

If you are building a VLM-as-judge pipeline, that is the most transferable thing in this dataset.

Layout

sdxl/
  p{prompt}_g{guidance}_s{seed}.png   # conditioned outputs
  uncond_s{seed}.png                    # empty-prompt baseline
  judgements_claude.json                  # judge A, blind
  judgements_qwen.json                    # judge B, blind (Qwen3-VL-32B)
  judgements_llama.json                   # dead rater, kept as evidence
  judgements_*_axes.json                  # Exp 03b, Suzuki axes
  l23_report_claude-qwen.json             # amended confirmatory panel
  l23_report.json                         # pre-amendment 3-judge record
  quality.json, metadata.json
  figures/, figures_claude-qwen/
sd35/   (same layout)
preregistration.json, preregistration_axes.json
analysis.md, analysis_axes.md, analysis_plan.md
posthoc_report.json, axes_report.json
human_subset.json, human_ratings.json     # blind human validation subset
probes/, archive/
manifest.json

Limitations, stated up front

• —One human rater, who is the author, and who knows the hypothesis. Guidance and model were hidden and order was shuffled.
• —Human–judge agreement is 0.34 (Claude) and 0.12 (Qwen3-VL-32B). The two models agree with each other more than either agrees with the human. Clearing an inter-judge threshold is not the same as measuring what a person sees.
• —The distortion field partly tracks rendering style rather than object structure: holding photographic realism fixed shrinks it from −0.49 to −0.18.
• —Two lineages of judge, not three.

Provenance

• —Code, pre-registration, full write-up: https://github.com/youssefhassan/operating-system-hypothesis-public/tree/main/experiments/03_l23_hardening
• —Git commit: 14fcc44a6a88f0de2cd7131170395c47f86901b7
• —Exported: 2026-08-19T20:59:37.907329+00:00
• —Files: {"png": 860, "figure": 12, "text": 40}

License

MIT, same as the parent repository. Generated images are research artifacts; the underlying Stable Diffusion weights remain under their own model licenses.