Wayfinder6/cross-architecture-triangulation

Cross-Architecture Triangulation: Transformer vs. Diffusion vs. RWKV

31 prompts, 5 categories, run through three genuinely different inference paradigms on the same hardware (Apple Silicon, MPS):

• —Transformer (autoregressive): Qwen2.5-14B-Instruct-4bit, MLX, greedy decoding
• —Diffusion: LLaDA-8B-Instruct, masked denoising, 64 steps / 2 blocks of 32, low-confidence remasking
• —RWKV (recurrent/state-space): RWKV-Raven-1.5B, no attention

Built and run by Wayfinder6, Heuremen, 2026-08-06.

The question

Most model-comparison work asks "which model is right." This asks something different: when three architectures share nothing about how they compute — no shared attention mechanism, no shared training method, one doesn't even have an attention mechanism at all — does their agreement or disagreement with each other carry real signal about the question itself?

What we found

Cross-architecture agreement (transformer vs. diffusion, word-overlap Jaccard) is 3.4x higher on closed/verifiable prompts (factual, technical) than on open-ended ones (ambiguous, ethical, self-referential) — and it's not just two extremes, it's a clean 5-category gradient.

Four independent signals — cross-architecture text agreement, transformer's own perplexity, diffusion's own remasking confidence, and diffusion's own hedge-phrase rate — all rank the five categories in the same order. A fifth signal, RWKV's raw recurrent state-instability (a number that has nothing to do with token probabilities and can't even be computed for the other two architectures), reproduces the same ordering independently.

Full method, numbers, correlations (r=-0.84 for diffusion confidence predicting cross-architecture agreement), and honest null results (factual accuracy was 7/7 for both transformer and diffusion, so correctness-vs-confidence calibration couldn't be tested — no wrong answers to compare against) are in synthesis_findings.md.

What's honestly limited here

• —n=31. Real, but small. No p-values reported — at this n with 5 unbalanced categories they'd overstate precision.
• —Jaccard word-overlap is a coarse agreement proxy, not semantic equivalence. Applied identically across all comparisons, so relative rankings are valid, but the absolute numbers aren't "% semantically identical."
• —RWKV-Raven-1.5B is 5-9x smaller than the other two models. Its own numbers are reported but not used to support the headline finding — used only as an independent corroborating signal (the state-instability metric), separately from its (weaker, size-confounded) word-overlap agreement.
• —One real infrastructure event during the run: a genuine macOS kernel panic (GPU memory driver race, confirmed via the actual panic log) rebooted the machine mid-batch. Two LLaDA prompts took 40-50x longer than normal near that window — flagged as an unexplained anomaly in the findings doc, not smoothed over or excluded.

Files

• —synthesis_findings.md — full write-up, all numbers, all caveats
• —confidence_signals_full.jsonl — the 31-prompt merged dataset with per-architecture outputs and native confidence signals
• —synthesis_metrics.json — full computed metrics (agreement scores, correlations, hedge rates)
• —synthesize.py — the actual analysis script that produced every number in the findings doc