Datasetpublic

yinita/ps4mas-castle-oracle

PS4MAS CASTLE oracle Current CASTLE / oracle experiment catalog This generated section is authoritative. Older tables above are historical. Each experiments/<name>/ contains unified episodes.parquet, meta.json, and an unmodified summary.json when available. Raw full-hop traces.jsonl and run_config.json are separate files; scores are never injected into raw traces. Partial snapshots have immutable content-derived names. Existing experiments are skipped unless… See the full description on the dataset page: https://huggingface.co/datasets/yinita/ps4mas-castle-oracle.

sourceHugging Facemitupdated 5d agoView on Hugging Face

0likes1.1kdownloads

Dataset Card

PS4MAS CASTLE oracle

Current CASTLE / oracle experiment catalog

This generated section is authoritative. Older tables above are historical.

Each experiments/<name>/ contains unified episodes.parquet, meta.json, and an unmodified summary.json when available. Raw full-hop traces.jsonl and run_config.json are separate files; scores are never injected into raw traces.

Partial snapshots have immutable content-derived names. Existing experiments are skipped unless --overwrite-existing is explicit. Routine uploads use additions only. Explicitly authorized, content-audited invalid-data cleanups are recorded separately under audits/.

Experiment	Model	Episode rows	Run	Judge	Full hops	Dataset
0911/oracle_castle_final/27b__partial_794aaaf534247ccd	oraclecastleqwen35_27b	2200	[x]	[ ]	0/2200	official CASTLE-200
0911/oracle_castle_final/9b__partial_b9e598664b53000c	oraclecastleqwen35_9b	2200	[x]	[ ]	0/2200	official CASTLE-200
base_olmo-hybrid-instruct-dpo-7b_20260915_2174970d76ea_oracle_22e7e1834970	allenai/Olmo-Hybrid-Instruct-DPO-7B	2200	[x]	[ ]	2200/2200	official CASTLE-200
base_qwen3.5-9b_20260915_2174970d76ea_oracle_e4a45bb93e61	Qwen/Qwen3.5-9B	2200	[x]	[ ]	2200/2200	official CASTLE-200
collab_h100/phi4_castle_oracle__partial_e128c11ed6b394c1	phi4castleoracle	2214	[ ]	[ ]	2214/2400	official CASTLE-200
llama31_8b_20260917/castle_oracle/llama31_8b/castle/oracle	meta-llama-Llama-3.1-8B-Instruct	2200	[x]	[ ]	2200/2200	official CASTLE-200

Run = all intended scenario/config pairs produced a record; blank answers remain flagged in meta. Judge = every intended pair has valid non-mock scores. Mock placeholders never count as completed judging. Score-only historical results are archived as snapshots and do not prove runtime profile correctness. Full-hop coverage is a separate requirement. No missing historical hops are regenerated automatically.

Final role-aware termination audit: baseqwen3.5-9b202609152174970d76eaoracle_e4a45bb93e61

The denominator remains 2,200 unique attempts, all with nonempty final outputs and complete observed hop records. This partition has 0 attempts with any-hop max_steps and 0 final-role max_steps. All final outputs match a saved final-output hop; no final-role identity is unknown. Across all four partitions, the 205 any-hop max_steps flags (PENGUIN cold 17, CASTLE cold 188, both oracle 0) are intermediate-hop limits.

Complete traces do not establish overall policy success. protocol_required_hop_completed remains null because no additional predeclared per-topology episode-success rule was supplied. No success/failure label is inferred, and no sample was removed or regenerated. Judge remains deferred and score columns remain null. The full-row final-role audit is complete; see final audit, frozen counting schema and partition summary.