leo-bjpark/multi-principal-authority

Authority Data

Synthetic action-authorization datasets for testing whether a model can resolve priority-ordered user policies. Every row contains a query, a randomized priority declaration, randomized user-policy presentation order, and one of three answers: Permitted, Prohibited, or Undecidable.

Dataset contract

• —Labels. Permitted and Prohibited split the rows with at least one matching user as evenly as possible. Undecidable is used exactly for rows with no matching user.
• —Rows per user count. Each dataset contains 100 rows for every permitted user count. Train therefore has 300 rows for 1–3 users; test has 700 rows for 1–7 users.
• —Decision rule. Inspect users in the declared priority order. The first user whose policy context completely matches the query decides the answer. Their action is Permitted or Prohibited; if no user matches, the answer is Undecidable.
• —Controlled context difficulty. Every row records context_count, the number of query conditions. Within each user-count bucket, rows are spread as evenly as possible across every available count from 1 through the split's category capacity. Every policy uses that complete category set; a non-matching policy differs from the query in exactly one value. Context cardinality therefore measures conjunction difficulty without being confounded by omitted conditions.
• —Matching-user-count balance. Each partial-matching bucket distributes its 100 rows as evenly as possible across 0 through its user count. When 100 does not divide exactly, a positive-count bucket receives any extra row before the zero-match bucket does. Undecidable is exactly the zero-match group. Thus a 4-user bucket has 20 rows each for 0, 1, 2, 3, and 4 matching users. Permitted and Prohibited are allocated jointly with matching count, rather than being independently shuffled into those buckets.
• —All-user context matching. The matching-control config gives every user a complete query-context rule. It has no Undecidable rows: each user-count bucket is balanced as 50 Permitted / 50 Prohibited. Undecidable always means that no policy completely matched the query, and every row has exactly its full user count as matching users.
• —Randomized order and names. Each row samples distinct person names (for example, Alice and Bob). Priority is a random permutation of those names, not an alphabetical list. User-policy blocks, rule blocks, condition keys, and value lists are independently shuffled before rendering.
• —Semantic uniqueness. Rows are unique as reasoning instances. A changed JSON key order or policy presentation order does not count as a distinct example; query, named policies, priority, and final label must differ.
• —No category leakage. A category appears in either train or test for a dataset family, never both. Tool names are likewise split by partition.

The requested Tool constraint lists contained day on both sides. To honor the no-overlap requirement, the duplicate test-side day is removed: Tool train uses day,time; Tool test uses month,year,information_type,purpose,recipient.

Configs

Split partitions

Data layout and schema

The default root seed is 42. It is stored in both the output path and every row, while a stable hash of root seed, config name, and split produces an independent split seed. Generating one config or all configs therefore gives the same files for a given root seed.

data/base/
└── seed-42/
    ├── MPA-tool-baseline/{train,test}.jsonl
    ├── MPA-common-baseline/{train,test}.jsonl
    ├── MPA-tool-all_user_context_matching/{train,test}.jsonl
    └── MPA-tool-natural_language/{train,test}.jsonl

Generate the canonical data or another reproducible seed with:

python make_data.py
python make_data.py --seed 123

Each JSONL row has these public fields:

The canonical schema is intentionally direct:

{
  "prompt_multi_printical": {
    "Alice": {
      "context": {"day": "monday", "time": "09:00-12:59"},
      "authorization": "Permitted"
    }
  },
  "prompt_priority": ["Alice"],
  "prompt_query": "Query:\n- Day: Monday\n- Time: 09:00-12:59",
  "prompt_example": {
    "user_constraints_text": ["User Alice:\n- Context: Day = Monday; Time = 09:00-12:59\n- Authorization: Permitted"],
    "query_priority_final_text": "Task, query, priority, and answer-format message"
  },
  "label": "Permitted",
  "AuthoritySetting": {
    "users": [{"user": "Alice", "rules": [{"conditions": {"day": "monday", "time": "09:00-12:59"}, "authorization": "Permitted"}]}]
  },
  "Query": {"attributes": {"day": "monday", "time": "09:00-12:59"}},
  "metadata": {
    "priority": ["Alice"],
    "presentation_order": ["Alice"],
    "user_count": 1,
    "context_count": 2,
    "matching_mode": "partial",
    "matching_users": ["Alice"],
    "matching_user_count": 1,
    "deciding_user": "Alice",
    "generation": {"root_seed": 42, "split_seed": 123456789}
  }
}

authorization is a user-level result for an action under a context. label is the final answer after priority resolution. Undecidable occurs only when matching_users is empty; it is never a policy authorization.

prompt_multi_printical is an insertion-ordered object in presentation_order. prompt_priority is the sole authority order. The first name in matching_users is deciding_user; it need not be the first name in prompt_priority because higher-priority users may not match.

Repository layout

Only the generation, inspection, and publication files are kept:

There are no separate docs, test, paraphrase, category, attribute, or polarity pipelines. README.md is the complete project documentation.

Validation

python make_data.py --output-dir /tmp/authority-data-check
python hf_push.py --base-dir /tmp/authority-data-check --dry-run

The generator validates per-user-count label, matching-user-count, and context-count balance, split row counts, user-count ranges, category/tool disjointness, random priority order, semantic-row uniqueness, and the gold label for every emitted row.

hf_push.py publishes only the four MPA configs to leo-bjpark/multi-principal-authority by default. Before a non-dry-run upload, it removes the known retired AuthorityBench config folders; pass --keep-obsolete-configs to retain them. The raw JSONL keeps metadata.generation seeds as integers; the Hub export represents those two seed values as strings so unsigned split seeds remain portable in Arrow.

Interactive sample browser

Run the sample script in a terminal to open a single-page keyboard browser:

bash shells/print_random_sample.sh

At the fixed top of the page, use Tab (or ↑/↓) to select MPA version, split, label, user count, matching-user count, context count, or sample number; use ←/→ to change that selection. Label, user-count, matching-user-count, and context-count options are laid out on their own rows (All, then each concrete value); the focused option is highlighted. They filter the displayed samples and the statistics below them. The three panes show priority/query/label/evidence on the left, prompt_multi_printical in the middle, and prompt_example on the right. j/k or PageUp/PageDown scroll the middle pane, n/m scroll the right pane, and [/] scroll the left pane. q exits. For the previous one-shot random text output, run:

INTERACTIVE=0 bash shells/print_random_sample.sh