xlr8harder/trellismark-qwen3-4b-open-corpus

TrellisMark Qwen3-4B open-corpus benchmark

This is the reproducible open-corpus companion to the main [TrellisMark Qwen3-4B confirmation corpus](https://huggingface.co/datasets/xlr8harder/trellismark-qwen3-4b). It is an explicitly derived subset of that release, not a separately generated corpus. The standalone detector, benchmark runner, tests, and exact Viterbi implementation are in the [TrellisMark GitHub repository](https://github.com/xlr8harder/trellismark).

The benchmark studies an intentionally difficult mixed setting: a corpus of separately prompted documents, with hidden authorship, an unknown number of represented watermarked users, and a mixture of marked and unwatermarked AI text. It is a deliberately simple baseline for candidate discovery, not a claim that clustering or greedy peeling has been optimized.

TrellisMark demonstrates a surveillance capability. This release exists so the claim can be checked and countermeasures can be studied; it is not a recommendation to deploy user-level watermarking.

At a glance

The selected marked rows comprise 64 uniformly sampled calibration users and 128 disjoint uniformly sampled test users. Every user has 128 independently prompted responses, each scored through its first 200 tokens. The release also includes every calibration and test unwatermarked control from the main corpus. Prompts and documents do not cross the calibration/test boundary.

The canonical source is main-dataset revision v1.0.2, commit c3ed277f6cb167bf8339e404828dcdc9099b04c4. Canonical document IDs are unchanged. subset-manifest.json records the selection contract and alignment checks.

Benchmark design

Each simulated corpus contains 16 represented watermarked users. The detector is not given the author grouping or the number of represented users. The main article condition adds 128 unwatermarked documents and varies evidence from each represented user at nested checkpoints:

There are 128 calibration corpora and 128 held-out test corpora per point. The same users and document ordering are retained as marked evidence accumulates, so adjacent points form nested curves rather than unrelated samples. A second zero-control-document arm is included in calibration and the raw scenario reports.

Candidate discovery uses average-linkage initialization, joint soft partition refinement, and exact full-registry Viterbi decoding. The frozen acceptance threshold for each detector is the most conservative of the empirical 99th percentile calibration thresholds across all tested evidence and null-mixture scenarios. This is an empirical operating point, not a posterior probability or deployment guarantee.

Interpreting a false lead

A false lead means that the detector accepted at least one address that was not represented in that simulated corpus. It does not mean a wrong global top-1. A corpus can recover all 16 true users and still count as a false-lead event if it also emits a spurious seventeenth address. The summary reports both user recall and the percentage of whole corpora with one or more such absent addresses.

Load the data

from datasets import load_dataset

documents = load_dataset(
    "xlr8harder/trellismark-qwen3-4b-open-corpus",
    "documents",
    split="corpus",
)
key_only = load_dataset(
    "xlr8harder/trellismark-qwen3-4b-open-corpus",
    "key_only_evidence",
    split="corpus",
)
model_assisted = load_dataset(
    "xlr8harder/trellismark-qwen3-4b-open-corpus",
    "model_assisted_evidence",
    split="corpus",
)
recipes = load_dataset(
    "xlr8harder/trellismark-qwen3-4b-open-corpus",
    "recipes",
    split="corpus",
)

Each evidence row contains a flattened 32 x 4 position_by_symbol_scores matrix. The NPZ copies under evidence/ are the faster input format used by the released runner; the Parquet copies expose the same rows through the standard datasets interface.

Reproduce the benchmark

Clone this dataset repository as open-corpus-data, then install the standalone TrellisMark repository and run:

python scripts/run_open_corpus_grid.py \
  --subset-directory open-corpus-data \
  --output-directory open-corpus-run \
  --workers 4 --threads-per-worker 4 --trials 128 --device cuda

python scripts/summarize_open_corpus.py \
  --scenario-reports open-corpus-run/scenario-reports/*.json \
  --output open-corpus-run/open-corpus-report-v1.json \
  --recipes-output open-corpus-run/open-corpus-recipes-v1.jsonl

CUDA is optional; --device cpu produces the same benchmark on a slower CPU path. The grid controller refuses to overwrite an existing run. --resume validates completed scenario reports before reusing them.

The published results/ directory contains all 24 raw scenario reports plus the frozen combined report. The generated recipes must match the 3,072 published rows exactly. export-manifest.json records a SHA-256 digest for every release artifact.

Data contract

documents preserves the exact raw response text and token IDs from the main corpus, without filtering or editing. Marked rows expose the actual 30-bit user_id; unwatermarked rows use null. Exact token IDs should be used for detector work because decoding and re-encoding text need not preserve every token boundary.

The public research key and complete generation configuration are under config/. Never reuse the key for a deployment. The key-only detector needs the public key and exact tokenizer behavior to score new text. Reproducing the released curves from cached evidence does not require model inference. Scoring new text with the model-assisted method additionally requires the frozen Qwen model and scorer configuration.

Scope and limitations

• —This is an experimental existence proof, not a production security system.
• —The simulated corpora contain exactly 16 marked users even though that count is hidden from the detector; other compositions need separate evaluation.
• —The threshold is calibrated on 128 corpora per scenario, so tail behavior is only coarsely resolved.
• —Evidence units are independent Qwen responses scored through their first 200 tokens, not natural message-board threads or human-authored posts.
• —Reported recoveries are finite held-out observations, not population guarantees.

Sources, attribution, and licensing

The generated responses use `Qwen/Qwen3-4B-Instruct-2507`, whose model card lists Apache-2.0. Prompt identifiers derive from `allenai/WildChat`, whose card lists ODC-By; prompt text is not included here.

The Hugging Face license: other label reflects the mixed contents of the repository. Our original TrellisMark contribution is MIT licensed, including the code, configuration, metadata schema, numeric evidence, recipes, and result reports. Generated prose and source-derived identifiers may remain subject to applicable upstream terms.

TrellisMark is an independent research project. It is not affiliated with or endorsed by Google, Qwen, AllenAI, or Hugging Face.