IDMedicine/code-graph-trajeval-v1

SWE-Bench Trajectory Eval Bundle (v1)

Companion artifact for the trajectory-probe downstream eval of the code-graph-v7 encoders (W1, I6, ...).

Contents

• —traj_full_bundle.tar.gz (488 MB) — contains:
• —specs.jsonl: 2456 SWE-Bench Verified agent trajectories harvested from swe-bench-submissions S3 bucket. Fields: instanceid, trajid, repo, base_commit, patches (1 entry = final model patch), resolved.
• —repos/: shallow (--filter=blob:none) clones of the 12 target repos (django, sympy, sphinx, matplotlib, scikit-learn, astropy, xarray, pytest, pylint, requests, seaborn, flask). ~671 MB uncompressed. Blobs pulled lazily per base_commit checkout.
• —graphjepa/: pipeline code (trajectorypipeline, trajectoryrealize, trajectoryprobe, trajectoryharvest) plus scripts/trajectory_full.sh.
• —harvest.log — stdout from the S3 harvester that produced specs.jsonl.

Downstream workflow

tar -xzf traj_full_bundle.tar.gz
rsync -a traj_full/graphjepa/ graphjepa/
mkdir -p outputs/traj_real
cp traj_full/specs.jsonl outputs/traj_real/
mv traj_full/repos outputs/traj_real/repos

# realize (4 sharded workers by repo)
SHARDS=4 bash graphjepa/scripts/trajectory_full.sh
tail -f outputs/traj_real/logs/realize_shard*.log

# merge manifests + probe with each encoder
cat outputs/traj_real/manifest_shard*.jsonl > outputs/traj_real/manifest.jsonl
for NAME in W1_softplus_s0 I6_joint_s0; do
  .venv/bin/python -m graphjepa.trajectory_probe \
    --manifest outputs/traj_real/manifest.jsonl \
    --ckpt outputs/$NAME/ckpt_final.pt \
    --pool mean --split-by repo \
    --output outputs/traj_real/probe_${NAME}.json
done

Provenance

Specs harvested from 5 SWE-Bench Verified submissions:

500 unique instanceids, 499 unique basecommits (median 5 trajectories per commit — different agents attempting the same task).