Datasetpublic

Mahmoud-queens/swe-agent-subset-selection-vectors

Trajectory embeddings for "Trajectory-Aware Benchmark Subset Selection for Cost-Efficient Software Engineering Agent Regression Testing" This repository holds the embeddings of sanitized agent trajectories used in the paper named above. They are the input of the selection stage of the paper's replication package: https://github.com/SAILResearch/swe-agent-subset-selection Publishing the vectors spares users the embedding step, which takes between several hours and several days… See the full description on the dataset page: https://huggingface.co/datasets/Mahmoud-queens/swe-agent-subset-selection-vectors.

sourceHugging Faceupdated 4d agoView on Hugging Face

0likes44downloads

Dataset Card

Trajectory embeddings for "Trajectory-Aware Benchmark Subset Selection for Cost-Efficient Software Engineering Agent Regression Testing"

This repository holds the embeddings of sanitized agent trajectories used in the paper named above. They are the input of the selection stage of the paper's replication package: https://github.com/SAILResearch/swe-agent-subset-selection

Publishing the vectors spares users the embedding step, which takes between several hours and several days depending on the dataset.

What the vectors are

Each trajectory (one agent run on one benchmark instance) was parsed into a unified step schema and sanitized (outcome-revealing and repository-specific terms masked). Every step was then embedded with nomic-ai/nomic-embed-text-v1.5 (revision e5cf08aadaa33385f5990def41f7a23405aec398, model code nomic-ai/nomic-bert-2048 at 7710840340a098cfb869c4f65e87cf2b1b70caca), 768 dimensions, float32.

There are two files per trajectory:

—..._ts.npy: time-series matrix, steps x 768;
—..._pooled.npy: pooled vector of 3,840 values (first, middle and last step, then the mean and the standard deviation over steps).

File names are <OUTCOME>_<repository>_<trajectory id>_{ts,pooled}.npy, with OUTCOME = SUCCESS or FAIL (whether the run resolved the instance). The replication package reads the outcomes from this prefix, so the files must not be renamed.

Datasets

Archive	Trajectories	Source of the raw trajectories
`multi_model_vectors.tar.gz` (391 MB)	3,000: six OpenHands runs with different models on SWE-Bench Verified	https://github.com/SWE-bench/experiments
`multi_agent_vectors.tar.gz` (473 MB)	3,499: seven runs of five agent frameworks on SWE-Bench Verified (one trajectory has no steps and therefore no vector)	https://github.com/SWE-bench/experiments
`single_setup_vectors.tar.gz` (4.7 GB)	25,279: reruns of OpenHands with one model on SWE-rebench, run groups with 5 to 10 reruns	https://huggingface.co/datasets/nebius/SWE-rebench-openhands-trajectories

Files

File	Size	md5
`multi_model_vectors.tar.gz`	391 MB	`bfc4a068805604e77e8f058a249e91da`
`multi_model_vectors.md5`	0.8 MB	`96e9f17daa8d391466cb1efdd9a80022`
`multi_agent_vectors.tar.gz`	473 MB	`98557ef1a883833a8adc73760b3b2be9`
`multi_agent_vectors.md5`	0.9 MB	`a513ff65ca1d1dfe38691f6f44f2f8d0`
`single_setup_vectors.tar.gz`	4.7 GB	`cf1dc92c9a8402a8b7c73de06a294f1b`
`single_setup_vectors.md5`	7.2 MB	`fd7c8693ec54f0ec8ce6718c1c78c423`
`single_setup_repository_map.json`	4.0 MB	`673304af64b9e4ea5e47cbacaf9dff26`

Each <dataset>_vectors.md5 lists the md5 of every file inside the corresponding archive, one line per file.

Layout inside the archives

<dataset>_vectors.tar.gz
    vectors/pooled/<run>/*.npy                              multi_model, multi_agent: one folder per run
    vectors/timeseries/<run>/*.npy
    vectors/{pooled,timeseries}/<N>_runs/run_<i>/*.npy      single_setup: run groups N = 5..10
    features/trajectory_features.npz                        multi_model, multi_agent: ten trajectory features
    features/trajectory_features_meta.json                  (used by some clustering configurations)

single_setup_repository_map.json                            trajectory id -> instance id, repository, outcome

Use with the replication package

From the root of the replication package:

bash

python pipeline/fetch.py --dataset multi_model --vectors      # or multi_agent, single_setup
python reproduce.py pipeline --dataset multi_model --rq 2

fetch.py --vectors downloads the archive and its manifest, unpacks it to data/<dataset>/, checks every file against the manifest and, for singlesetup, places the repository map as `data/singlesetup/repository_map.json. reproduce.py pipeline` then runs the selection and evaluation stages on these vectors.

Use without the replication package

python

import numpy as np

pooled = np.load("vectors/pooled/<run>/SUCCESS_<repository>_<trajectory id>_pooled.npy")   # shape (3840,)
steps = np.load("vectors/timeseries/<run>/SUCCESS_<repository>_<trajectory id>_ts.npy")    # shape (n_steps, 768)

Licence and citation

The raw trajectories come from the sources listed above and remain under their terms. If you use these vectors, please cite the paper by its title; the full reference will be added on publication.