Aymen-Lassoued/ABACUS_Bench

ABACUS — A Benchmark for Image Editing Models to Assess Object Category and Counting Under Scene Conservation

Instruction-based image editing models can now add, remove, or change a specified number of objects in a scene. Yet they often fail to produce the correct number, and these failures grow sharply as scenes become more complex and the requested count increases. Current benchmarks largely miss this problem: they score edits with holistic similarity metrics or with VLM judges that are themselves poor at counting, so numerical errors go unnoticed or are graded incorrectly.

ABACUS is a benchmark and judge-free evaluation method for numerical image editing. The protocol uses no VLM/MLLM judge; instead it verifies each edit at the instance level against panoptic ground truth confirmed by detector consensus, checking both that the exact requested count was achieved and that the rest of the scene was preserved — across the add, remove and change operations.

Contents

At N = 1 the instruction uses an article rather than the digit — Add an airplane to the scene., Change 1 handbag to a backpack in the scene.

Each example provides the source (before) image and the instruction. Ground-truth instance counts are not shipped in the dataset — see Fields below.

Load

from datasets import load_dataset

add    = load_dataset("Aymen-Lassoued/ABACUS_Bench", "add")                        # easy/medium/hard splits
remove = load_dataset("Aymen-Lassoued/ABACUS_Bench", "remove", split="hard")
change = load_dataset("Aymen-Lassoued/ABACUS_Bench", "change", split="medium")

Fields

Minimal by design — each example ships the task only:

• —image — the source (before) image
• —prompt — the edit instruction (the requested count is also given in count)
• —count — how many objects to add / remove / change
• —category — the object category the instruction acts on (change uses source_category + target_category instead)
• —id, image_id, coco_split — identifiers / COCO provenance

The difficulty tier is the split name (easy / medium / hard), not a column. Before/after ground-truth instance counts are not stored: the judge-free verifier recomputes the exact instance count from COCO (instances_*) via image_id + category, and takes the requested edit size from count. This keeps the released set lean and avoids duplicating COCO ground truth.

⚠️ add contains several instructions per image

Unlike remove and change, the add config has 3,896 instructions over 1,183 unique images (3.29 per image; one image carries up to 17). The same image_id appears with different count values, and id is add_{image_id}_{category}_K{count} so it stays unique. Do not assume one row per image when aggregating — group by id, not image_id, and bootstrap over image_id when computing confidence intervals, since instructions on one image are correlated.

remove and change keep one instruction per (image, category) and use {op}_{image_id}_{category}.

The add split is held out from training

The add config is disjoint at the image level from the training corpus used to fine-tune the accompanying editing model: no image in add was ever trained on, under any category. It is built to reproduce the count statistics reported in the paper (mean 3.20, max 15, counts spanning 1–15) while covering all 80 COCO thing-categories.

remove and change are the original release and were likewise never trained on.

Judge-free evaluation protocol

For a model output, detect instances of the relevant category (detector consensus + panoptic GT) and check both:

1. 1.Count correctness — recompute the COCO ground-truth count of category in image_id, derive the expected post-edit count (GT ± count; for change, the source_category count drops by count and the target_category count rises by count), and verify the detected count in the model's output matches.
2. 2.Scene preservation — non-target regions are unchanged.

No VLM/MLLM judge is used at any point, so numerical errors cannot be masked or misgraded.

Construction

• —Source images & GT: MS-COCO 2017 (train2017 / val2017); instance counts are read directly from COCO instances_* annotations.
• —Difficulty tiers: a continuous 3-axis difficulty score (scene clutter, target visibility, edit capacity), rank-normalised and split into easy/medium/hard terciles. The add config is not balanced across tiers (938/1,238/1,720); remove and change are close to balanced.
• —Requested count `N`:
• —remove / change — drawn from the true instance count of the target category, so N never exceeds what is present.
• —add — chosen to match the paper's reported count distribution, and always within the scene's verified capacity (a count demonstrated to fit that scene). Because adding does not consume existing instances, N for add is not bounded by the number of objects already present.
• —Crowd annotations: for remove and change, any (image, category) whose target class carries an iscrowd=1 annotation is excluded, since crowd masks are uncountable. The add config is not filtered on iscrowd; roughly 15% of its scenes contain some crowd annotation somewhere in the image, so verify the target category if your protocol depends on it.
• —`change` pairs: semantically reasonable, hand-curated within-domain swaps (e.g. car↔truck, cat↔dog, zebra→horse, toilet↔sink); classes with no plausible swap are excluded. 72 source categories.

Licensing & provenance

Prompts, tiers and annotations in ABACUS are released under CC BY 4.0. The images originate from MS-COCO (https://cocodataset.org) and remain subject to their original terms. ABACUS is provided for research and benchmarking.

Citation

@misc{abacus2026,
  title  = {ABACUS: A Benchmark for Image Editing Models to Assess Object
            Category and Counting Under Scene Conservation},
  author = {Lassoued, Aymen and others},
  year   = {2026}
}

COCO: Lin et al., Microsoft COCO: Common Objects in Context, ECCV 2014.