viperoptic/VisualTraceBench

VisualTraceBench v1.0 — Dataset Card

Benchmark for Visual Execution-Trace Differencing in Software Performance Regression Diagnosis

Following the Datasheet for Datasets framework (Gebru et al., 2021).

Motivation

Purpose: VisualTraceBench is a curated dataset of 192 real-world software performance regression bug reports from a large enterprise database system (2016–2025), annotated with trace artifact type (visual vs. text-only) and resolution timing. It enables empirical measurement of the Visualization Performance Enhancement Rate (VPER) — the efficiency gain from visual execution-trace differencing over text-only trace inspection — and supports benchmarking of AI systems on multimodal root-cause analysis tasks.

Who created it: The dataset was created by a researcher at a large enterprise software company as part of an internal retrospective study on developer productivity in performance regression workflows.

Funding: Internal research. No external funding.

Composition

Instances: 192 bug reports (performance regression incidents), each representing one resolved bug from the production CI/CD pipeline of a large-scale column-store database system.

Groups: | Group | N | Description | |-------|---|-------------| | A | 79 | Bugs where a visual execution-trace artifact (flame graph diff / PlanViz) was attached during diagnosis | | B | 103 | Bugs where only text-format execution traces were available | | C | 2 | Bugs with both visual and text trace artifacts | | Unclassified | 8 | Incomplete annotation |

Features per instance:

Label / target for ML tasks: group (A vs B) and timing columns (post_attach_gap_hours, total_fix_hours) are the primary labels. The summary_scrubbed field supports NLP tasks.

Missing data: total_fix_hours and post_attach_gap_hours contain blank values where timing data was unavailable (formulas in original tracker). Subsample sizes used in analysis: metric1 An=41, Bn=87; metric2 An=75, Bn=49; metric3 An=37, Bn=33.

Sensitive data: All personally identifiable information has been removed. Email addresses replaced with pseudonyms (devNNN, reporterNNN). Real customer names replaced with scenario tags. Bug IDs remapped to sequential anonymous IDs. Internal file paths and Gerrit links removed from free-text fields.

Collection Process

How collected: Retrospective extraction from an internal bug tracking system. Bugs were selected based on keyword search for performance regression indicators and manual annotation of trace artifact type by the original researcher.

Time span: 2016–2025 (approximately 9 years of production incidents).

Sampling: Consecutive incidents matching selection criteria — not a random sample. Represents the full population of qualifying bugs in the studied system during the period.

Direct collection: No crowdsourcing. All bugs represent real production incidents resolved by software engineers.

Preprocessing / Cleaning / Labeling

Preprocessing applied for this release:

• —All SAP employee email addresses replaced with stable pseudonyms
• —All real customer/company names replaced with anonymized scenario tags
• —Original bug IDs remapped to sequential anonymous IDs (BUG-1 … BUG-192)
• —Internal file system paths and Gerrit review links removed from free-text fields
• —Free-text columns with high PII risk (steps to reproduce, solution text, delivery remarks) excluded entirely
• —group column derived from raw annotation columns (F, G) using rule: A=(F=O,G=O), B=(F=X,G≠X), C=(F=X,G=X)

Raw data: The original dataset with internal identifiers is retained by the authors and is not released.

Who performed labeling: The visualized/text_traced annotation was performed by the researcher conducting the retrospective study, based on inspection of bug report attachments.

Uses

Intended uses:

1. 1.Empirical software engineering research on visualization effectiveness in debugging
2. 2.Benchmarking AI systems on multimodal root-cause analysis (AI-VPER benchmark)
3. 3.Fault localization and performance regression detection research
4. 4.Training/evaluation of NLP models on bug report classification

Out-of-scope uses:

• —Identifying individual developers or customers from the anonymized data
• —Drawing conclusions about the performance of specific individuals
• —Use as a representative sample of all software systems (single-system study)

Potential misuse: The pseudonymization is stable (dev_001 always refers to the same person) to enable longitudinal analysis, but should not be used to attempt re-identification.

Distribution

License: CC BY 4.0 (Creative Commons Attribution 4.0 International)

Citation:

@dataset{visualtracebench2026,
  title        = {VisualTraceBench: A Dataset for Visual Execution-Trace Differencing in Performance Regression Diagnosis},
  author       = {Anonymous},
  year         = {2026},
  note         = {Under double-blind review. Author details withheld.},
  url          = {[dataset URL]}
}

Version: v1.0 (2026-06-04)

Maintenance

Maintainer: Authors (identity withheld for double-blind review). Contact via paper submission portal.

Updates: Bug fixes to anonymization or annotation errors will be versioned (v1.1, v1.2). Major additions will increment the minor version.

Errata: None known at v1.0.

Key Statistics (from analysis_results.json)

Note: p=0.689 reflects underpowered subsample (n=37+33). Effect size r=0.049. Acknowledged limitation — see paper.