CoolFace
Datasetpublic

Akirayasha/cisb-bench-llvm-O2

CISB-Bench LLVM-O2: Assembly Transpilation Benchmark Overview This dataset is derived from the Compiler-Introduced Security Bugs (CISB) benchmark, compiled with LLVM/Clang-17 at -O2 optimization level. It supports bidirectional assembly transpilation (x86-64 ↔ AArch64) and is designed for evaluating transpilation models on security-critical code. Dataset Structure Column Description task_name Full task identifier including flag (e.g.… See the full description on the dataset page: https://huggingface.co/datasets/Akirayasha/cisb-bench-llvm-O2.

sourceHugging Facemitupdated 2mo agoView on Hugging Face
0likes27downloads
Dataset Card

CISB-Bench LLVM-O2: Assembly Transpilation Benchmark

Overview

This dataset is derived from the Compiler-Introduced Security Bugs (CISB) benchmark, compiled with LLVM/Clang-17 at -O2 optimization level. It supports bidirectional assembly transpilation (x86-64 ↔ AArch64) and is designed for evaluating transpilation models on security-critical code.

Dataset Structure

ColumnDescription
task_nameFull task identifier including flag (e.g., b-4-var-0-safe, b-4-var-0-unsafe)
flag"safe" or "unsafe" — whether the assembly preserves the security check
source_codeOriginal C source from cisb-bench/<variant>/code.c (shared by safe/unsafe pairs)
x86_64x86-64 assembly (code.s from .build_x86/)
aarch64_linuxAArch64 assembly (code.s from .build_arm_linux/)
test_cC source of the verifier harness (test.c) -- lets a consumer rebuild and rerun the verifier without needing local access to benchmarks/cisb-bench-llvm-O2/
run_outputCombined stdout+stderr of the verifier harness (test.c), run against the x86_64 build

Statistics

  • —Total records: 72
  • —Safe records: 36
  • —Unsafe records: 36

Transpilation Directions

This dataset supports both directions:

  • —x86 → ARM: Use x86_64 as input, aarch64_linux as target
  • —ARM → x86: Use aarch64_linux as input, x86_64 as target

Security Oracle

The test harness returns exit codes:

  • —0 → SAFE (security check preserved)
  • —1 → UNSAFE (security check eliminated)
  • —2 → HARNESS_ERROR (transpilation error)
  • —>128 → CRASH

The run_output column captures what the harness printed while producing that verdict for the safe/unsafe x86_64 build, so it can be fed to a model as additional context.

Usage

python
from datasets import load_dataset

ds = load_dataset("Akirayasha/cisb-bench-llvm-O2", split="test")

# Filter by flag
safe = ds.filter(lambda x: x["flag"] == "safe")
unsafe = ds.filter(lambda x: x["flag"] == "unsafe")

# x86 -> ARM transpilation
record = ds[0]
x86_input = record["x86_64"]
arm_target = record["aarch64_linux"]

# ARM -> x86 transpilation
arm_input = record["aarch64_linux"]
x86_target = record["x86_64"]

# Rebuild/rerun the verifier against a candidate
harness = record["test_c"]
failure = record["run_output"]

Citation

Part of the CISC-to-RISC transpilation research project at MBZUAI.