CoolFace
Datasetpublic

Glide-py/r_judge_labelled

R-Judge with LLM-Judge Labels This dataset augments the R-Judge benchmark with automated safety labels produced by an LLM judge. R-Judge is a benchmark for evaluating the safety judgment capability of LLMs in multi-turn agent scenarios, spanning five application domains. Files File Description r_judge_data.csv Base dataset extracted from R-Judge (568 rows, deduplicated) r_judge_labelled_anthropic_claude-sonnet-4-6.csv Base dataset augmented with… See the full description on the dataset page: https://huggingface.co/datasets/Glide-py/r_judge_labelled.

sourceHugging Facecc-by-4.0updated 3mo agoView on Hugging Face
0likes262downloads
Dataset Card

R-Judge with LLM-Judge Labels

This dataset augments the R-Judge benchmark with automated safety labels produced by an LLM judge. R-Judge is a benchmark for evaluating the safety judgment capability of LLMs in multi-turn agent scenarios, spanning five application domains.

Files

FileDescription
r_judge_data.csvBase dataset extracted from R-Judge (568 rows, deduplicated)
r_judge_labelled_anthropic_claude-sonnet-4-6.csvBase dataset augmented with LLM-judge labels from Claude Sonnet 4.6

Dataset statistics

  • —Rows: 568 (after dropping 3 duplicate conversations from the source)
  • —Domains: Application, Finance, IoT, Program, Web
  • —Ground-truth label distribution: binary (0 = safe, 1 = unsafe)

Schema

Base columns (r_judge_data.csv)

ColumnDescription
applicationDomain category (Application, Finance, IoT, Program, Web)
file_pathSource file path within the R-Judge repository
idScenario identifier
scenarioShort scenario name
goalUser goal stated in the scenario
profileSystem prompt / agent profile
contentsJSON-serialised list of conversation rounds
labelGround-truth safety label (0 = safe, 1 = unsafe)
risk_descriptionHuman-written description of the risk
attack_typeAttack category (injection or unintended)

Additional columns in labelled files

ColumnDescription
llm_verdictLLM-judge safety label (0 = safe, 1 = unsafe)
llm_confidenceJudge confidence score (0 to 10)
llm_raw_outputRaw text output from the judge model
llm_parse_error"parse_error" if the output could not be parsed, otherwise empty
promptFull prompt sent to the judge model

Reproduction

Requirements

Install dependencies with pip:

bash
pip install -r requirements.txt

Or pass them inline with uv run as shown below.

Step 1: Extract base dataset

Clones the R-Judge repository, parses all JSON scenario files, drops duplicate conversations, and writes r_judge_data.csv.

bash
uv run --with pandas python extract_to_csv.py
# Saved 568 rows to r_judge_data.csv (3 duplicates dropped).

Step 2: Run LLM judge

Reads r_judge_data.csv and writes a new CSV with LLM-judge columns appended. The default model is Anthropic Claude Sonnet 4.6, which is what was used to produce the labelled file included in this dataset.

bash
uv run --with pandas --with anthropic --with python-dotenv python run_llm_judge.py
# Using anthropic / claude-sonnet-4-6
# [1/568] id=...  verdict=1, confidence=8
# ...
# Saved 568 rows to r_judge_labelled_anthropic_claude-sonnet-4-6_<timestamp>.csv

To use a different provider or model:

bash
uv run --with pandas --with openai --with python-dotenv python run_llm_judge.py \
    --provider openai --model gpt-4o

API keys

The judge script reads API credentials from environment variables or a .env file in the working directory:

  • —Anthropic: ANTHROPIC_API_KEY
  • —OpenAI: OPENAI_API_KEY

Source

R-Judge: https://github.com/Lordog/R-Judge

bibtex
@article{yuan2024rjudge,
  title={R-Judge: Benchmarking Safety Risk Awareness for LLM Agents},
  author={Yuan, Tongxin and He, Zhiwei and Dong, Lingzhong and Wang, Yiming and Zhao, Ruijie and Xia, Tian and Xu, Lizhen and Zhou, Binglin and Li, Fangqi and Zhang, Zhuosheng and Wang, Rui and Guan, Gongshen},
  journal={arXiv preprint arXiv:2401.10019},
  year={2024}
}