helloelwin/Qwen3-32B_10episodes_comparisons_full

Qwen3-32B10episodescomparisons_full

This is a pairwise comparison dataset created from SWE-bench evaluation results.

Dataset Description

• —Model: Qwen3-32B
• —Episodes used: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
• —Total comparison pairs: 225
• —Resolved pairs: 67
• —Unresolved pairs: 158
• —Problems analyzed: 500
• —Problems with successes: 70
• —Problems with failures: 497
• —Problems skipped: 275

Files

• —Qwen3-32B_10episodes_comparisons_full_comparison_pairs.jsonl: JSONL file containing comparison pairs

Metadata

{
  "dataset_name": "Qwen3-32B_10episodes_comparisons_full",
  "model_name": "Qwen3-32B",
  "num_episodes": 10,
  "episodes_used": [
    1,
    2,
    3,
    4,
    5,
    6,
    7,
    8,
    9,
    10
  ],
  "statistics": {
    "total_problems": 500,
    "problems_with_success": 70,
    "problems_with_failure": 497,
    "problems_skipped": 275,
    "comparison_pairs_created": 225,
    "resolved_pairs": 67,
    "unresolved_pairs": 158
  }
}

Usage

import json
from datasets import load_dataset

# Load using Hugging Face datasets
dataset = load_dataset("helloelwin/Qwen3-32B_10episodes_comparisons_full")

# Or load manually
pairs = []
with open('Qwen3-32B_10episodes_comparisons_full_comparison_pairs.jsonl', 'r') as f:
    for line in f:
        pairs.append(json.loads(line))

Format

Each comparison pair contains:

• —instance_id: SWE-bench problem identifier
• —correct_response: Label ("A" or "B") indicating which response is correct
• —pair_type: Type of comparison pair ("resolved" or "unresolved")
• —episode_a, episode_b: Episode numbers for responses A and B
• —text_a, text_b: Model response text for responses A and B
• —full_output_a, full_output_b: Full model output for responses A and B
• —model_patch_a, model_patch_b: Model-generated patches for responses A and B
• —resolved_a, resolved_b: Resolution status (True/False) for responses A and B
• —num_passed_tests_a, num_passed_tests_b: Number of passed tests for responses A and B
• —model_name: Name of the model used
• —num_episodes_used: Number of episodes used in dataset creation
• —success_episode, failure_episode: Which episodes contained the successful and failed responses (for resolved pairs)
• —better_episode, worse_episode: Which episodes contained the better and worse responses (for unresolved pairs)

Generated using createcomparisondataset.py