kogi-jwu/sakuraeval

SakuraEval

Dataset Description

SakuraEval is a Japan-specific code generation benchmark dataset. It is designed independently and does not rely on translation from English benchmarks such as HumanEval or JHumanEval. The dataset is currently being reviewed for official release.

Dataset Structure

from datasets import load_dataset
load_dataset("kogi-jwu/sakuraeval", "ja")
DatasetDict({
    test: Dataset({
        features: ['task_id', 'category', 'prompt', 'canonical_solution', 'test', 'entry_point'],
        num_rows: 164
    })
})

Data Fields

• —task_id: Identifier for the data sample.
• —category: Task category.
• —prompt: Input for the model, including the function header and docstring that describes the task.
• —canonical_solution: Solution to the problem presented in the prompt.
• —test: Function(s) to test the generated code for correctness.
• —entry_point: Entry point function to begin testing.

Category Breakdown

Languages

The dataset contains coding problems in 2 natural languages: English and Japanese.