sinatras/pmpp-eval

1---2pretty_name: pmpp-eval3license: mit4task_categories:5- text-generation6- question-answering7language:8- en9tags:10- pmpp11- cuda12- coding13- mcq14- qa15size_categories:16- n<1K17configs:18- config_name: qa19  data_files: "pmpp_qa.jsonl"20  default: true21- config_name: coding22  data_files: "pmpp_coding.jsonl"23---24# PMPP Dataset25 26This repository provides two CUDA-focused datasets prepared by Sinatras and sponsored by Prime Intellect. Both datasets are based on *Programming Massively Parallel Processors* (4th Ed.) with additional coding evaluation harnesses at https://github.com/SinatrasC/pmpp-eval to be used by PMMP env in prime-environments.27  28## Overview29- **Languages:** English30- **License:** MIT31- **Curated by:** Sinatras (https://github.com/SinatrasC)32- **Sponsored by:** Prime Intellect33- **Derived from:** PMPP 4th Edition (Kirk & Hwu)34 35## Dataset Details36### pmpp_qa37- Composition: 61 MCQ + 77 short-answer items.38- Fields: `chapter`, `exercise`, `type`, `question`, `answer`, `explanation`, `topic_tags`, and optional `choices`.39- Topics emphasize CUDA indexing, occupancy, memory hierarchy, MPI, and dynamic parallelism.40 41### pmpp_coding42- 53 coding tasks. Every entry corresponds to `evaluation-tasks/<id>/student_kernel.cu` and includes runner metadata.43- Fields: `id`, `task_dir`, `student_file` (always `student_kernel.cu`), optional test targets/executables, and the trimmed CUDA skeleton.44- Some tasks export host wrappers (e.g., device property collection, one-pass radix) rather than `__global__` kernels. Tests under the same directory call the exported symbols.45 46## Coding Dataset Evaluation Sample47 48Dataset was evaluated using the coding evaluation harness within the PMPP env on prime-environments (https://github.com/PrimeIntellect-ai/prime-environments/)49 50### Model Performance51| Model | Total Tasks | Success Rate | Rollouts |52|-------|-------------|--------------|----------|53| Qwen/Qwen3-Next-80B-A3B-Thinking | 53 | 24.5% (39/159) | 3 per task |54 55### Top Performing Tasks56| Task | Success | Description |57|------|---------|-------------|58| ch02-vecadd-single-turn | 3/3 | Vector addition kernel |59| ch03-rgb2gray-single-turn | 3/3 | RGB to grayscale conversion |60| ch09-histogram-naive-single-turn | 3/3 | Histogram computation |61| ch09-histogram-shared-single-turn | 3/3 | Histogram computation |62| ch14-spmv-csr-thread-per-row-single | 3/3 | Sparse matrix-vector multiply |63| ch14-spmv-coo-single | 3/3 | Sparse matrix-vector multiply |64| ch14-spmv-ell-single | 3/3 | Sparse matrix-vector multiply |65| ch18-energy-gather-coarsened-single | 3/3 | Energy simulation kernel |66 67### Most Challenging Areas (0% Success)68| Challenge Category | Failed Tasks | Examples |69|-------------------|--------------|-----------|70| **Matrix Operations** | 6 tasks | Matrix multiplication variants, tiled algorithms |71| **Advanced Algorithms** | 8 tasks | Sorting, reduction, merge operations |72| **Memory Optimization** | 12 tasks | Shared memory, coalescing, thread coarsening |73| **MPI Integration** | 3 tasks | Multi-GPU communication patterns |74| **Dynamic Parallelism** | 3 tasks | Parent-child kernel launches |75| **Graph Algorithms** | 3 tasks | BFS, sparse data structures |76 77## Intended Use78- **pmpp_qa:** Evaluate or fine-tune GPU aware assistants on conceptual CUDA/MPI reasoning.79- **pmpp_code:** Evaluate or fine-tune code-generation capabilities.80 81## Limitations82- Specialized, CUDA-focused coding evaluation harness only.83- Some coding tasks require runtime configuration (e.g., enabling device heap). The pmpp-eval harness handles those details.84## Acknowledgements & Citation85Grateful acknowledgment to Prime Intellect for sponsoring this release and to the PMPP community for foundational materials. Additional inspiration and reference code were drawn from the open solution set at https://github.com/tugot17/pmpp. If you build on these datasets, please cite both sources:86```87@book{kirk2016programming,88  title     = {Programming Massively Parallel Processors: A Hands-on Approach},89  author    = {Kirk, David B. and Hwu, Wen-mei W.},90  edition   = {4th},91  year      = {2016},92  publisher = {Morgan Kaufmann}93}94 95@misc{pmpp_eval,96  author = {Sinatras},97  title  = {pmpp-eval},98  year   = {2025},99  url    = {https://github.com/SinatrasC/pmpp-eval}100}101```102 103For questions or contributions, open an issue in https://github.com/SinatrasC/pmpp-eval.