GPUMODE/kernelbot-data

1# Kernelbot Data Processing Skills2 3This document describes how to extract and process submission data from the Kernelbot database.4 5## Database Connection6 7The production database is hosted on Heroku. **NEVER run write operations (INSERT, UPDATE, DELETE) on this database.**8 9```bash10# Get DATABASE_URL from Heroku11heroku config:get DATABASE_URL --app discord-cluster-manager12```13 14## Database Schema15 16The relevant tables are in the `leaderboard` schema:17 18| Table | Description |19|-------|-------------|20| `leaderboard.leaderboard` | Problem definitions (id, name, deadline, task, description) |21| `leaderboard.submission` | User submissions (id, leaderboard_id, user_id, code_id, submission_time, status) |22| `leaderboard.runs` | Execution results (submission_id, score, passed, mode, runner, result) |23| `leaderboard.user_info` | User details (id, user_name) |24| `leaderboard.gpu_type` | GPU types per problem (leaderboard_id, gpu_type) |25| `leaderboard.code_files` | Actual submission code content (old_code text, code bytea) |26 27## Key Problem IDs28 29### NVFP4 Problems30- **595**: nvfp4_gemv31- **597**: nvfp4_gemm32- **598**: nvfp4_dual_gemm33- **730**: nvfp4_group_gemm34 35### AMD Problems36- **398**: amd-identity37- **399**: amd-fp8-mm38- **430**: amd-mixture-of-experts39- **463**: amd-mla-decode40- **563**: amd-all2all41- **564**: amd-gemm-rs42- **565**: amd-ag-gemm43- **763**: amd-mxfp4-mm44- **764**: amd-moe-mxfp445- **765**: amd-mixed-mla46 47### Other Completed Public Problems48- **496**: trimul49 50### PMPP v2 Problems51- **537**: conv2d_v252- **538**: grayscale_v253- **539**: histogram_v254- **540**: matmul_v255- **541**: prefixsum_v256- **542**: sort_v257- **543**: vectoradd_v258- **544**: vectorsum_v259 60### Released Helion / B200_Nebius Problems61- **766**: causal_conv1d62- **767**: fp8_quant63- **768**: gated_deltanet_chunk_fwd_h64- **769**: gated_deltanet_chunk_fwd_o65- **770**: gated_deltanet_recompute_w_u66 67### Linear Algebra Problems68- **774**: qr_v269- **775**: eigh70- **776**: cholesky71 72## Additional Export Files73 74- `amd_1_1m_competition_submissions.parquet`: deduplicated submissions with code for leaderboards 763, 764, and 76575- `trimul_submissions.parquet`: deduplicated submissions with code for leaderboard 49676- `helion_b200_nebius_submissions.parquet`: deduplicated submissions with code for leaderboards 766, 767, 768, 769, and 77077- `pmpp_v2_submissions.parquet`: all submissions with code for leaderboards 537, 538, 539, 540, 541, 542, 543, and 54478- `linalg_submissions.parquet`: deduplicated submissions with code for leaderboards 774 (`qr_v2`), 775 (`eigh`), and 776 (`cholesky`)79 80`trimul` is exported separately because it spans multiple GPU families and is not part of the AMD 1.1M competition set.81The Helion export is released separately because it targets `B200_Nebius`; measurements for that problem set are brittle and should be interpreted cautiously.82 83## Run Modes84 85| Mode | Description | Has Score? |86|------|-------------|------------|87| `test` | Correctness tests | No |88| `benchmark` | Performance benchmarks (internal) | No |89| `leaderboard` | Official leaderboard runs | **Yes** |90| `profile.0-3` | Profiling runs | No |91 92**Important:**93- Use `mode = 'leaderboard'` when joining runs to get scores.94- **Lower scores are better** (scores are execution time in seconds).95 96## SQL Queries97 98All SQL queries are in `queries.sql`. Key queries:99- List all problems100- Check submission counts101- Export deduplicated submissions with code102- Get top N submissions103- Get user progression over time104 105## Adding Support for a New Problem106 107### Step 1: Find the Problem ID108Use the "LIST ALL PROBLEMS" query from `queries.sql`.109 110### Step 2: Check Submission Counts111Use the "CHECK SUBMISSION COUNTS" query from `queries.sql`.112 113### Step 3: Export Deduplicated Submissions114Use the "EXPORT DEDUPLICATED SUBMISSIONS WITH CODE" query from `queries.sql`.115 116```python117import pandas as pd118import psycopg2119 120DATABASE_URL = "..."  # from heroku config:get121conn = psycopg2.connect(DATABASE_URL)122 123# Read query from queries.sql and modify problem IDs as needed124with open('queries.sql') as f:125    # Find and use the export query section126    pass127 128df = pd.read_sql(query, conn)129df.to_parquet('new_problem_submissions.parquet', index=False)130```131 132### Step 4: Verify Data Quality133```python134from analyze_submissions import load_submissions, leaderboard_summary135 136df = load_submissions('new_problem_submissions.parquet')137print(leaderboard_summary(df))138```139 140## Accessing Submission Code141 142The parquet files include the full code content for each submission:143 144```python145from analyze_submissions import load_submissions146 147df = load_submissions()148 149# Get a specific user's best submission150user_subs = df[(df['user_name'] == 'gau.nernst') & (df['problem_name'] == 'nvfp4_gemv')]151best = user_subs.sort_values('score').head(1)152 153# Access the code154code = best['code'].values[0]155print(code)156```157 158## Helper Functions159 160Use `analyze_submissions.py`:161 162```python163from analyze_submissions import (164    load_submissions,      # Load parquet file165    author_progression,    # See user's submissions over time166    top_contestants,       # Get leaderboard rankings167    leaderboard_summary,   # Summary stats per problem168    user_stats,            # Stats for a specific user169    format_score           # Format score with units (us, ms, s)170)171```172 173## Environment Setup174 175```bash176uv venv .venv177source .venv/bin/activate178uv pip install pandas pyarrow psycopg2-binary179```180 181## Files182 183| File | Description |184|------|-------------|185| `nvidia_nvfp4_submissions.parquet` | Deduplicated NVIDIA NVFP4 submissions with code (~1.4 GB) |186| `queries.sql` | All SQL queries for data extraction |187| `scripts/nvfp4/analyze_submissions.py` | Helper functions library |188| `scripts/nvfp4/get_fastest_submission.py` | Print user's fastest submission |189| `scripts/nvfp4/query_submissions.py` | List submission IDs or query specific ID |190 191## Review Checklist Before Pushing192 1931. Verify submission counts match expectations1942. Check for any anomalies in scores (negative, extremely large, etc.)1953. Confirm deduplication worked correctly1964. Test helper functions work with the new data1975. Run `python scripts/nvfp4/query_submissions.py` to verify198