GPUMODE/kernelbot-data

1# Kernelbot Data Processing Skills2 3This document describes how to extract and process submission data from the Kernelbot database.4 5## Database Connection6 7The production database is hosted on Heroku. **NEVER run write operations (INSERT, UPDATE, DELETE) on this database.**8 9```bash10# Get DATABASE_URL from Heroku11heroku config:get DATABASE_URL --app discord-cluster-manager12```13 14## Database Schema15 16The relevant tables are in the `leaderboard` schema:17 18| Table | Description |19|-------|-------------|20| `leaderboard.leaderboard` | Problem definitions (id, name, deadline, task, description) |21| `leaderboard.submission` | User submissions (id, leaderboard_id, user_id, code_id, submission_time, status) |22| `leaderboard.runs` | Execution results (submission_id, score, passed, mode, runner, result) |23| `leaderboard.user_info` | User details (id, user_name) |24| `leaderboard.gpu_type` | GPU types per problem (leaderboard_id, gpu_type) |25| `leaderboard.code_files` | Actual submission code content (old_code text, code bytea) |26 27## Key Problem IDs28 29### NVFP4 Problems30- **595**: nvfp4_gemv31- **597**: nvfp4_gemm32- **598**: nvfp4_dual_gemm33- **730**: nvfp4_group_gemm (not released yet)34 35### AMD Problems36- **398**: amd-identity37- **399**: amd-fp8-mm38- **430**: amd-mixture-of-experts39- **463**: amd-mla-decode40- **563**: amd-all2all41- **564**: amd-gemm-rs42- **565**: amd-ag-gemm43 44## Run Modes45 46| Mode | Description | Has Score? |47|------|-------------|------------|48| `test` | Correctness tests | No |49| `benchmark` | Performance benchmarks (internal) | No |50| `leaderboard` | Official leaderboard runs | **Yes** |51| `profile.0-3` | Profiling runs | No |52 53**Important:**54- Use `mode = 'leaderboard'` when joining runs to get scores.55- **Lower scores are better** (scores are execution time in seconds).56 57## SQL Queries58 59All SQL queries are in `queries.sql`. Key queries:60- List all problems61- Check submission counts62- Export deduplicated submissions with code63- Get top N submissions64- Get user progression over time65 66## Adding Support for a New Problem67 68### Step 1: Find the Problem ID69Use the "LIST ALL PROBLEMS" query from `queries.sql`.70 71### Step 2: Check Submission Counts72Use the "CHECK SUBMISSION COUNTS" query from `queries.sql`.73 74### Step 3: Export Deduplicated Submissions75Use the "EXPORT DEDUPLICATED SUBMISSIONS WITH CODE" query from `queries.sql`.76 77```python78import pandas as pd79import psycopg280 81DATABASE_URL = "..."  # from heroku config:get82conn = psycopg2.connect(DATABASE_URL)83 84# Read query from queries.sql and modify problem IDs as needed85with open('queries.sql') as f:86    # Find and use the export query section87    pass88 89df = pd.read_sql(query, conn)90df.to_parquet('new_problem_submissions.parquet', index=False)91```92 93### Step 4: Verify Data Quality94```python95from analyze_submissions import load_submissions, leaderboard_summary96 97df = load_submissions('new_problem_submissions.parquet')98print(leaderboard_summary(df))99```100 101## Accessing Submission Code102 103The parquet files include the full code content for each submission:104 105```python106from analyze_submissions import load_submissions107 108df = load_submissions()109 110# Get a specific user's best submission111user_subs = df[(df['user_name'] == 'gau.nernst') & (df['problem_name'] == 'nvfp4_gemv')]112best = user_subs.sort_values('score').head(1)113 114# Access the code115code = best['code'].values[0]116print(code)117```118 119## Helper Functions120 121Use `analyze_submissions.py`:122 123```python124from analyze_submissions import (125    load_submissions,      # Load parquet file126    author_progression,    # See user's submissions over time127    top_contestants,       # Get leaderboard rankings128    leaderboard_summary,   # Summary stats per problem129    user_stats,            # Stats for a specific user130    format_score           # Format score with units (us, ms, s)131)132```133 134## Environment Setup135 136```bash137uv venv .venv138source .venv/bin/activate139uv pip install pandas pyarrow psycopg2-binary140```141 142## Files143 144| File | Description |145|------|-------------|146| `nvidia_nvfp4_submissions.parquet` | Deduplicated NVIDIA NVFP4 submissions with code (~1.4 GB) |147| `queries.sql` | All SQL queries for data extraction |148| `scripts/nvfp4/analyze_submissions.py` | Helper functions library |149| `scripts/nvfp4/get_fastest_submission.py` | Print user's fastest submission |150| `scripts/nvfp4/query_submissions.py` | List submission IDs or query specific ID |151 152## Review Checklist Before Pushing153 1541. Verify submission counts match expectations1552. Check for any anomalies in scores (negative, extremely large, etc.)1563. Confirm deduplication worked correctly1574. Test helper functions work with the new data1585. Run `python scripts/nvfp4/query_submissions.py` to verify159