vinsblack/CodeReality

1# CodeReality-1T Evaluation Subset2 3## Location4The completed evaluation subset (19GB) is located at:5```6/mnt/z/CodeReality_Final/codereality-1t/eval/subset/data/7```8 9## Subset Statistics10 11### Overall Metrics12- **Files**: 323 JSONL files13- **Size**: 19.0 GB14- **Repositories**: 2,049 estimated15- **Creation**: Research value scoring with diversity sampling16 17### Research Characteristics18| Characteristic | Count | Percentage | Details |19|----------------|--------|------------|---------|20| Multi-repo files (5+ repos) | 323 | 100% | Files containing 5+ repositories each |21| Files with commit history | 2,049 | 100% | Complete git history available |22| Cross-language content | 1,495 | 73% | Repositories with multiple programming languages |23| Build system configurations | 798 | 39% | Makefile, package.json, build.gradle, pom.xml |24| Issue tracking data | 1,209 | 59% | GitHub/GitLab issues and discussions |25| Bug-fix commits | 1,845 | 90% | Commits identified as bug fixes |26| Test coverage | 1,332 | 65% | Repositories with test files |27| Documentation | 1,843 | 90% | README and documentation files |28 29### Repository Size Distribution30| Size Range | Repositories | Average Size |31|------------|-------------|--------------|32| Small (< 10MB) | ~800 | 3.2 MB |33| Medium (10-100MB) | ~1,100 | 42.1 MB |34| Large (> 100MB) | ~149 | 187.3 MB |35 36### Language Diversity (Estimated)37- **JavaScript/TypeScript**: ~35% of content38- **Python**: ~20% of content39- **Java/C/C++**: ~25% of content40- **Mixed/Other**: ~20% of content41 42## Structure43```44eval_subset/45├── data/               # 280 curated JSONL files (15.1GB)46├── docs/               # Usage examples and documentation47├── eval_metadata.json  # Complete subset metadata48└── USAGE_EXAMPLES.md   # Demonstration code examples49```50 51## Usage52The subset is ready for:53- Code completion benchmarks (Pass@k evaluation)54- License detection training/testing55- Cross-language analysis56- Bug detection studies57- Repository classification58 59## Access60To use the evaluation subset:61 62```python63import json64import os65 66eval_dir = "/mnt/z/CodeReality_Final/codereality-1t/eval/subset"67 68# Load metadata69with open(os.path.join(eval_dir, 'eval_metadata.json'), 'r') as f:70    metadata = json.load(f)71 72print(f"Subset: {metadata['eval_subset_info']['name']}")73print(f"Files: {metadata['subset_statistics']['total_files']}")74print(f"Size: {metadata['subset_statistics']['total_size_gb']} GB")75 76# Load sample data77data_dir = os.path.join(eval_dir, 'data')78for filename in os.listdir(data_dir)[:5]:  # First 5 files79    file_path = os.path.join(data_dir, filename)80    with open(file_path, 'r', encoding='utf-8', errors='ignore') as f:81        for line in f:82            repo_data = json.loads(line)83            print(f"Repository: {repo_data.get('name', 'Unknown')}")84            break  # Just first repo from each file85```86 87## Benchmarks88Demonstration benchmarks available in `../benchmarks/`:89- `license_detection_benchmark.py`90- `code_completion_benchmark.py`91 92See `../benchmarks/README.md` for detailed usage instructions.93 94## Metadata95Complete metadata available in `eval_metadata.json` includes:96- Selection methodology97- Research characteristics98- Size distribution99- File manifest with checksums100- Recommended evaluation tasks101 102This subset represents a curated, research-ready portion of the complete CodeReality-1T dataset optimized for standardized evaluation and benchmarking.