vinsblack/CodeReality

CodeReality: Evaluation Subset - Deliberately Noisy Code Dataset

⚠️ Important Limitations

⚠️ Not Enterprise-Ready: This dataset is deliberately noisy and designed for research only. Contains mixed/unknown licenses, possible secrets, potential security vulnerabilities, duplicate code, and experimental repositories. Requires substantial preprocessing for production use. Use at your own risk - this is a research dataset for robustness testing and data curation method development.

Overview

CodeReality Evaluation Subset is a curated research subset extracted from the complete CodeReality dataset (3.05TB, 397,475 repositories). This subset contains 2,049 repositories in 19GB of data, specifically selected for standardized evaluation and benchmarking of code understanding models on deliberately noisy data. For complete Dataset 3tb, please contact me at vincenzo.gallo77@hotmail.com

Key Features

• —✅ Curated Selection: Research value scoring with diversity sampling from 397,475 repositories
• —✅ Research Grade: Comprehensive analysis with transparent methodology
• —✅ Deliberately Noisy: Includes duplicates, incomplete code, and experimental projects
• —✅ Rich Metadata: Enhanced Blueprint metadata with cross-domain classification
• —✅ Professional Grade: 63.7-hour comprehensive analysis with open source tools

Quick Start

Dataset Structure

codereality-1t/
├── data_csv/               # Evaluation subset data (CSV format, 2,387 repositories)
│   ├── codereality_unified.csv  # Main dataset file with unified schema
│   └── metadata.json           # Dataset metadata and column information
├── analysis/               # Analysis results and tools
│   ├── dataset_index.json  # File index and metadata
│   └── metrics.json        # Analysis results
├── docs/                   # Documentation
│   ├── DATASET_CARD.md     # Comprehensive dataset card
│   └── LICENSE.md          # Licensing information
├── benchmarks/             # Benchmarking scripts and frameworks
├── results/                # Evaluation results and metrics
├── Notebook/               # Analysis notebooks and visualizations
├── eval_metadata.json      # Evaluation metadata and statistics
└── eval_subset_stats.json  # Statistical analysis of the subset

Loading the Dataset

📊 Unified CSV Format

This dataset has been converted to CSV format with a unified schema to ensure compatibility with Hugging Face's dataset viewer and eliminate schema inconsistencies that were present in the original JSONL format.

How to Use This Dataset

Option 1: Standard Hugging Face Datasets (Recommended)

from datasets import load_dataset

# Load the complete dataset
dataset = load_dataset("vinsblack/CodeReality")

# Access the data
print(f"Total samples: {len(dataset['train'])}")
print(f"Columns: {dataset['train'].column_names}")

# Sample record
sample = dataset['train'][0]
print(f"Repository: {sample['repo_name']}")
print(f"Language: {sample['primary_language']}")
print(f"Quality Score: {sample['quality_score']}")

Option 2: Direct CSV Access

import pandas as pd
from huggingface_hub import snapshot_download

# Download the dataset
repo_path = snapshot_download(repo_id="vinsblack/CodeReality", repo_type="dataset")

# Load CSV files
import glob
csv_files = glob.glob(f"{repo_path}/data_csv/*.csv")
df = pd.concat([pd.read_csv(f) for f in csv_files], ignore_index=True)

print(f"Total records: {len(df)}")
print(f"Columns: {list(df.columns)}")

Option 3: Metadata and Analysis

# Load evaluation subset metadata
with open('eval_metadata.json', 'r') as f:
    metadata = json.load(f)

print(f"Subset: {metadata['eval_subset_info']['name']}")
print(f"Files: {metadata['subset_statistics']['total_files']}")
print(f"Repositories: {metadata['subset_statistics']['estimated_repositories']}")
print(f"Size: {metadata['subset_statistics']['total_size_gb']} GB")

# Access evaluation data files
data_dir = "data/"  # Local evaluation subset data
for filename in os.listdir(data_dir)[:5]:  # First 5 files
    file_path = os.path.join(data_dir, filename)
    with open(file_path, 'r', encoding='utf-8', errors='ignore') as f:
        for line in f:
            repo_data = json.loads(line)
            print(f"Repository: {repo_data.get('name', 'Unknown')}")
            break  # Just first repo from each file

Dataset Statistics

Evaluation Subset Scale

• —Total Repositories: 2,049 (curated from 397,475)
• —Total Files: 323 JSONL archives
• —Total Size: 19GB uncompressed
• —Languages Detected: Multiple (JavaScript, Python, Java, C/C++, mixed)
• —Selection: Research value scoring with diversity sampling
• —Source Dataset: CodeReality complete dataset (3.05TB)

Language Distribution (Top 10)

Duplicate Analysis

Exact Duplicates: 0% exact SHA256 duplicates detected across file-level content Semantic Duplicates: ~18% estimated semantic duplicates and forks preserved by design Research Value: Duplicates intentionally maintained for real-world code distribution studies

License Analysis

License Detection: 0% detection rate (design decision for noisy dataset research) Unknown Licenses: 96.4% of repositories marked as "Unknown" by design Research Purpose: Preserved to test license detection systems and curation methods

Security Analysis

⚠️ Security Warning: Dataset contains potential secrets

• —Password patterns: 1,231,942 occurrences
• —Token patterns: 353,266 occurrences
• —Secret patterns: 71,778 occurrences
• —API key patterns: 4,899 occurrences

Research Applications

Primary Use Cases

1. 1.Code LLM Robustness: Testing model performance on noisy, real-world data
2. 2.Data Curation Research: Developing automated filtering and cleaning methods
3. 3.License Detection: Training and evaluating license classification systems
4. 4.Bug-Fix Studies: Before/after commit analysis for automated debugging
5. 5.Cross-Language Analysis: Multi-language repository understanding

About This Evaluation Subset

This repository contains the 19GB evaluation subset designed for standardized benchmarks:

• —323 files containing 2,049 repositories
• —Research value scoring with diversity sampling
• —Cross-language implementations and multi-repo analysis
• —Complete build system configurations
• —Enhanced metadata with commit history and issue tracking

Note: The complete 3.05TB CodeReality dataset with all 397,475 repositories is available separately. Contact vincenzo.gallo77@hotmail.com for access to the full dataset.

Demonstration Benchmarks available in eval/benchmarks/:

• —License Detection: Automated license classification evaluation
• —Code Completion: Pass@k metrics for code generation models
• —Extensible Framework: Easy to add new evaluation tasks

Benchmarks & Results

📊 Baseline Performance

Demonstration benchmark results available in eval/results/:

• —`license_detection_sample_results.json` - 9.8% accuracy (challenging baseline)
• —`code_completion_sample_results.json` - 14.2% Pass@1 (noisy data challenge)

🏃 Quick Start Benchmarking

cd eval/benchmarks
python3 license_detection_benchmark.py    # License classification
python3 code_completion_benchmark.py      # Code generation Pass@k

Note: These are demonstration baselines, not production-ready models. Results show expected challenges of deliberately noisy data.

📊 Benchmarks & Results

• —License Detection: 9.8% accuracy baseline (`license_detection_sample_results.json`)
• —Code Completion: 14.2% Pass@1, 34.6% Pass@5 (`code_completion_sample_results.json`)
• —Framework Scaffolds: Bug detection and cross-language translation ready for community implementation
• —Complete Analysis: `benchmark_summary.csv` - All metrics for easy comparison and research use

Usage Guidelines

✅ Recommended Uses

• —Academic research and education
• —Robustness testing of code models
• —Development of data curation methods
• —License detection research
• —Security pattern analysis

❌ Important Limitations

• —No Commercial Use without individual license verification
• —Research Only: Many repositories have unknown licensing
• —Security Risk: Contains potential secrets and vulnerabilities
• —Deliberately Noisy: Requires preprocessing for most applications

⚠️ Important: Dataset vs Evaluation Subset

This repository contains the 19GB evaluation subset only. Some files within this repository (such as docs/DATASET_CARD.md, notebooks in Notebook/, and analysis results) reference or describe the complete 3.05TB CodeReality dataset. This is intentional for research context and documentation completeness.

What's in this repository:

• —✅ Evaluation subset data: 19GB, 2,049 repositories in data/ directory
• —✅ Analysis tools and scripts: For working with both subset and full dataset
• —✅ Documentation: Describes both the subset and the complete dataset methodology
• —✅ Benchmarks: Ready to use with the evaluation subset

Complete Dataset Access (3.05TB):

• —📧 Contact: vincenzo.gallo77@hotmail.com for access to the full dataset
• —📊 Full Scale: 397,475 repositories across 21 programming languages
• —🗂️ Size: 3.05TB uncompressed, 52,692 JSONL files

Who Should Use the Complete Dataset:

• —🎯 Large-scale ML researchers training foundation models on massive code corpora
• —🏢 Enterprise teams developing production code understanding systems
• —🔬 Academic institutions conducting comprehensive code analysis studies
• —📊 Data scientists performing statistical analysis on repository distributions
• —🛠️ Tool developers building large-scale code curation and filtering systems

Advantages of Complete Dataset vs Evaluation Subset:

Choose Complete Dataset When:

• —✅ Training large language models requiring massive code corpora
• —✅ Developing data curation algorithms at scale
• —✅ Studying real-world code distribution patterns
• —✅ Building production-grade code understanding systems
• —✅ Researching cross-language programming patterns
• —✅ Creating comprehensive code quality metrics

Choose Evaluation Subset When:

• —✅ Benchmarking existing models
• —✅ Quick prototyping and testing
• —✅ Learning to work with noisy code datasets
• —✅ Limited storage or computational resources
• —✅ Focused evaluation on curated, high-value repositories

Configuration Files (YAML)

The project includes comprehensive YAML configuration files for easy programmatic access:

Using Configuration Files

import yaml

# Load dataset configuration
with open('dataset-config.yaml', 'r') as f:
    dataset_config = yaml.safe_load(f)

print(f"Dataset: {dataset_config['dataset']['name']}")
print(f"Version: {dataset_config['dataset']['version']}")
print(f"Total repositories: {dataset_config['dataset']['metadata']['total_repositories']}")

# Load analysis configuration
with open('analysis-config.yaml', 'r') as f:
    analysis_config = yaml.safe_load(f)

print(f"Analysis time: {analysis_config['analysis']['methodology']['total_time_hours']} hours")
print(f"Coverage: {analysis_config['analysis']['methodology']['coverage_percentage']}%")

# Load benchmarks configuration
with open('benchmarks-config.yaml', 'r') as f:
    benchmarks_config = yaml.safe_load(f)

for benchmark in benchmarks_config['benchmarks']['available_benchmarks']:
    print(f"Benchmark: {benchmark}")

Documentation

Verification

Verify dataset integrity:

# Check evaluation subset counts
python3 -c "
import json
with open('eval_metadata.json', 'r') as f:
    metadata = json.load(f)
    print(f'Files: {metadata[\"subset_statistics\"][\"total_files\"]}')
    print(f'Repositories: {metadata[\"subset_statistics\"][\"estimated_repositories\"]}')
    print(f'Size: {metadata[\"subset_statistics\"][\"total_size_gb\"]} GB')
"

# Expected output:
# Files: 323
# Repositories: 2049
# Size: 19.0 GB

Citation

@misc{codereality2025,
  title={CodeReality Evaluation Subset: A Curated Research Dataset for Robust Code Understanding},
  author={Vincenzo Gallo},
  year={2025},
  note={Version 1.0.0 - Evaluation Subset (19GB from 3.05TB source)}
}

Community Contributions

We welcome community contributions to improve CodeReality-1T:

🛠️ Data Curation Scripts

• —Contribute filtering and cleaning scripts for the noisy dataset
• —Share deduplication algorithms and quality improvement tools
• —Submit license detection and classification improvements

📊 New Benchmarks

• —Add evaluation tasks beyond license detection and code completion
• —Contribute cross-language analysis benchmarks
• —Share bug detection and security analysis evaluations

📈 Future Versions

• —v1.1.0: Enhanced evaluation subset with community feedback
• —v1.2.0: Improved license detection and filtering tools
• —v2.0.0: Community-curated clean variant with quality filters

🤝 How to Contribute

Community contributions are actively welcomed and encouraged! Help improve the largest deliberately noisy code dataset.

🎯 Priority Contribution Areas:

• —Data Curation: Cleaning scripts, deduplication algorithms, quality filters
• —Benchmarks: New evaluation tasks, improved baselines, framework implementations
• —Analysis Tools: Visualization, statistics, metadata enhancement
• —Documentation: Usage examples, tutorials, case studies

📋 Contribution Process:

1. 1.Clone the repository locally
2. 2.Review existing analysis in the analysis/ directory
3. 3.Develop improvements or new features
4. 4.Test your contributions thoroughly
5. 5.Submit your improvements via standard collaboration methods

💡 Join the Community: Share your research, tools, and insights using CodeReality!

Support & Access

Evaluation Subset (This Repository)

• —Documentation: See docs/ directory for comprehensive information
• —Analysis: Check analysis/ directory for current research insights
• —Usage: All benchmarks and tools work directly with the 19GB subset

Complete Dataset Access (3.05TB)

• —🔗 Full Dataset Request: Contact vincenzo.gallo77@hotmail.com
• —📋 Include in your request:
• —Research purpose and intended use
• —Institutional affiliation (if applicable)
• —Technical requirements and storage capacity
• —⚡ Response time: Typically within 24-48 hours

General Support

• —Technical Questions: vincenzo.gallo77@hotmail.com
• —Documentation Issues: Check docs/ directory first
• —Benchmark Problems: Review benchmarks/ and results/ directories

Dataset created using transparent research methodology with complete reproducibility. Analysis completed in 63.7 hours with 100% coverage and no sampling.