me-aas/CodeX-2M-Thinking

1---2license: apache-2.03pretty_name: CodeX-5M-Thinking4dataset_name: Modotte/CodeX-5M-Thinking5size_categories:6- 1M<n<10M7language:8- en9task_categories:10- text-generation11- question-answering12tags:13- Coding14- Code15- CodeX16- Modotte17- LLM-training18- synthetic19- curated20- benchmark21- reasoning-dataset22- artifact23annotations_creators:24- machine-generated25- expert-verified26source_datasets:27- Modotte internal synthetic generation28multilinguality:29- monolingual30---31# Modotte32---33 34<p align="center">35  <img 36    src="https://cdn-uploads.huggingface.co/production/uploads/677fcdf29b9a9863eba3f29f/ZP4YDDIRewH5M-jKmE4Rt.png" 37    alt="CodeX Banner" 38    width="70%" 39    style="border-radius:15px;"40  />41 42 43> Note: This dataset is part of the lineup CodeX by Modotte. You can get lots of datasets in this same lineup, with the main focus on providing very high-quality datasets for model training and fine-tuning. 44 45This dataset is fully synthetic, curated from high-quality public sources and enhanced with synthetic data generated using both closed and open-source models. It serves as a strong foundation for instruction-based model tuning and fine-tuning, offering one of the most refined and extensive corpora available for coding tasks with reasoning.46 47### Key Features48- **Scale**: 2 million examples of highly curated coding data49- **Diversity**: Comprehensive coverage of programming domains from basic syntax to advanced software engineering50- **Quality**: Multi-stage filtering and verification processes, including ranking-based filtering and expert selections51- **Thinking Focus**: Step-by-step reasoning included in responses, optimized for instruction training with detailed thought processes52- **Accuracy**: Verified code executions and correctness validation using automated testing frameworks53 54## Dataset Overview55**CodeX-2M-Thinking** is a meticulously curated coding dataset designed specifically for instruction-based model tuning and fine-tuning of existing models with enhanced code generation and reasoning capabilities. This fully synthetic dataset represents a large and comprehensively filtered corpus of coding data on the Hugging Face platform, emphasizing a thinking approach with step-by-step reasoning for deeper model training.56 57## How to Use?58```bash59pip install -U datasets fsspec60```61```python62from datasets import load_dataset63dataset = load_dataset("Modotte/CodeX-2M-Thinking")64```65 66### Key Features67- **Scale**: 2 million examples of highly curated coding data68- **Diversity**: Comprehensive coverage of programming domains from basic syntax to advanced software engineering69- **Quality**: Multi-stage filtering and verification processes, including ranking-based filtering and expert selections70- **Thinking Focus**: Step-by-step reasoning included in responses, optimized for instruction training with detailed thought processes71- **Accuracy**: Verified code executions and correctness validation using automated testing frameworks72 73## Data Curation Process74This dataset has been carefully constructed through a fully synthetic approach, selectively generating and merging examples to enrich the overall dataset for generation models.75 76### Data Sources77- **High-Quality Existing Datasets**: Curated from multiple premium coding datasets available online (e.g., from NVIDIA and Modotte's internal collections)78- **Synthetic Generation**: Fully generated using both closed-source and open-source language models (Modotte)79- **Expert Validation**: Human-verified code solutions, reasoning, and implementations (Modotte)80 81### Filtering Pipeline82Our rigorous filtering process includes open and closed-source filtering techniques, ensuring only the highest-quality examples are retained:831. **Deduplication**: Removal of duplicate problems and code solutions842. **Normalization**: Code formatting standardization and syntax cleanup853. **Stopword Processing**: Intelligent removal of non-essential comments or boilerplate864. **Quality Scoring**: Multi-dimensional quality assessment using metrics like code complexity, readability, and efficiency875. **Ranking-Based Filtering**: Advanced ranking algorithms to prioritize top-tier examples based on relevance, novelty, and utility886. **Expert Selections**: Manual curation by coding experts to select exemplary samples897. **Answer Verification**: Automated testing and execution validation using frameworks like pytest or unit tests908. **Content Filtering**: Removal of inappropriate, outdated, or incorrect code919. **Diversity Balancing**: Ensuring balanced representation across languages and domains through algorithmic sampling92 93### Problem Complexity Distribution94- **Basic Level** (30%): Fundamental programming concepts, simple syntax, and basic operations95- **Intermediate Level** (30%): Multi-function problems requiring modular code and basic algorithms96- **Advanced Level** (40%): Complex challenges involving data structures, optimization, and system design97 98### Programming Domains Covered99- Algorithms and Data Structures100- Web Development and Frameworks101- Machine Learning and AI Implementations102- System Programming and Operating Systems103- Database Management and SQL/NoSQL104- Software Engineering Best Practices105- Competitive Programming Problems106> Note: Domains are for reference only. The actual data is very diverse and covers more domains than stated. The actual data includes more complex and high-level questions than stated, spanning multiple programming languages such as Python, Java, C++, JavaScript, and others.107 108## Use Cases109- **Fine-tuning** code generation and reasoning capabilities in language models110- **Training** instruction-following models with a coding and reasoning focus111- **Benchmarking** model performance on coding tasks, problem-solving, and logical reasoning112- **Research** in AI-assisted programming, automated code completion, and explainable AI113- **Educational** applications requiring step-by-step code explanations and reasoning114 115## Dataset Format116Each example contains:117- **Problem Statement**: Clear coding challenge or task description118- **Step-by-Step Solution**: Detailed reasoning process119- **Code Solution**: Final executable code with integrated reasoning120 121## Quality Assurance122- **Automated Verification**: All code solutions verified using execution environments and testing suites123- **Correctness Guarantee**: Only problems with verified correct and functional code are included124- **Human Review**: Sample validation by coding experts125- **Automated Checks**: Static analysis, linting, and runtime verification where applicable126- **Open and Closed-Source Filtering**: Integration of proprietary and community-driven tools for enhanced quality control127 128## Performance Metrics129Models trained on this dataset show significant improvements in:130- Code generation accuracy with reasoning131- Efficiency in producing detailed, step-by-step solutions132- Problem-solving speed and logical coherence133- Cross-language and cross-domain code transfer134- Reduction in hallucinated or erroneous code outputs through better reasoning135 136## Acknowledgments137Special thanks to our partners and contributors:138- **NVIDIA** - Reference datasets; CodeX contains many examples taken from NVIDIA's existing datasets139- **Modotte Team** - Dataset curation, quality assurance, along with customly generated examples140 141## Citation142**Anyone** can freely use and modify this dataset.143 144## License145This dataset is released under [apache-2.0].146 147```bibtex148@dataset{codex2024,149  title={CodeX-2M-Thinking: Large-Scale Coding Dataset with Reasoning},150  author={Parvesh Rawal at Modotte},151  year={2024},152  publisher={Modotte},153  url={https://huggingface.co/datasets/Modotte/CodeX-2M-Thinking}154}155```156 157## Contact158For questions, suggestions, or collaboration opportunities:159- **Email**: [Modotte](team@modotte.com)160- **Twitter**: [@Modotte]161- **GitHub**: [Modotte]162 163---164*Built with ❤️ by Modotte - Advancing AI through high-quality data*