SciCodePile/SciCode-Domain-Code

1---2license: apache-2.03task_categories:4  - text-generation5language:6  - code7tags:8  - code9  - scientific-computing10  - domain-specific11  - chemistry12  - biology13  - physics14size_categories:15  - 1M<n<10M16---17 18# DATA1: Domain-Specific Code Dataset19 20## Dataset Overview21 22DATA1 is a large-scale domain-specific code dataset focusing on code samples from interdisciplinary fields such as biology, chemistry, materials science, and related areas. The dataset is collected and organized from GitHub repositories, covering 178 different domain topics with over 1.1 billion lines of code.23 24## Dataset Statistics25 26- **Total Datasets**: 178 CSV files27- **Total Data Size**: ~115 GB28- **Total Lines of Code**: Over 1.1 billion lines29- **Data Format**: CSV (Comma-Separated Values)30- **Encoding**: UTF-831 32## Dataset Structure33 34Each CSV file corresponds to a specific domain topic, with the naming format `dataset_{Topic}.csv`, where `{Topic}` is the domain keyword (e.g., Protein, Drug, Genomics).35 36### Data Field Description37 38Each CSV file contains the following fields:39 40| Field Name | Type | Description |41|------------|------|-------------|42| `keyword` | String | Domain keyword used to identify the domain of the code sample |43| `repo_name` | String | GitHub repository name (format: owner/repo) |44| `file_path` | String | Relative path of the file in the repository |45| `file_extension` | String | File extension (e.g., .py, .java, .cpp) |46| `file_size` | Integer | File size in bytes |47| `line_count` | Integer | Number of lines of code in the file |48| `content` | String | Complete file content |49| `language` | String | Programming language (e.g., Python, Java, C++) |50 51## Domain Categories52 53The dataset covers the following major domain categories:54 55### Biology-Related56- **Molecular Biology**: Protein, DNA, RNA, Gene, Enzyme, Receptor, Ligand57- **Cell Biology**: Cell_biology, Single_cell, Cell_atlas, Organoid58- **Genomics**: Genomics, Genotype, Phenotype, Epigenetics, Metagenomics59- **Transcriptomics**: Transcriptomics, Spatial_Transcriptomics, Transcription, Translation60- **Proteomics**: Proteomics, Protein_Protein_Interactions, Folding61- **Metabolomics**: Metabolomics, Metabolic, Lipidomics, Glycomics62- **Systems Biology**: System_biology, Signaling, Pathway, Networks63 64### Chemistry-Related65- **Computational Chemistry**: Computational_Chemistry, Quantum_Chemistry, DFT, QM_MM66- **Medicinal Chemistry**: Drug, ADMET, QSAR, Docking, Lead_discovery, Lead_optimization67- **Materials Chemistry**: Material, Crystal, Conformation, Chemical_space68- **Reaction Chemistry**: Reaction, Kinetics, Mechanism, Redox69 70### Medicine and Pharmacology71- **Pharmacology**: Pharmacology, Pharmacokinetics, Pharmacogenomics, Pharmacogenetics72- **Medicine**: Medicine, Disease, Diagnostics, Pathology, Vaccine73- **Toxicology**: Toxicology, Biomarker, Marker74 75### Computational Methods76- **Machine Learning**: Transformer, GAN, VAE, Diffusion, Flow_matching, Reinforcement_learning77- **Quantum Computing**: Quantum_mechanics, Quantum_biology, Electronic_structure78- **Modeling Methods**: Modeling, Multi_scale_modeling, Agent_based_model, Stochastic_modeling79- **Numerical Methods**: Monte_Carlo, Finite_element_method, Phase_field_technique80 81### Other Specialized Fields82- **Bioinformatics**: Bioinformatics, Cheminformatics, Next_generation_sequencing83- **Bioengineering**: Bioengineering, Biotechnology, Biosensors84- **Immunology**: Immunology, Antibody, Antigen, Antagonist85- **Virology**: Viral, Pandemic, Pathogens, AMR (Antimicrobial Resistance)86 87## Data Source88 89The data is collected from open-source repositories on GitHub through the following process:90 911. **Keyword Search**: Search for relevant repositories on GitHub using domain-specific keywords922. **Repository Filtering**: Filter repositories based on relevance scores and code quality933. **File Extraction**: Extract code files from filtered repositories944. **Categorization**: Classify files into corresponding topic datasets based on keywords and domain characteristics95 96## Dataset Characteristics97 981. **Wide Domain Coverage**: Covers multiple interdisciplinary fields including biology, chemistry, materials science, and medicine992. **Diverse Code Types**: Includes multiple programming languages such as Python, Java, C++, R, and MATLAB1003. **Large Scale**: Over 1.1 billion lines of code with a total data size of 115 GB1014. **Structured Storage**: Each domain topic is stored independently as a CSV file for convenient on-demand usage1025. **Rich Metadata**: Contains comprehensive metadata including repository information, file paths, and language types103 104## Usage Guidelines105 106### Data Loading107 108```python109import pandas as pd110 111# Load dataset for a specific domain112df = pd.read_csv('dataset_Protein.csv')113 114# View basic dataset information115print(f"Dataset size: {len(df)} files")116print(f"Programming language distribution: {df['language'].value_counts()}")117print(f"File type distribution: {df['file_extension'].value_counts()}")118```119 120### Data Filtering121 122```python123# Filter by programming language124python_files = df[df['language'] == 'Python']125 126# Filter by file size (e.g., files smaller than 100KB)127small_files = df[df['file_size'] < 100000]128 129# Filter by line count130medium_files = df[(df['line_count'] > 50) & (df['line_count'] < 1000)]131```132 133### Domain-Specific Analysis134 135```python136# Analyze code characteristics for a specific domain137protein_df = pd.read_csv('dataset_Protein.csv')138print(f"Number of code files in Protein domain: {len(protein_df)}")139print(f"Average file size: {protein_df['file_size'].mean():.2f} bytes")140print(f"Average line count: {protein_df['line_count'].mean():.2f} lines")141```142 143## Important Notes144 1451. **File Size**: Some dataset files are large (up to several GB), please be mindful of memory usage when loading1462. **Encoding**: All files use UTF-8 encoding; ensure proper handling of special characters if encountered1473. **Data Quality**: Data is sourced from public repositories and may vary in code quality; preprocessing is recommended before use1484. **License Compliance**: Please comply with the license requirements of the original repositories when using the data149 150 151