Arinkc/pydoc-llama-codesearchnet-curated

PyDoc-Llama: Curated Python Docstring Dataset

A curated subset of CodeSearchNet specifically prepared for supervised fine-tuning of Llama 3.1 8B to generate Google-style Python docstrings.

Dataset Summary

• —Source: CodeSearchNet (Python subset, 412,178 raw examples)
• —Final size: 24,970 examples
• —Format: Llama 3.1 chat conversations (system + user + assistant messages)
• —Target output: Google-style Python docstrings

Curation Pipeline

1. 1.Multi-criteria quality filter (30.66% pass rate):
2. 2.Code length: 100-2000 characters
3. 3.Docstring length: 30-500 characters
4. 4.Minimum 5-8 words depending on structure
5. 5.<5% non-ASCII content
6. 6.No TODO/FIXME markers

1. 1.Style enforcement — accepts only:
2. 2.Docstrings with Google sections (Args:, Returns:, Raises:)
3. 3.Clean prose docstrings (≤4 lines, <400 chars)
4. 4.Rejects reStructuredText, JavaDoc, and NumPy styles

1. 1.Random sampling of 25,000 examples (seed=42)

1. 1.AST-based docstring stripping to prevent input-output leakage. Filters out structurally-invalid examples where the "docstring" was placed mid-function rather than as the first statement.

1. 1.Chat formatting for Llama 3.1 with system prompt

Splits

Token Length Distribution

Measured with the Llama 3.1 tokenizer:

• —Median: 201 tokens
• —p90: 359 tokens
• —p99: 514 tokens
• —Maximum: 799 tokens

The short token lengths enable training with max_seq_length=1024, roughly halving memory requirements compared to standard 2048-token training.

Example

{
  "messages": [
    {"role": "system", "content": "You are an expert Python documentation writer..."},
    {"role": "user", "content": "Generate a Google-style docstring for this function:\n\n```python\ndef ...\n```"},
    {"role": "assistant", "content": "Returns the name of the port that this motor is connected to."}
  ]
}

Usage

from datasets import load_dataset

dataset = load_dataset("Arinkc/pydoc-llama-codesearchnet-curated")
print(dataset['train'][0])

Evaluation Results

Model trained on this dataset achieved:

• —0% hallucination rate (vs 11% for base model) on 200 held-out examples
• —100% format compliance (capital-letter starts, no preamble)
• —Training loss: 2.3 → 0.63 over 4,212 steps

See GitHub repo for full results.

Limitations

• —Source data quality varies. While the filter removes most low-quality examples, some grammatically awkward but well-formatted docstrings remain.
• —Google-style bias. The dataset is filtered to one specific docstring style. Models fine-tuned on this data will produce Google-style output even when other styles might be appropriate.
• —English only. Non-English content is filtered out.

Citation

If you use this dataset, please cite the original CodeSearchNet paper:

@article{husain2019codesearchnet,
  title={CodeSearchNet Challenge: Evaluating the State of Semantic Code Search},
  author={Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc},
  journal={arXiv preprint arXiv:1909.09436},
  year={2019}
}