jamie8johnson/cqs-code-search-200k

CQS Code Search 200K

Balanced code search training dataset with call graph metadata.

Overview

199,998 (query, code) pairs across 9 programming languages, extracted from ~5,000 high-quality GitHub repositories using cqs semantic code indexing.

Unique features:

• —Perfectly balanced: 22,222 pairs per language
• —Call graph metadata: caller/callee names per function (from tree-sitter AST analysis)
• —Enriched NL queries: generated from type signatures, doc comments, and call context
• —9 languages: Go, Java, JavaScript, PHP, Python, Ruby, Rust, TypeScript, C++

Intended Use

Training code search embedding models. This dataset produced both `e5-base-v2-code-search` and `bge-large-v1.5-code-search`. The call graph metadata enables:

• —False-negative filtering: skip hard negatives that are callers/callees of the positive
• —Structural training signals: teach embeddings that callers are semantically related

Dataset Structure

Each record contains:

Example row

{
  "query": "query: Test helper function to filter a collection of compilation database entries fn filter_entries<I>(filter: &SourceEntryFilter, entries: I) -> Vec<Entry> ...",
  "positive": "passage: fn filter_entries<I>(filter: &SourceEntryFilter, entries: I) -> Vec<Entry>\n    where I: IntoIterator<Item = Entry>,\n{\n    entries.into_iter().filter(|entry| filter.should_include(entry)).collect()\n}",
  "language": "rust",
  "function_name": "filter_entries",
  "file": "bear/src/output/clang/filter_sources.rs",
  "repo": "rizsotto/Bear",
  "source": "stack-cqs-index",
  "callers": ["get_entries", "read_directories", "run", "test_filter_entries_method"],
  "callees": ["filter", "into_iter", "should_include", "collect"]
}

Files

• —cqs-code-search-200k.jsonl — the dataset (199,998 rows, ~229 MB).
• —processing_manifest.jsonl — per-repo extraction provenance (sidecar, not part of the dataset). One row per source repo with {repo, language, status, pairs, edges, source_files, note}.

Creation

Extracted using cqs v1.7.0:

1. 1.Clone repos from GitHub (filtered by stars, size).
2. 2.cqs index each repo (tree-sitter parsing, call graph extraction, NL query generation).
3. 3.Extract (NL, code) pairs from the cqs index.
4. 4.Balance to 22,222 per language.
5. 5.Attach call graph edges from the same index.

The dataset reflects the v1.7.0-era extraction pipeline. cqs has matured significantly since (current release: v1.33.0) — newer extractions would use higher-quality NL generation and richer metadata, but the underlying (query, code) pairs remain useful as-is for training general-purpose code-search embeddings.