ftosoni/mediawiki-code2code-search

MediaWiki Code2Code Search — Dataset

Pre-computed retrieval artifacts for MediaWiki Code2Code Search, a neural system for the semantic discovery of open-source software entities (functions, types, templates) across the MediaWiki / Wikimedia ecosystem.

This Hugging Face dataset is a complementary mirror of the pre-computed artifacts archived on Zenodo (10.5281/zenodo.20586256). The Hugging Face copy makes the corpus browsable in the dataset Viewer and easy to pull with the datasets / huggingface_hub libraries.

If you use this dataset, please cite the paper (see Citation). Zenodo is the archival home of the artifacts; the paper is the reference to cite.

• —Code / system: <https://github.com/ftosoni/mediawiki-code2code-search>
• —Paper: MediaWiki Code2Code Search: Neural Retrieval for the Semantic Discovery of Open-Source Software Entities — Francesco Tosoni, Sant'Anna School of Advanced Studies, Pisa. A peer-reviewed version is *forthcoming in SoftwareX***. See Citation.

Contents

Note on `.npy` / `.index` / `.pkl`: these binary artifacts do not render in the Viewer by design; the Parquet shards provide the browsable view of the same underlying rows. Also on Zenodo only: the raw pre-migration corpus raw_snippets.json (~1.75 GiB) lives in the Zenodo record but is intentionally not mirrored here — its content is already captured by snippets.db and the Parquet shards.

Data fields (snippets table / Parquet)

Corpus statistics

• —1,289,452 structural entities:
• —function: 1,050,748
• —type: 237,653
• —template: 1,051
• —2,242 active repositories (distinct (repo_group, repo_name) pairs) across the MediaWiki / Wikimedia ecosystem.
• —12 programming languages extracted via language-specific structural parsing.
• —Embedding model: `Qwen/Qwen3-Embedding-0.6B` (1024-dim). Retrieval is nearest-neighbour by Euclidean (L2) distance over L2-normalised vectors.

Usage

Browse the corpus in the Viewer above, or load it programmatically.

Tabular rows (Parquet) with `datasets`:

from datasets import load_dataset

ds = load_dataset("ftosoni/mediawiki-code2code-search", split="train")
print(ds)
print(ds[0]["name"], ds[0]["type"])
print(ds[0]["swhid"])

Raw serving artifacts with `huggingface_hub`:

from huggingface_hub import hf_hub_download

repo = "ftosoni/mediawiki-code2code-search"
db    = hf_hub_download(repo, "snippets.db",      repo_type="dataset")
emb   = hf_hub_download(repo, "embeddings.npy",   repo_type="dataset")
index = hf_hub_download(repo, "mediawiki.index",  repo_type="dataset")
bm25  = hf_hub_download(repo, "bm25_index.pkl",   repo_type="dataset")

import numpy as np, faiss
X = np.load(emb, mmap_mode="r")        # (1289452, 1024) float32, L2-normalised
faiss_index = faiss.read_index(index)  # IndexIVFPQ (L2)

embeddings.npy row i corresponds to snippets.db row id = i and to FAISS vector i.

Licensing (important)

This dataset combines two distinct layers, and they carry different licenses:

1. 1.Tooling / packaging (schema, indexes, embeddings, this card): Apache-2.0, matching the canonical Zenodo record.
2. 2.The `code` snippets in snippets.db / the Parquet shards are derived from upstream MediaWiki / Wikimedia repositories, each under its own free/open-source license (predominantly GPL-2.0-or-later, with some MIT / BSD / Apache-2.0 and others). These upstream licenses govern the snippet text; redistribution here relies on their permission to redistribute.

There is no per-row `license` column. Instead, every entity carries an `swhid` with an origin= qualifier, so the exact upstream repository — and therefore its authoritative license and copyright — can be resolved on Software Heritage and at the origin repository. Snippets are fragments and may not carry their file's original license header; consult the origin before reuse.

See `LICENSE.md` for the full statement.

Citation

Please cite the paper:

@misc{tosoni2026mediawikicode2codesearchneural,
  title         = {MediaWiki Code2Code Search: Neural Retrieval for the Semantic Discovery of Open-Source Software Entities},
  author        = {Francesco Tosoni},
  year          = {2026},
  eprint        = {2607.26766},
  archivePrefix = {arXiv},
  primaryClass  = {cs.IR},
  url           = {https://arxiv.org/abs/2607.26766}
}

The pre-computed artifacts are archived on Zenodo: 10.5281/zenodo.20586256. A peer-reviewed version of the paper is forthcoming in SoftwareX.

Acknowledgements

Corpus derived from the MediaWiki / Wikimedia software ecosystem. Provenance is tracked with SWHID identifiers resolvable on Software Heritage. Embeddings computed with `Qwen/Qwen3-Embedding-0.6B`; indexing with FAISS.