gregoiremarty/basketball-science-oa

Basketball Science Open Corpus

214 basketball research papers, 8,970 passages, redistributed in full text — because every single document was checked, one by one, against the licence its publisher declared.

Most retrieval corpora assembled from the open web are redistributed on the assumption that "open access" means "reusable". It does not. This corpus starts from the opposite assumption: nothing is included until an authority has been asked, per document, what the licence actually is.

Pipeline, benchmark and the five retractions behind it: github.com/gregoirembv/hoops-brain

Why this exists

It was assembled as the theory half of a working system. Its counterpart is Metrics, a statistical tracker for the WNBA and women's college basketball: numbers on their own say very little, and a shot profile only means something against the theory it is supposed to express. This corpus is the literature that sits beside the tracker.

That made it the substrate for a cross-lingual retrieval experiment: an almost entirely English corpus, queried in French. That asymmetry is the normal condition of most real systems outside the anglosphere, and it is rare in public benchmarks, which tend to be either monolingual or built on general-domain text (news, Wikipedia).

Specialised, technical, and modest in size is also the regime most production retrieval systems actually live in — and the regime where domain vocabulary makes lexical search look deceptively strong.

Files

documents.jsonl     214 rows  doc_id, titre, auteurs, annee, doi, url,
                              licence, langue, axe, type
passages.jsonl    8,970 rows  passage_id, doc_id, chapitre, rang, langue, texte
ATTRIBUTION.tsv     215 rows  attribution table required by the BY clause
statistiques.json             the counters reproduced on this page

Field names are French because the pipeline that produces them is French. They are stable and documented; renaming them would break the provenance chain back to the source repository.

from datasets import load_dataset

docs = load_dataset("gregoiremarty/basketball-science-oa", "documents")["train"]
passages = load_dataset("gregoiremarty/basketball-science-oa", "passages")["train"]

print(passages[0]["texte"][:300])

How it was built

1. 1.Discovery. OpenAlex, filtered at query time on best_oa_location.license:cc0|cc-by|cc-by-sa over the Basketball concept.
2. 2.Retrieval. PDF download, text extraction, splitting on the publisher's declared chapters.
3. 3.Fallback. Several publishers (MDPI, Hindawi) return HTTP 403 to automated clients while the article itself is CC BY. For those, the full text is fetched as JATS XML from Europe PMC, which additionally supplies the publisher's own section structure — better input than PDF extraction. 32 documents came in this way, and would otherwise have been lost. Overall ingestion success: 198 documents of 308 attempted (64%).
4. 4.Post-hoc licence audit. Every document is re-queried against OpenAlex by DOI, falling back to title match with Jaccard ≥ 0.8. The licence recorded in this dataset is the answer to that query, not an inference from the discovery filter. This step matters: 342 of 374 ingested documents carried no licence field, because the discovery queue predated the field. Assuming they were CC would have been convenient and wrong.

Passages target 1,400 characters with 200 characters of overlap, cut on paragraph boundaries; fragments under 250 characters are dropped. Documents short enough to fit a single note are chunked as one unit, others chapter by chapter.

What was excluded, and why

Counts are in notes (a document is one or more notes), as reported by jeu/exporter.py:

The audit found 244 documents whose licence permits redistribution. 214 are published here — the remaining 30 produced no passage long enough to keep (extraction failure, or a stub note with no body).

ND is excluded because a corpus split into passages is a derivative work, which the clause forbids. NC is excluded for a different reason: redistributing it would not be unlawful, but it would contaminate the dataset and forbid its users any commercial use. An evaluation set nobody is allowed to use is not an evaluation set.

Known limitations

• —The corpus is monolingual in practice (99% English). It supports measuring the query→document language gap in one direction only.
• —PDF extraction leaves debris. Journal banners, submission dates and page numbers are fused into the text of short documents. The first passage of a single-note document frequently opens with the journal masthead. This has not been cleaned, because cleaning it silently would be a second, undocumented transformation of the source.
• —The licence is the one OpenAlex reports, which is itself the publisher's declaration. An upstream error propagates here.
• —Personal data has been redacted, not perfected. PDF extraction pulls in authors' correspondence addresses; 420 email addresses and file paths were replaced with [courriel retire] / [chemin retire] by a regular expression. Regular expressions miss things. If you find residual personal data, open an issue and it will be removed.
• —No relevance judgments are included in this version. Queries and qrels ship separately, together with their measured error rate — see below.
• —Retraction status is not tracked. A paper withdrawn after publication keeps its CC licence and stays in the corpus. For retrieval evaluation this is harmless; for anyone mining the text for claims about the sport, it is not. Cross-check against Retraction Watch before treating any document as evidence.

Why the queries are not here yet

The queries that accompany this corpus went through three construction protocols, and the protocol changed the verdict of the same comparison three times. Hand-written queries borrowed the vocabulary of their targets and favoured lexical search. Machine-generated queries built from a one-sentence prompt hallucinated: roughly half described something other than their target, often the opposing side of the play. Queries generated from the full note and verified blind are faithful, but easier than hand-written ones.

None of the three sets is neutral. What can exist is a set whose bias is known, measured and published — which is why the query release waits on a human validation pass whose error rate will ship with it.

Attribution

ATTRIBUTION.tsv lists title, authors, year, licence and source URL for every document, plus the DOI where one exists (187 of 214). The BY clause requires this table to travel with any redistribution. The 31 CC BY-SA documents further require derivative works to be shared under the same terms.

Licence of the aggregate

This dataset aggregates CC BY and CC BY-SA works. The most restrictive term governs the whole: CC BY-SA 4.0.

Citation

@misc{basketball_science_oa_2026,
  title  = {Basketball Science Open Corpus: a licence-audited retrieval corpus},
  author = {Marty, Gr\'egoire},
  year   = {2026},
  note   = {CC BY-SA 4.0}
}