kadubon/paper-tex-corpus

K. Takahashi Paper TeX Corpus

This dataset publishes K. Takahashi's TeX research corpus in four complementary views: canonical DOI records, uncatalogued archive records, section-aware retrieval chunks, and an inventory of the original source ZIPs. The unmodified ZIP files are available under raw/, while all viewer-facing data is provided directly as Parquet.

What this dataset is for

The purpose of this dataset is to make a collection of scholarly TeX sources usable as a searchable, citable, and auditable research corpus. A directory of ZIP files preserves the manuscripts, but it is difficult to search across papers, connect a passage to its DOI, or determine which file and version produced a result. This dataset adds those missing layers without replacing the original sources:

• —a bibliographic layer links each catalogued work to its DOI, title, abstract, publication date, keywords, and canonical URL;
• —a source layer preserves the full TeX text and byte-identical source ZIP so that a result can be inspected in its original context;
• —a retrieval layer supplies section-aware chunks that retain TeX mathematics and avoid splitting structural environments where possible; and
• —a provenance layer records checksums, archive members, mappings, duplicate relationships, and quality flags so that downstream results can be traced and rebuilt.

The intended outcome is not merely easier downloading. It is a reproducible path from finding a relevant passage, to identifying the paper and DOI, to checking the underlying TeX source and archive. This is useful when answers, search results, or corpus statistics need evidence that can be followed back to a specific scholarly document.

Research content overview

The papers form a connected, theory-oriented research program on how autonomous and self-modifying intelligent systems can remain viable, interpretable, governable, and physically grounded when no infallible external evaluator is available. The corpus asks how claims about intelligence, safety, autonomy, value, persistence, or improvement can be stated in operational terms and checked using finite observations, explicit assumptions, resource constraints, and reproducible evidence.

Major, overlapping research strands include:

• —Self-organizing and self-improving intelligence. Early and continuing papers study computational autopoiesis, active inference, collective adaptive intelligence, teleogenesis, and architectures that can revise their own models or organization while preserving specified viability or value constraints.
• —Observable-only and "no-meta" assurance. A large part of the corpus examines agents that cannot rely on a trusted meta-judge. It develops audit gates, proof- or evidence-carrying claims, typed contracts, replayable records, provenance rules, fail-closed controls, and institutional mechanisms for deciding what can be supported from observable data.
• —Persistence, semantics, observation, and memory. Papers analyze how identity, meaning, values, and predictive organization behave under coarse-graining, self-modification, finite context, partial logging, ontology drift, and non-Markovian memory. Related work studies semantic phase transitions and limits on stable representation.
• —Physical and thermodynamic constraints. The research treats computation and agency as processes embedded in open physical systems. Topics include free-energy and entropy-production principles, exergy and resource accounting, energy-memory-compute trade-offs, stochastic thermodynamics, and physically explicit boundaries and ledgers.
• —Mathematical structures for comparison and dynamics. The corpus uses category theory, information geometry, optimal transport and Hellinger--Kantorovich/Bures geometry, gradient flows, dynamical systems, control theory, information theory, and causal inference to compare models and describe change across scales.
• —AI systems, scaling, and multi-agent operation. Applied theoretical papers address LLM routing, inference reuse, memory telemetry, long-running agents, training bottlenecks, silent data corruption, compute and I/O limits, multi-agent coordination, and the conditions under which distributed inference or verification is beneficial.
• —Governance, welfare, and human--AI coexistence. Other papers connect the technical framework to rights, consent, non-coercive assistance, public claim certification, institutional accountability, work and welfare, benevolent propagation, and human--AI or organizational systems.

Across the collection, the emphasis shifts from broad architectures and axiomatic proposals for self-organizing intelligence toward increasingly operational frameworks based on measurable interfaces, causal identification, uncertainty sets, runtime monitoring, physical accounting, and machine-checkable certificates. This is a thematic guide, not a claim that every paper uses all of these concepts or that the proposed theories have been empirically validated. The authoritative description of each work is its catalog title, abstract, keywords, and linked DOI record.

Dataset snapshot

Source snapshot:

• —TeX archive commit: 1f1d98bd6e3f1b0bfd4002d81b67f22db51f9229
• —Research catalog state: 2026-08-30
• —Dataset release: v1.1.0

Which config should I start with?

All configs have one train split representing the complete corpus; train does not mean that the records have been assigned to a machine-learning training partition.

For most document-level analysis, begin with papers. For retrieval systems, begin with chunks and use its paper/document identifier and DOI fields to join back to papers. Use archives when exact source provenance matters. Add archive_only only when coverage beyond the current publication catalog is required.

Configs

papers (default)

One row per canonical scholarly DOI in the machine-readable publication catalog. Catalog metadata is authoritative. Source fields are empty when no TeX source can be established without guessing. A DOI can point to multiple component source manuscripts; the primary source is kept in the singular fields and every source is listed in the source_* arrays.

archive_only

ZIPs that are not safely attributable to a current catalog DOI. These include older versions, supplements, derivative manuscripts, and unresolved title candidates. No DOI is inferred for these rows.

chunks

Deterministic, section-aware chunks built from unique valid primary TeX sources. Chunking targets about 4,000 characters, allows up to 6,000 characters, and reuses up to 400 characters of complete trailing blocks. The pipeline does not split equation, theorem, proof, or verbatim environments. char_start and char_end refer to the comment-stripped document body. Both the original TeX fragment and a conservative readable projection are included.

archives

One row per original ZIP. It records SHA-256 checksums, member metadata, DOI mappings, duplicate relationships, and quality flags. The raw/*.zip files are byte-for-byte copies of the source backup.

Quick use

from datasets import load_dataset

papers = load_dataset("kadubon/paper-tex-corpus", "papers", split="train")
chunks = load_dataset("kadubon/paper-tex-corpus", "chunks", split="train")

print(papers[0]["title"], papers[0]["doi"])
print(chunks[0]["section_title"], chunks[0]["chunk_text"][:500])

DuckDB can query the Parquet files directly:

SELECT doi, title, mapping_status
FROM read_parquet(
  'https://huggingface.co/datasets/kadubon/paper-tex-corpus/resolve/main/data/papers/train-00000-of-00001.parquet'
)
LIMIT 10;

Provenance and mapping

Bibliographic metadata comes from the publication index and its `research-catalog.json`. Mappings use, in order, explicit evidence recorded in config/manual_mappings.json, a unique DOI in the manuscript front matter, exact normalized titles, or a strong unique title match. Similarity-only candidates below the acceptance threshold remain ambiguous or archive_only. The complete decision record is metadata/catalog-crosswalk.csv.

metadata/source-state.json, build-report.json, and checksums.sha256 make the release auditable. scripts/build_dataset.py regenerates all Parquet files and raw copies; scripts/validate_dataset.py performs structural, checksum, security-pattern, cross-config, PyArrow, DuckDB, and optional 🤗 Datasets checks. CITATION.cff provides machine-readable corpus citation metadata.

Practical use cases

Citation-grounded search and RAG

Index chunk_text for readable retrieval or chunk_tex when exact TeX syntax matters. After retrieval, carry the DOI, paper/document identifier, section path, and position into the application response. A user or evaluator can then open the corresponding papers row, inspect the complete tex_source, and follow canonical_url to the publication. This structure supports citation-grounded systems, but the dataset does not itself verify that a generated answer is entailed by a retrieved chunk.

Math- and TeX-aware retrieval research

The corpus retains equations in TeX instead of replacing them with MathML or a normalized formula language. It can therefore support experiments on tokenization, lexical and semantic retrieval, reranking, chunking, or representation learning for documents in which mathematical notation and document structure are important. Results may depend on author-specific macros and conservative TeX-to-text conversion, so comparisons should report the fields and preprocessing used.

Corpus analysis and reproducible preprocessing

The papers view supports document-level analyses using publication metadata and complete source text. The chunks view supports passage-level analyses while preserving section context. Because source and generated artifacts have SHA-256 identifiers and the build scripts are included, researchers can describe an input snapshot precisely and compare alternative extraction, parsing, deduplication, or chunking pipelines.

Archival and provenance work

The archives config and raw/ directory can be used to check whether a derived record matches an original package, inspect auxiliary files, study source-package composition, or reconstruct the corpus. Invalid, duplicated, multi-manuscript, ambiguous, and catalogue-external cases are represented explicitly rather than removed, which allows users to define and report their own inclusion policy.

Training and tool development

Under CC BY 4.0 attribution, the corpus can be used as input for model pretraining, fine-tuning, parser development, LaTeX tooling, or other preprocessing research. Users should create their own task-specific splits and evaluation criteria, prevent unintended train/test overlap caused by related or duplicate manuscripts, and retain paper-level attribution where outputs expose source content.

What this dataset does not provide

• —It is not an evaluation benchmark, answer key, or set of verified ground-truth answers.
• —It does not certify the scientific correctness, novelty, or current validity of a paper.
• —It does not include embeddings, a vector database, a retrieval service, or a trained model.
• —It does not normalize equations or convert them to MathML.
• —It does not guarantee that general-purpose TeX parsers can expand every author macro.
• —It does not define a train/validation/test split; each config's train split is the complete released view.

Limitations

• —Inclusion does not certify the scientific correctness of a manuscript.
• —TeX-to-text conversion is conservative and may retain formatting commands.
• —No MathML conversion or equation normalization is included in v1.
• —A small number of source packages are invalid, duplicated, multi-manuscript, or not attributable to a current DOI; these states are explicit rather than silently repaired.
• —Public author contact information and ORCID values present in the source are retained.
• —The corpus records source provenance, not whether any particular writing tool or assistance process was used.

License and citation

The dataset and included source materials are released under CC BY 4.0. Attribute K. Takahashi, cite this dataset, and cite each paper's DOI when using individual works.

Suggested BibTeX:

@dataset{takahashi_paper_tex_corpus_2026,
  author    = {Takahashi, K.},
  title     = {K. Takahashi Paper TeX Corpus},
  year      = {2026},
  version   = {1.1.0},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/kadubon/paper-tex-corpus}
}

Canonical publication records and paper-specific citation links are available at <https://kadubon.github.io/github.io/works.html>.