aschachner/cy-database

cy-database — Calabi–Yau geometries for string-compactification workflows

Precomputed Calabi–Yau threefold data, organised as a family of sub-datasets, accessed through the `stringforge` infrastructure package. Each sub-dataset covers one class of Calabi–Yau constructions and is keyed by a class-specific identifier system.

This top-level card describes the conventions, layout, and loading interface that are common to all sub-datasets. Each sub-dataset has its own card with construction-specific details.

Available sub-datasets

toric/ is much the largest sub-dataset — 113,913,015 FRST phases over 1,056,121 polytopes, plus 3,497,945 VEX phases — and is the one that differs structurally from the others: see Where `toric/` differs before using it.

Quick start

pip install stringforge

from stringforge import CYDatabase, TDFDatabase, CICYDatabase, LCSDatabase, KKLTDatabase

# 1.  Pure I/O on the TDF sub-dataset (no JAXVacua import)
db = TDFDatabase()                                # downloads catalogue only (~10 MB)
df = db.query(h11=2, has_conifolds=True)          # catalogue-level filter, no shard I/O

# 2.  Same as 1, but in mirror convention and producing JAXVacua model objects
lcs = LCSDatabase(dataset="tdf")                  # mirror-convention wrapper
tree = lcs.load(                                  # returns a jaxvacua.lcs.lcs_tree
    ks_id=int(df.iloc[0]["ks_id"]),
    triang_id=int(df.iloc[0]["triang_id"]),
    h12=int(df.iloc[0]["h12"]),                   # h12 in mirror convention
    include_gv=True,
    include_conifolds=True,
)

# 3.  Plain CICY access
cicy = CICYDatabase()
cicy_df = cicy.query(h11=3)

# 4.  Curated KKLT index (logical links into TDF; no geometry duplication)
kklt = KKLTDatabase()
polys = kklt.query_polytopes(Q_min=100)

# 5.  Toric CY phases (FRST / VEX). Local build only -- see "Where toric/ differs".
from stringforge import ToricCYDatabase
toric = ToricCYDatabase.from_local("/path/to/cy-database")   # the dir holding toric/, or toric/
pcat  = toric.query_polytopes(h11=4, fav_N=True)             # shared polytope layer
frst  = toric.query("frst", h11=4, ks_id=1)                  # phases of one polytope
geom  = toric.load("frst", h11=4, ks_id=1, triang_id=0, in_basis=True)

The LCSDatabase class is the recommended entry point for JAXVacua workflows: it operates in the mirror convention used by lcs_tree / FluxVacuaFinder and exposes load_model(...) to construct a fully initialised FluxVacuaFinder in one call.

Repository layout

aschachner/cy-database/
    README.md                       ← this file (umbrella card)

    tdf/                            ← Trilayer, Double Favourable toric models
        README.md                   ← TDF card
        catalog.parquet
        conifold_catalog.parquet
        schema.json
        manifest.json
        lcs_data/h11_{N}/
        gv/h11_{N}/
        conifolds/h11_{N}/
        polytope/
        extra/

    cicy/                           ← Complete-intersection threefolds
        README.md                   ← CICY card
        catalog.parquet
        schema.json
        manifest.json
        lcs_data/h11_{N}/
        gv/

    kklt/                           ← Curated KKLT index over tdf/
        README.md                   ← KKLT card
        catalog.parquet                       ← polytope-grain
        conifold_class_catalog.parquet        ← class-grain
        conifold_catalog.parquet              ← conifold-grain (with TDF link)
        schema.json
        gv/h11_{N}/

    toric/                          ← FRST + VEX phases of the Kreuzer–Skarke polytopes
        README.md                   ← toric card
        schema.json                           ← NOTE: no monolithic catalog.parquet
        manifest.json
        polytope_catalog/h11_{N}/             ← shared polytope layer, + both modes' counts
        polytope/h11_{N}/                     ← vertices, GLSM basis, charge matrix
        polytope_vex_counts/h11_{N}/
        frst/catalog/h11_{N}/                 ← thin per-phase catalogue, sharded
        frst/geom/h11_{N}/                    ← heights, kappa (COO), c2
        vex/catalog/h11_{N}/
        vex/geom/h11_{N}/

Shared design

All sub-datasets follow the same conventions.

Format

• —Apache Parquet shards for all bulk data.
• —One small catalog.parquet per sub-dataset, serving as the lazy-download entry point. KKLT additionally carries a class-grain and a conifold-grain catalogue.

Lazy, on-demand access

The `stringforge.cy_io.CYDatabase` class downloads only the files required by a given query:

• —db.query(...) → catalogue only (~10 MB).
• —db.load(...) → catalogue + the specific shard(s) needed for one model.
• —db.load_batch(...) → catalogue + shards for a batch of models.

Constructing a database object performs no network access; the first query downloads the catalogue, and only subsequent model-loading steps download shard files.

Cache modes

• —cache_mode="persistent" (default): shards are cached in memory (LRU) and on disk. Optimal for repeated access.
• —cache_mode="none": shards are downloaded, the requested row is read, and the file is deleted immediately. Ideal for scanning millions of models without filling local disk.

Offline mode

For HPC clusters without outbound network access, set offline=True. All data is served from the local cache; any missing shard raises FileNotFoundError instead of triggering a network call. The common pattern is to warm the cache on a login node and replay on worker nodes.

Schema versioning

Each sub-dataset carries a schema.json file with an integer schema_version. CYDatabase checks it against the client library's stringforge.SCHEMA_VERSION and raises a clear SchemaVersionError on incompatibility. toric/ is versioned independently against stringforge.toric_normalize.TORIC_SCHEMA_VERSION, because its layout evolves separately from the monolithic sub-datasets.

Bucketing by $h^{1,1}$

Where row sizes scale strongly with $h^{1,1}$ (e.g. triple intersection tensors are $O(h^3)$), data is sub-bucketed by $h^{1,1}$ (directories named h11_{N}/). This keeps small-$h^{1,1}$ users from downloading large-$h^{1,1}$ data, and vice versa. Flat splits (small, uniform rows) are not bucketed.

Adaptive shard sizing

For large buckets (e.g. conifolds at high $h^{1,1}$ with millions of rows), shard sizes are chosen adaptively to target ≈ 30 files per bucket, clamped to $[500,\; 50\,000]$ rows. See each sub-dataset's card for specifics.

Mirror convention

Catalogues store `h11` and `h12` in catalogue convention (typically small h11, large h12). JAXVacua works in the mirror convention (the two are swapped). Use `stringforge.lcs_database.LCSDatabase` — which inherits from CYDatabase — when working with mirror-convention models; it transparently swaps the two columns at the boundary.

Where toric/ differs

toric/ holds over \\(10^{8}\\) phases, which forces two departures from the shared design above. Both are visible in the API, so read this before treating it like tdf/ or cicy/.

Two consequences worth stating plainly:

• —*`ks_id` is unique only within one \\(h^{1,1}\\).* It is the Kreuzer–Skarke emission order, so the same ks_id names a different polytope at each \\(h^{1,1}\\). An identifier without h11 is ambiguous, and CYPhase.from_database raises rather than guessing.
• —`mode` selects the equivalence, and the two are not nested as phases. frst counts distinct CYTools cy()-classes of fine star triangulations; vex counts Wall classes — equal in-basis \\((\kappa, c_2)\\) — of not-necessarily-fine ones. Every VEX polytope is also an FRST polytope (vex ⊆ frst as sets of polytopes, which is why the polytope layer is shared), but that is not a containment on phases: the non-fine family is larger, so a polytope routinely has more VEX classes than FRST classes.

Because attribute queries stream one shard with predicate pushdown and point lookups go through the per-\\(h^{1,1}\\) _ksid_index, neither operation ever materialises a whole bucket — but you must always pass h11, and at \\(h^{1,1} \ge 10\\) you should never ask for an unfiltered catalogue.

Geometry objects: CYPhase

Alongside the database readers, stringforge exposes a small class family that wraps one geometry as an object. It serves the stored data immediately and only materialises CYTools when you ask for something that genuinely needs it, so import cytools never happens on the fast path.

The base class carries exactly the Wall data \\((h^{1,1}, h^{2,1}, \kappa, c_2)\\) — by Wall's theorem (Invent. Math. 1 (1966) 355), together with torsion, that fixes the diffeomorphism type of a smooth simply-connected threefold. Construction-specific material lives in the subclasses.

from stringforge import CYPhase, ToricCYDatabase

db = ToricCYDatabase.from_local("/path/to/cy-database")

# from_database dispatches on the sub-dataset and returns a ToricCYPhase.
# All four of (mode, h11, ks_id, triang_id) are required.
phase = CYPhase.from_database(db, mode="frst", h11=4, ks_id=1, triang_id=0)

phase.hodge_numbers                          # (h11, h21)
phase.euler_characteristic                   # 2 * (h11 - h21)
phase.intersection_numbers(in_basis=True)    # kappa, COO rows (i, j, k, value)
phase.second_chern_class(in_basis=True)      # c2
phase.wall_hash.hex()                        # diffeomorphism pre-filter

# CYTools only from here on, materialised once and cached
cy = phase.to_cytools()                      # CalabiYau for frst, Fan for vex
phase.kahler_cone(version="cup")
phase.gv_invariants(max_deg=3)               # a degree cutoff is required by CYTools
phase.to_lcs_tree()                          # bridge to JAXVacua
assert phase.verify()                        # stored values == a fresh CYTools recompute

in_basis=True slices the stored prime-toric form to the GLSM basis. The stored form is out-of-basis, indexed by prime toric divisors, so in_basis=False (the default) returns a longer vector — oob_dim = basis_dim + 4 entries — and to_dense() likewise expands to oob_dim.

Two cases refuse rather than return something misleading:

• —Non-favorable polytopes (fav_N=False): the GLSM basis spans a proper subspace of \\(H^{1,1}(X)\\), so basis_is_complete is False, in-basis quantities describe only the toric part and warn when accessed, and full_intersection_numbers() / full_second_chern_class() raise.
• —VEX phases: a not-necessarily-fine triangulation has no cy(), so to_cytools() returns a CYTools Fan and the CalabiYau-only features — GV invariants, mori_cone(version="cap"), kahler_cone(version="cup"), to_lcs_tree() — raise NotImplementedError.

CICY geometries load the same way. They live in the lcs_data split rather than a catalogue, so they come through LCSDatabase, and the loader un-swaps the stored mirror convention (see the CICY card) so that h11, h12 and chi mean the same thing as they do for a toric phase:

from stringforge import CICYPhase, LCSDatabase

quintic = CICYPhase.from_database(LCSDatabase(dataset="cicy"), cicy_id=7890)
quintic.hodge_numbers            # (1, 101)
quintic.euler_characteristic     # -200
quintic.basis_is_complete        # True only for the 4,511 Kaehler-favourable rows

CICYPhase deliberately exposes no wall_hash: that sub-dataset records no basis identification, so a CICY wall_hash would not be comparable to anything.

Loading without stringforge

All sub-datasets are plain Parquet and can be read with any compatible tool:

import pandas as pd
from huggingface_hub import hf_hub_download

catalog_path = hf_hub_download(
    repo_id="aschachner/cy-database",
    filename="tdf/catalog.parquet",
    repo_type="dataset",
)
catalog = pd.read_parquet(catalog_path)

Each catalogue contains (shard_id, row_index) pointers into the sub-dataset's data splits, so you can resolve individual rows by path construction. See each sub-dataset card for the catalogue schema and resolved path templates.

Versioning and rebuilds

Builds are incremental: unchanged models (tracked by SHA-256 hash of their source files) are skipped, and only new or changed models are appended. Major layout changes bump SCHEMA_VERSION in `stringforge/cy_io.py`.

Scope and limitations

• —Data is precomputed; not all fields are present for every model. Use has_gv=True, has_conifolds=True, etc. to filter in query().
• —Catalogues use pandas nullable Int64 for optional shard pointers; consumers that do not support nullable integers should cast or drop missing rows.
• —Sub-datasets are independent: different constructions live in separate top-level directories and use different identifier schemes.

Citation

If you use this dataset, please cite:

@article{XXX
}

Application papers include arXiv:2307.15749, arXiv:2308.15525, and arXiv:2501.03984.

Each sub-dataset card lists additional references specific to its construction.

Licence

GPL-3.0, matching the `stringforge` and `jaxvacua` libraries.

Contact

Issues, questions, and contributions: <https://github.com/AndreasSchachner/stringforge/issues>.