EliasHossain/nanobubbleeval

NanoBubbleEval v1.0

## ⚠ For NeurIPS reviewers — use this Croissant URL Please do NOT use the URL exposed by the "Use this dataset → Croissant" button at the top-right of this page. That URL triggers a known bug in mlcroissant==1.0.16 (the version pinned by the NeurIPS Croissant validator Space) and produces a FilterFiles error that does not reflect a problem with the dataset itself. Use this URL instead — copy the line below verbatim into the validator's "URL Input" tab: ``text https://huggingface.co/datasets/EliasHossain/nanobubbleeval/resolve/main/croissant.json ` The same file is also available at [metadata/croissant.json](https://huggingface.co/datasets/EliasHossain/nanobubbleeval/blob/main/metadata/croissant.json) — open that link to view the file in the HuggingFace viewer (the resolve/ URL above is served as text/plain so a browser will display the JSON as text rather than rendering it; that is expected and does not affect the validator, which fetches the bytes programmatically). Equivalent code snippet for mlcroissant: `python from mlcroissant import Dataset ds = Dataset(jsonld="https://huggingface.co/datasets/EliasHossain/nanobubbleeval/resolve/main/croissant.json") records = ds.records("gold-hard-records") # note: id has hyphens, not underscores `` This URL passes JSON Format ✓, Croissant Schema ✓, and Records Generation ✓ on the same validator the AC uses.

An evidence-grounded benchmark for schema extraction, numerical grounding, and evidence attribution in the nanobubble and nanocarrier literature.

Summary

NanoBubbleEval operationalises three failure modes of scientific information extraction as decomposed evaluation tasks over a shared record set:

1. 1.Schema-fill hallucination — emitting a value when the source is silent.
2. 2.Numerical grounding under unit normalisation — drifting between nm and µm, hours and days, percent and fraction.
3. 3.Verbatim evidence attribution — citing a span that genuinely contains the answer.

The benchmark contributes:

• —a 18-field normalised schema with an explicit NOT_REPORTED convention scored under abstention-calibrated F1,
• —a tolerance-bounded numerical match metric over canonicalised units that disentangles unit drift from value drift, and
• —an answer–evidence consistency rate that detects cite-hallucination — predictions that emit a correct value paired with an unsupported span.

v1.0 release at a glance

• —Final warehouse size: 51,566 deduplicated records (post-recovery, 2026-05 snapshot).
• —Gold-hard tier containment: 40/40 records resolve into the released warehouse.
• —Provenance branch: B — original 2026-03 warehouse unrecovered; May 2026 re-harvest merged with direct-identifier re-fetches for the 14 records the query-based re-harvest did not return.
• —Abstract cross-check (X / Y / Z):
• —X = 14 records: source-API abstract is byte-identical to the annotation-time abstract.
• —Y = 19 records: differ only after Unicode/whitespace normalisation (mojibake repair, NFKC, whitespace collapse).
• —Z = 7 records: source-side editorial revisions or mismatches (e.g., PubMed structured-abstract section labels added post-annotation).
• —14 + 19 + 7 = 40.

The full audit log lives in `verification/` and is machine-readable via `verification/summary.json`.

Annotation tiers

Schema

Six headline fields (numerical: size, zeta_potential, stability, loading_efficiency; text: payload, release_profile) and twelve provisional fields (bubble_type, material_identity, application_category, generation_method, characterization_method, medium_environment, outcome_claim, evidence_span, ambiguity_flag, plus three additional unit slots).

Numerical headline fields carry a canonical unit (size: nm; zeta potential: mV; stability: h; loading efficiency: %) and a unit-synonym table. Out-of-vocabulary unit strings are flagged but never silently coerced.

Files in this release

v1.0 (shipped here)

v1.1 (scheduled, regenerable from the released pipeline scripts)

Provenance and recovery audit

The original 2026-03 warehouse from which the gold-hard tier was sampled was destroyed in a project-deletion incident. Local recovery was exhausted (filesystem, git history, and HuggingFace dataset history were all checked). Rather than ship a warehouse from which the evaluation tier could not be derived, we re-anchored the warehouse to the gold-hard records by direct identifier lookup against the source APIs:

• —26 / 40 records were already present in the May 2026 query-based re-harvest (matched by DOI or normalised title).
• —14 / 40 records were missing from the re-harvest. For each missing record, the canonical source-API identifier encoded in the gold-hard record_id (12 PubMed DOIs, 27 OpenAlex DOIs, 1 Europe PMC DOI across the full tier) was used to re-fetch metadata directly via PubMed E-utilities (idconv → efetch), OpenAlex /works/doi:, or EuropePMC search-by-DOI. All 14 re-fetches succeeded.
• —After deduplication (DOI → PMID/PMCID → normalised title → URL), the released warehouse contains 51,566 records and is a strict superset of the gold-hard tier by record_id (40/40 resolve directly via the containment audit in verification/gold_hard_containment_after_merge.csv).
• —Annotation-time abstracts from the gold-hard file are preserved verbatim in the warehouse for the 14 re-fetched rows; baseline predictions remain exactly reproducible from the released warehouse.

The full identifier-anchored recovery pipeline lives in the GitHub repo at `scripts/verification/` and is rerunnable end-to-end from a clean checkout.

Baselines

Bootstrap 95% CIs computed by record-level resampling with replacement (n_boot=1000, seed=42).

Headline finding

On the gold-hard tier, B3 dominates B2 by 21.3 points and B1 by 26.5 points of macro Acal-F1, with non-overlapping CIs across all three baselines. Unit accuracy is 1.000 across all three baselines, indicating that the bottleneck is which property to extract rather than how to canonicalise its unit. B3's macro Acal-F1 falls from 0.786 on original-research abstracts to 0.449 on review-paper abstracts — a 33.7-point drop, more than twice the corresponding drop for B1 (12.6 points) or B2 (15.6 points), suggesting the LLM is more sensitive to review-paper paraphrase than the simpler baselines.

Versioning

The benchmark is versioned semantically. v1.0 ships the post-recovery deduplicated warehouse manifest, the 40-record gold-hard tier, the three baseline-prediction files, the gold-hard recovery audit logs (verification/), the Croissant 1.1 metadata, the schema specification, and the slice and leakage reports. v1.1 will add the 8,006-record high-precision core, the 500-record gold pool, the 460-record gold-lite extension tier, the three task-view CSVs, the deterministic splits.json, finalised dual annotation on the gold-hard tier, an open-access full-text gold subset under the same schema, and a same-family-larger ceiling baseline plus a retrieval-augmented variant. Items in the v1.1 list are regenerable end-to-end from the released pipeline scripts; we did not ship them in v1.0 to keep the release scope strictly to what has been audited and validated.

Code

The accompanying Python package, pipeline scripts, and unit-test suite are released at:

https://github.com/eliashossain001/nanobubbleeval

License

The dataset is released under CC BY-NC 4.0 (research use only). The codebase is released under MIT.