pin-team/oercommons-v1-optimized

OERCommons v1 Optimized

Authors: Junjie Wang and Yuhan Sun Hosted by: PIN Team Dataset: `pin-team/oercommons-v1-optimized`

OERCommons v1 Optimized is a provenance-preserving multimodal pretraining corpus built on the OERCommons subset of The Common Pile v0.1, which serves as its upstream data and licensing baseline. We extend it with full-page recovery, canonical Markdown, ordered image/PDF/link metadata, conservative corrections, and integrity evidence.

At a glance

The distributed Parquet has exactly seven top-level fields and 6,251 unique IDs. quality_signals.ppl is null for every row because no frozen PPL scorer was run.

What we built

• —Audited all 9,339 immutable upstream rows and consolidated duplicates into 6,251 stable lesson IDs without discarding provenance.
• —Recovered valid article.lesson pages and used the immutable Hugging Face snapshot only when live lesson content was unavailable.
• —Rendered headings, lists, tables, links, formulas, authored questions, and image positions as canonical Markdown instead of flat excerpts.
• —Aligned text with ordered image, PDF, and link objects carrying paths, source URLs, hashes, byte sizes, retrieval state, and license evidence.
• —Limited GLM-4.7 to narrow semantic windows; accepted patches must replay exactly and preserve numbers, URLs, Markdown, and media ordering.
• —Validated schema, patch replay, quality counts, Parquet read-back, media paths, MIME types, sizes, and SHA-256 hashes deterministically.

Common Pile: 9,339 rows
          │ audit + deduplicate
          ▼
   6,251 stable lessons
      ┌───┴────────────┐
      ▼                ▼
live pages: 6,221   HF snapshots: 30
      └───┬────────────┘
          ▼
Markdown + ordered media + provenance
          ▼
conservative patches → Parquet + media/

Dataset structure

record
├── content
│   ├── original
│   └── optimized
├── content_media
│   ├── images ──> media/images/...
│   ├── pdfs   ──> media/pdfs/...
│   └── links  ──> source URLs
├── meta + license
├── quality_signals
└── optimization + replayable patches

content.original is canonical Markdown, not raw HTML or flattened Common Pile text. content.optimized differs only where a localized correction passed all acceptance checks.

Quick start

from datasets import load_dataset
from huggingface_hub import hf_hub_download

repo_id = "pin-team/oercommons-v1-optimized"
dataset = load_dataset(repo_id, split="train")

row = dataset[0]
text = row["content"]["optimized"]
changed = row["content"]["original"] != text
print(row["id"], changed, text[:300])

def download_first_pathful(items):
    item = next((item for item in items if item["file_path"]), None)
    if item is None:
        return None
    return hf_hub_download(
        repo_id=repo_id,
        repo_type="dataset",
        filename=item["file_path"],
    )

local_image = download_first_pathful(row["content_media"]["images"])
local_pdf = download_first_pathful(row["content_media"]["pdfs"])

load_dataset() reads the Parquet main table; it does not automatically download the entire media/ tree. Text-only users can remove standalone <img src="media/images/..." /> lines. Multimodal users can resolve those paths on demand and verify them against the same-row metadata.

Standard record and real example

Every full row follows the seven-field structure above. This is a compact, selected-field view of the real Parquet record oercommons:lesson:122659; values are unchanged, while unshown fields remain in the dataset.

{
  "id": "oercommons:lesson:122659",
  "license": "CC-BY-4.0",
  "content_media": {
    "images": [{
      "id": "img_7ec2274d41ac82a24e1b",
      "file_path": "media/images/7e/img_7ec2274d41ac82a24e1b.jpg",
      "mime_type": "image/jpeg",
      "redistribution_status": "downloaded_local"
    }],
    "pdfs": [{
      "id": "119075",
      "file_path": "media/pdfs/68/68ca95cb814e898f9ecbdf0d4d5a61684a582120433ed4c61a195d08483e52a6.pdf",
      "mime_type": "application/pdf",
      "redistribution_status": "downloaded_local"
    }]
  },
  "quality_signals": {
    "char_count": {"original": 799, "optimized": 796},
    "missing_image_source_count": 0,
    "unavailable_resource_count": 0,
    "ppl": null
  },
  "optimization": {
    "status": "accepted",
    "content_source": "live_page",
    "patches": [
      {"before": "Informatation", "after": "Information", "category": "typo"},
      {"before": "science ,", "after": "science,", "category": "spacing"}
    ]
  }
}

The record's Markdown contains <img src="media/images/7e/img_7ec2274d41ac82a24e1b.jpg" /> at the source position, matching content_media.images[0].file_path. Its linked PDF is a separate, hash-addressed media object. This ordered text-media relationship is the multimodal unit; provenance and rights status remain explicit per object.

Validation, licensing, and limitations

The local all-media package passed structural acceptance:

• —Parquet read-back: 6,251 rows and 6,251 unique IDs.
• —Missing required top-level fields: 0.
• —Patch replay, media path/hash, missing-file, and orphan-file failures: 0.
• —Source gaps remain explicit: 530 image occurrences lack a recoverable source URL, and 19 resources are unavailable across 13 records.

Media states are intentionally distinct:

• —downloaded_local: byte-verified local research copy; rights not independently cleared.
• —source_only: source provenance retained, with no distributed local path.
• —source_missing_url: the page contains an image occurrence but no recoverable URL.
• —unavailable: the resource could not be retrieved or verified.

The current all-media acceptance reports hub_upload_ready=false: 22,889 media occurrences are downloaded_local, and none are bundled. Uploading those binaries is therefore a publisher policy decision, not evidence of cleared rights. The mixed corpus keeps license: other; inspect license, meta.ori_meta.license_url, and media-level status before use or redistribution.

Other limitations are concise but important: OER Commons pages and links can change; 30 records depend on frozen snapshots; source metadata can be empty; the corpus is English-dominant rather than guaranteed English-only; and the localized optimizer does not certify pedagogical quality, factual correctness, accessibility, or currentness.

Citation and provenance

Please cite this dataset release first:

@misc{wang2026oercommonsv1optimized,
  title        = {OERCommons v1 Optimized},
  author       = {Wang, Junjie and Sun, Yuhan},
  year         = {2026},
  publisher    = {PIN Team},
  howpublished = {Hugging Face dataset},
  url          = {https://huggingface.co/datasets/pin-team/oercommons-v1-optimized}
}

For upstream provenance, also cite the Common Pile paper:

@article{kandpal2025common,
  title   = {The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text},
  author  = {Kandpal, Nikhil and others},
  journal = {arXiv preprint arXiv:2506.05209},
  year    = {2025}
}

Primary provenance links:

• —This dataset
• —OER Commons
• —Common Pile OERCommons
• —Common Pile source code
• —The Common Pile v0.1 paper