Datasetpublic

Svngoku/great-works-from-africa-and-the-pacific-bernard-de-grunne-new-york-2008

large-chunk-ocr-data Dataset created with PDF2Dataset -- OCR + structure-aware chunking pipeline. Dataset Summary Metric Value Total chunks 31 Avg chars/chunk 731 Avg images/chunk 1.58 Source files 1 Duplicates removed 0 Quality filtered 1 Schema Column Type Description chunk_id string Unique identifier: filename_chunk_N text string Raw markdown chunk with image refs text_clean string Cleaned text without… See the full description on the dataset page: https://huggingface.co/datasets/Svngoku/great-works-from-africa-and-the-pacific-bernard-de-grunne-new-york-2008.

sourceHugging Facemitupdated 4mo agoView on Hugging Face

0likes16downloads

Dataset Card

large-chunk-ocr-data

Dataset created with PDF2Dataset -- OCR + structure-aware chunking pipeline.

Dataset Summary

Metric	Value
Total chunks	31
Avg chars/chunk	731
Avg images/chunk	1.58
Source files	1
Duplicates removed	0
Quality filtered	1

Schema

Column	Type	Description
`chunk_id`	`string`	Unique identifier: `filename_chunk_N`
`text`	`string`	Raw markdown chunk with image refs
`text_clean`	`string`	Cleaned text without markdown formatting
`chapter`	`string`	H1 header active at chunk position
`section`	`string`	H2 header active at chunk position
`subsection`	`string`	H3+ header active at chunk position
`images`	`list[Image]`	Rendered images extracted from chunk (viewable in Dataset Viewer)
`image_refs`	`list[string]`	Image reference IDs in chunk text
`num_images`	`int`	Number of images in chunk
`has_images`	`bool`	Whether chunk contains images
`source_filename`	`string`	Original source file name
`start_index`	`int`	Character offset in source document
`char_count`	`int`	Character count of chunk text

Source Files

File	Chunks
`great-works-from-africa-and-the-pacific-bernard-de-grunne-new-york-2008.pdf`	31

Document Structure

Chapters found in the source documents:

—Is African sculpture classical?
—VII. KONGO KINGDOM, DEMOCRATIC REPUBLIC OF CONGO
—XVII. LOWER SEPIK RIVER, SINGRIN STYLE, PAPUA NEW GUINEA
—XVIII. LOWER SEPIK RIVER, PAPUA NEW GUINEA
—XXI. EASTER ISLAND

Pipeline

This dataset was processed through the PDF2Dataset pipeline:

1.OCR -- Mistral OCR extracts text and images from PDF/image files
2.Chunking -- Structure-aware recursive splitting preserves document hierarchy
3.Validation -- Schema validation ensures every chunk has required fields
4.Deduplication -- Content-hash based dedup removes identical chunks
5.Quality Filtering -- Removes empty, too-short, or malformed chunks