CoolFace
Datasetpublic

uv-scripts/dataset-creation

Dataset Creation Scripts Ready-to-run scripts for creating Hugging Face datasets from local files. Available Scripts 📄 pdf-to-dataset.py Convert directories of PDF files into Hugging Face datasets. Features: 📁 Uploads PDFs as dataset objects for flexible processing 🏷️ Automatic labeling from folder structure 🚀 Zero configuration - just point at your PDFs 📤 Direct upload to Hugging Face Hub Usage: # Basic usage uv run pdf-to-dataset.py… See the full description on the dataset page: https://huggingface.co/datasets/uv-scripts/dataset-creation.

sourceHugging Faceupdated 1y agoView on Hugging Face
7likes72downloads
Dataset Card

Dataset Creation Scripts

Ready-to-run scripts for creating Hugging Face datasets from local files.

Available Scripts

📄 pdf-to-dataset.py

Convert directories of PDF files into Hugging Face datasets.

Features:

  • —📁 Uploads PDFs as dataset objects for flexible processing
  • —🏷️ Automatic labeling from folder structure
  • —🚀 Zero configuration - just point at your PDFs
  • —📤 Direct upload to Hugging Face Hub

Usage:

bash
# Basic usage
uv run pdf-to-dataset.py /path/to/pdfs username/my-dataset

# Create private dataset
uv run pdf-to-dataset.py /path/to/pdfs username/my-dataset --private

# Organized by categories (folder structure creates labels)
# /pdfs/invoice/doc1.pdf → label: "invoice"
# /pdfs/receipt/doc2.pdf → label: "receipt"
uv run pdf-to-dataset.py /path/to/organized-pdfs username/categorized-docs

Output Format: The script creates a dataset where each example contains a pdf object that can be processed using the datasets library. Users can then extract text, convert to images, or perform other operations as needed.

python
from datasets import load_dataset

# Load your uploaded dataset
dataset = load_dataset("username/my-dataset")

# Access PDF objects
pdf = dataset["train"][0]["pdf"]

Requirements:

  • —Directory containing PDF files
  • —Hugging Face account (for uploading)
  • —No GPU needed - runs on CPU

Installation

No installation needed! Just run with uv:

bash
# Run directly from GitHub
uv run https://huggingface.co/datasets/uv-scripts/dataset-creation/resolve/main/pdf-to-dataset.py --help

# Or clone and run locally
git clone https://huggingface.co/datasets/uv-scripts/dataset-creation
cd dataset-creation
uv run pdf-to-dataset.py /path/to/pdfs my-dataset

Authentication

Scripts use Hugging Face authentication:

  1. 1.Pass token via --hf-token argument
  2. 2.Set HF_TOKEN environment variable
  3. 3.Use cached credentials from huggingface-cli login

Examples

Create a Dataset from Research Papers

bash
uv run pdf-to-dataset.py ~/Documents/papers username/research-papers

Organize Documents by Type

bash
# Directory structure:
# documents/
# ├── invoices/
# │   ├── invoice1.pdf
# │   └── invoice2.pdf
# └── receipts/
#     ├── receipt1.pdf
#     └── receipt2.pdf

uv run pdf-to-dataset.py documents/ username/financial-docs
# Creates dataset with labels: "invoices" and "receipts"

Tips

  • —Large PDFs: The script handles large PDFs efficiently by uploading them as objects
  • —Organization: Use subdirectories to automatically create labeled datasets
  • —Privacy: Use --private flag for sensitive documents
  • —Processing: After upload, use the datasets library to extract text, images, or metadata as needed

License

MIT