NationalLibraryOfScotland/Britain-and-UK-Handbooks-Dataset

Britain and UK Handbooks Dataset

Dataset Description

This dataset contains digitised Britain and UK Handbooks from the National Library of Scotland's digital collections. These annual reference books were originally published by the British Information Service (starting in 1946) to provide overseas readers with comprehensive information about the United Kingdom.

Dataset Summary

• —Source: National Library of Scotland - Britain and UK Handbooks
• —Time Period: 1954-2005
• —Format: Image-text pairs with OCR output
• —Processing: Converted from ALTO XML and JPG images to Hugging Face dataset format
• —Contents: Official annual reference books covering UK government policy, economy, social welfare, defence, religion, science, and arts
• —Size: 27,915 pages, 13.9 million words

Dataset Structure

Data Fields

Each record in the dataset contains the following fields:

• —document_id (string): Unique identifier for the handbook document
• —page_number (string): Page number within the document
• —image (image): Scanned image of the handbook page
• —text (string): OCR-extracted text from the page
• —alto_xml (string): Raw ALTO XML containing detailed OCR information
• —has_image (bool): Whether the page has an associated image file
• —has_alto (bool): Whether the page has associated ALTO OCR data

Usage

Loading the Dataset

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("davanstrien/nls-handbooks-test")

# Access the data
for example in dataset['train']:
    print(f"Document: {example['document_id']}")
    print(f"Page: {example['page_number']}")
    print(f"Text preview: {example['text'][:200]}...")
    break

Filtering for Complete Records

# Filter for pages with both images and OCR text
complete_pages = dataset.filter(lambda x: x['has_image'] and x['has_alto'])
print(f"Complete pages: {len(complete_pages)}")

Dataset Creation

Source Data

The original data comes from the National Library of Scotland's digitisation of Britain and UK Handbooks - annual reference publications that provided factual overviews of the UK, including government policy, national economy, social welfare, defence, religion, science, and arts.

Processing Pipeline

1. 1.Original Format: High-resolution JPG scans and ALTO XML files with OCR output
2. 2.Conversion Process: Using the convert_to_hf_dataset.py script to pair images with corresponding ALTO XML and extract plain text while preserving metadata

Considerations for Using this Data

Historical Context

These handbooks represent official UK government information from 1954 to 2005 and should be understood within their historical and political context.

OCR Quality

Text extraction quality varies based on original document condition and print quality.

Additional Information

Licensing

• —Items up to 1968: Public domain
• —Items from 1969 onwards: Open Government License v.3.0

Please refer to the National Library of Scotland's terms of use for complete licensing information.

Citation

@misc{nls_uk_handbooks,
  title={Britain and UK Handbooks},
  author={National Library of Scotland},
  howpublished={\url{https://data.nls.uk/data/digitised-collections/britain-uk-handbooks/}},
}

@misc{uk_handbooks_hf,
  title={Britain and UK Handbooks - Hugging Face Dataset},
  author={van Strien, Daniel},
  year={2024},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/datasets/davanstrien/nls-handbooks-test}},
}

Acknowledgments

Thanks to the National Library of Scotland for digitising and making these historical reference materials available for research.