CoolFace
Apppublic

MrKAMELEON/ReferencesExtractor

sourceHugging Facemitupdated 1y agoView on Hugging Face
0likes
App README

PDF References Extractor

A specialized Python tool for automatically extracting and formatting bibliographic references from academic PDF papers. The tool supports multiple languages and reference formats, making it particularly useful for processing research papers, theses, and academic publications.

Key Features

  • —Intelligent Reference Detection: Automatically locates the references section using multiple language markers (References, Bibliographie, Referenzen, etc.)
  • —Smart Reference Parsing: Identifies individual references based on their structure and formatting
  • —Multi-Language Support: Works with references in various languages and formats
  • —Duplicate Detection: Automatically filters out similar or duplicate references using fuzzy matching
  • —Format Standardization: Normalizes reference formatting while preserving the original content
  • —Batch Processing: Process multiple PDF files in bulk through the CLI
  • —Web Interface: User-friendly Streamlit interface for processing individual files
  • —Quality Validation: Built-in testing framework to validate extraction accuracy
  • —Docker Support: Easy deployment using containerization

Setup

This project can be run either locally with Poetry or using Docker.

Local Setup with Poetry

  1. 1.Install Poetry (if you haven't already):
bash
curl -sSL https://install.python-poetry.org | python3 -
  1. 1.Install dependencies:
bash
poetry install

Docker Setup

  1. 1.Build the Docker image:
bash
docker build -t pdf-references-extractor .
  1. 1.Run the container:
bash
docker run -p 8501:8501 pdf-references-extractor

Usage

Command Line Interface (Batch Processing)

  1. 1.Create an input_files directory and place your PDF files in it:
bash
mkdir input_files
cp your-papers/*.pdf input_files/
  1. 1.Run the extraction:
bash
poetry run python src/pdf_processor.py

The script will:

  • —Process all PDF files in the input_files directory
  • —Create an output_files directory with extracted references
  • —Save references for each paper as Bibliografia {paper_name}.txt
  • —Display progress and any processing issues

Web Interface (Single File Processing)

  1. 1.Start the Streamlit app:
bash
# Using Poetry
poetry run streamlit run src/app.py

# Using Docker
docker run -p 8501:8501 pdf-references-extractor
  1. 1.Open http://localhost:8501 in your browser
  2. 2.Upload a PDF file
  3. 3.View extracted references directly in the browser
  4. 4.Download the formatted references as a text file

Testing and Validation

The tool includes a testing framework to validate extraction accuracy:

  1. 1.Place test PDFs in test_files/input_files/
  2. 2.Add corresponding ground truth files in test_files/test_files/
  3. 3.Run the validation:
bash
poetry run python src/test_extraction.py

The test framework will:

  • —Process the test PDFs
  • —Compare extracted references with ground truth
  • —Calculate accuracy metrics (precision, recall, F1 score)
  • —Display detailed results for each file

Technical Details

  • —Uses PyPDF2 for PDF text extraction
  • —Employs regex and heuristic-based reference detection
  • —Implements fuzzy matching for deduplication (90% similarity threshold)
  • —Supports various reference formats and styles
  • —Processes references in 500-character windows for optimal performance
  • —Focuses on the last 40% of the document for reference detection