Careerflow/ResumeExtractBench

ResumeExtractBench

ResumeExtractBench is a benchmark for schema-guided structured extraction from resume documents. Given a resume PDF and a JSON Schema, systems must return structured data covering personal details, work history, education, skills, and more.

• —Dataset Size: 38 documents (handwritten + adversarial distractors)
• —Schema Sections Scored: 9 (basics, experience, education, projects, summary, certifications, awards, volunteering, skills)
• —Domains: 6 (engineering, healthcare, legal, data science, government, general)
• —License: CC-BY-4.0

Quick Links: 💻 Code & CLI | 🏢 Careerflow

Dataset Introduction

Document Composition

Source Composition

• —Handwritten documents: 28
• —Synthetic (LaTeX-generated): 10
• —All documents classified as "hard" difficulty

Challenge Categories

C1: Handwritten Resumes — Scanned handwritten documents with natural variation in handwriting, layout, and legibility. Tests OCR accuracy, spatial reasoning, and robustness to non-standard formatting. Typical failures: misread characters, merged/split words, missed sections, hallucinated content from ambiguous handwriting.

C2: Adversarial Distractors — Synthetic resumes with intentional parsing challenges designed to stress-test extraction robustness:

Domain Coverage

Usage

Loading with Datasets

from huggingface_hub import snapshot_download
import json

root = snapshot_download(
    repo_id="careerflow/ResumeExtractBench",
    repo_type="dataset",
)

with open(f"{root}/test.jsonl") as f:
    cases = [json.loads(line) for line in f if line.strip()]

schema = json.load(open(f"{root}/schema.json"))

for case in cases:
    pdf_path = f"{root}/{case['files']['pdf']}"
    ground_truth = case["ground_truth"]
    # Run your extractor on pdf_path against schema
    # Compare output to ground_truth

Running Evaluation with CLI

pip install git+https://github.com/careerflow/resume-extract-bench.git
resume-bench download                                          # download dataset
resume-bench run --pipeline gpt-5.6 --split test               # run extraction
resume-bench grade --split test                                 # score against GT
resume-bench leaderboard                                        # view results

Bring Your Own Predictions

resume-bench grade-file my_predictions.jsonl --split test

Predictions JSONL supports two formats:

Flat format (recommended):

{"resume_id": "board-certified-ocularist-jane-doe", "basics": {"fname": "Jane", "lname": "Doe", ...}, "experience": [...], ...}

Wrapped format:

{"resume_id": "board-certified-ocularist-jane-doe", "prediction": {"basics": {...}, "experience": [...], ...}}

Dataset Files

• —test.jsonl: 38 test cases (one JSON object per line)
• —schema.json: Target JSON Schema for extraction (resume_v1)
• —pdfs/: Source resume PDFs

Dataset Format

Each JSONL line represents one resume test case:

{
    "resume_id": "distractor-date-chaos-inconsistent-date-formats-throughout",
    "files": {"pdf": "pdfs/distractor-date-chaos-inconsistent-date-formats-throughout.pdf"},
    "ground_truth": { ... },
    "difficulty": "hard",
    "layout_tags": ["distractor", "inconsistent-date-formats"],
    "source": "synthetic-distractor",
    "domain": "software-engineering",
    "schema_version": "resume_v1"
}

Field Definitions

Extraction Schema (resume_v1)

The schema defines 9 sections that must be extracted from each resume:

See schema.json for the full JSON Schema definition.

Evaluation Metrics

Scoring Methods

1. 1.Entity-Level F1: Entities (experience entries, education entries, etc.) are aligned between prediction and ground truth using the Hungarian algorithm (optimal bipartite matching via scipy.optimize.linear_sum_assignment). Match quality is determined by Jaro-Winkler similarity on key fields (threshold: 0.5).

1. 1.Description Token F1: For matched entities with description arrays, a bag-of-words token F1 score measures description quality.

1. 1.Hallucination Rate: Fraction of predicted entities that have no match in ground truth (spurious predictions).

1. 1.Omission Rate: Fraction of ground truth entities that have no match in predictions (missed extractions).

All scoring is deterministic with no model in the loop.

Aggregate Metrics

Per-section scores are averaged across all resumes. The overall F1 is the mean of all section F1 scores, providing a single leaderboard ranking metric.

Leaderboard

Results from the full 155-resume benchmark (includes this dataset plus additional medium-difficulty resumes):

19 models benchmarked. Full results available in the [code repository](https://github.com/careerflow/resume-extract-bench).

Tag Taxonomy

ResumeExtractBench tags documents along three axes:

Layout tags provide additional visual/structural metadata per resume (e.g., handwritten, fancy-templates, canva).

Citation

@misc{careerflow2026resumeextractbench,
    title={ResumeExtractBench: A Benchmark for Schema-Guided Resume Extraction},
    author={Careerflow and LlamaIndex},
    year={2026},
    url={https://huggingface.co/datasets/careerflow/ResumeExtractBench},
}

License

All documents are synthetic or expert-created with fictional personal information. Released under CC-BY-4.0.