abigailhaddad/usajobs-scraping

USAJOBS announcement text

The full text of federal job announcements, scraped from usajobs.gov and joined to the structured fields from the USAJOBS Historical API. About 3.2 million announcements from September 2013 through September 2026, updated daily.

Why this exists

The USAJOBS API is a poor source for announcement text, in two ways.

The Search API only lists jobs that are open right now, so anything that opens and closes between two collection runs is never captured. Measured against the Historical API, which does report closed postings, that was about 3% of 2026 — 5,274 announcements — and it isn't spread evenly. OPM had text for 159 of its 724 postings. SSA for 200 of 422.

For postings the API does return, MatchedObjectDescriptor drops content the announcement page shows. Confirmed cases exist where the page says "Rule of Many" and the API record does not.

Neither gap can be backfilled from the API afterward: the Historical API returns metadata only, and Search only returns open jobs. So this scrapes the page, which usajobs.gov serves indefinitely after an announcement closes.

What's in it

One parquet file per month under data/, one row per announcement, 53 columns. The month files are shard boundaries, not something you have to navigate — the dataset is configured as a single table, so load_dataset gives you all of it. Self contained; you don't need any other source to use it.

Twelve columns come off the announcement page. Eleven are the page's own sections, split on its headings:

jobSummary, majorDuties, requirements, conditionsOfEmployment, qualificationSummary, education, additionalInformation, benefits, howYouWillBeEvaluated, requiredDocuments, howToApply

and text is the whole page as plain text, tags stripped and whitespace collapsed. education is genuinely absent from 10-13% of announcements (measured on months in 2024 and 2026); text and the structured fields are populated on effectively every row.

Everything else comes from the USAJOBS Historical API: usajobsControlNumber, positionTitle, announcementNumber, hiringAgencyCode, hiringAgencyName, hiringDepartmentCode, hiringDepartmentName, hiringSubelementName, agencyLevel, agencyLevelSort, appointmentType, workSchedule, serviceType, whoMayApply, payScale, salaryType, minimumSalary, maximumSalary, minimumGrade, maximumGrade, promotionPotential, supervisoryStatus, totalOpenings, positionOpeningStatus, announcementClosingTypeCode, announcementClosingTypeDescription, positionOpenDate, positionCloseDate, positionExpireDate, travelRequirement, teleworkEligible, relocationExpensesReimbursed, securityClearanceRequired, securityClearance, drugTestRequired, disableApplyOnline, vendor, hiringPaths, jobCategories, positionLocations, and occupationalSeries (the four-digit series pulled out of jobCategories).

hiringPaths, jobCategories, and positionLocations are JSON strings, since one posting can carry several of each.

Every column is a string except agencyLevel (BIGINT) and the two salary columns (DOUBLE). That is enforced at publish time rather than assumed.

manifest.csv lists every usajobsControlNumber in the dataset, so you can check coverage without downloading the parquet files.

Querying it

DuckDB reads the month files over HTTP, so a query costs the columns it touches rather than the whole dataset:

SELECT count(*)
FROM read_parquet('hf://datasets/abigailhaddad/usajobs-scraping/data/2025_*.parquet');

Go a year at a time. Globbing all 152 files at once gets an anonymous reader a 429 partway through.

The dataset viewer's filter and search are a different thing. HuggingFace indexes only a fixed slice of a dataset this size — here, well under 5% — and the page does not say so. Counts read off the viewer are not counts of this dataset. Use DuckDB for anything you plan to quote.

How it's built

The posting list comes from the USAJOBS Historical API, mirrored by usajobs_historical, which reports closed announcements and is therefore the complete universe. Each usajobs.gov/job/{controlNumber} page is fetched, script/style stripped, and the remaining text collapsed to single spaces. The structured fields are joined on usajobsControlNumber from the same Historical API.

A daily job scrapes whatever is in the Historical mirror and not yet here, and re-joins metadata for the months it touches. Close dates and opening status change after a posting first appears, so a monthly job refreshes every month's metadata.

A month file is rewritten wholesale, so the publisher refuses to write a month when the local join is missing announcements the dataset already holds, rather than silently shrinking it.

Caveats

25 announcements in the Historical mirror are absent here and always will be: usajobs.gov returns 503 for 24 of them and 404 for one. That is the entire known gap.

The text is the rendered page, so it includes site chrome — the nav bar, the footer, the cookie language. Anchor your searches accordingly.

A handful of announcements are agency test postings with placeholder content ("There will be duties to perform for this position").

Announcements can be amended after posting. Each page is fetched once, so the text is whatever it said when scraped, not necessarily the final version. The structured fields are refreshed on a schedule, the text isn't.

Coverage before 2017 is thin — the Historical API itself holds very little from those years, so 2013 through 2016 is a few thousand announcements, not a few hundred thousand.

This is not an official USAJOBS project.

License

Work of the U.S. federal government, not subject to copyright.