Datasetpublic

incrediblecrab/crs-research-papers

US Congressional Research Service Products Every product of the Congressional Research Service (CRS), the research service of the United States Congress, that the Congress.gov API lists, active and archived, with full text and metadata: Reports, Posts, Resources, Testimony and Infographics, as the API names them. CRS writes them for Members of Congress; Congress.gov publishes them. Nothing here is edited by hand. The pipeline, its tests and its schedule are in… See the full description on the dataset page: https://huggingface.co/datasets/incrediblecrab/crs-research-papers.

sourceHugging Faceotherupdated 3h agoView on Hugging Face

1likes1.4kdownloads

Dataset Card

US Congressional Research Service Products

Every product of the Congressional Research Service (CRS), the research service of the United States Congress, that the Congress.gov API lists, active and archived, with full text and metadata: Reports, Posts, Resources, Testimony and Infographics, as the API names them. CRS writes them for Members of Congress; Congress.gov publishes them.

Nothing here is edited by hand. The pipeline, its tests and its schedule are in github.com/incrediblecrab/crs-service-products, and this card is rendered from manifest.json in the same commit.

Status

14,145 of 14,145 products (100.0%) as of 2026-09-26T16:14:07Z UTC, when the API last listed 14,145. 14,137 rows have text. 0 products failed 3 times and are retried every 24 hours; their errors are in manifest.json.

Last complete sync: 2026-09-26T16:14:07Z UTC.

Series	Rows	With text	Listed	Partitions complete
R	5,564	5,558	5,564	10 of 10
IF	3,205	3,204	3,205	4 of 4
IN	1,973	1,973	1,973	3 of 3
LSB	1,417	1,417	1,417	2 of 2
RL	1,079	1,078	1,079	5 of 5
RS	488	488	488	3 of 3
numeric	224	224	224	1 of 1
TE	98	98	98	1 of 1
IG	97	97	97	1 of 1

Listed counts the products in partitions a sync has reached; partitions not reached yet are not in the table.

Use

python

from datasets import load_dataset
crs = load_dataset("incrediblecrab/crs-research-papers", split="train")

sql

-- DuckDB, straight from the Hub
SELECT id, title, publish_date FROM 'hf://datasets/incrediblecrab/crs-research-papers/data/*.parquet' WHERE status = 'Active' ORDER BY publish_date DESC LIMIT 10;

Files

—data/{partition}.parquet: one row per product. A partition is the series letters plus the thousands of the number (R49 holds R49000 to R49999); ids numbered by year, like 98-684, are in numeric. Rows are sorted by id.
—manifest.json: per partition, the row count, SHA-256, rows with text and whether it is complete; failed products with their errors; the last listing; the last 20 runs.

Schema

Column	Type	Description
`id`	string	CRS product number, for example R49359, IN12740 or 98-684
`content_type`	string	Reports, Posts, Resources, Testimony or Infographics, as the API names them
`status`	string	Active or Archived
`version`	int32	The API's currentVersion; only the current version is kept
`title`	string	Title
`authors`	list<item: string>	Distinct author names, in the API's order
`topics`	list<item: string>	CRS topic names, in no fixed order (the API's order varies between requests); often empty
`publish_date`	string	Publication date (YYYY-MM-DD)
`updated_at`	string	The listing's updateDate when this row was written. A changed value triggers a re-fetch; one that finds the product unchanged keeps the row and records the new value in `manifest.json` instead, so this can be older than the listing's
`url`	string	The product's page on congress.gov
`summary`	large_string	The API's summary, as plain text
`text`	large_string	Full text of the current version: the PDF's text layer, else the HTML rendition's text; null when neither gave text
`text_source`	string	pdf or html; null when text is null
`text_url`	string	The URL the text was extracted from; for a record that lists only HTML, possibly the PDF at the path the HTML implies
`text_sha256`	string	SHA-256 of the bytes fetched from text_url
`metadata`	large_string	The API's detail record as JSON: formats, related bills and laws, authors as listed
`fetched_at`	string	When the fetch that wrote this row ran (UTC); a later fetch that found the product unchanged is not recorded

How it stays current

A GitHub Actions job is scheduled at 00:00 and 12:00 UTC. It asks the API for its product count and its most recently updated product: one request. When either changed, or the last full listing is a day old, the job lists every product and fetches the ones that are new or whose updateDate changed, removes the ones the API no longer has, and commits the changed partitions with this card. A job that runs out of time while still fetching starts the next one itself.

A product whose updateDate changed while its record and text did not keeps its row, its partition is not rewritten, and manifest.json records the new updateDate, so the product is not fetched again until its updateDate next changes. Congress.gov re-stamps some products about every hour without changing them: on September 25, 2026, 26 of 26 such re-fetches of 13 products returned the same record and the same PDF or HTML file, 3 of them with topics in another order, and each rewrite of their 10 partitions added about 100 MB to this repository's history.

The check reads who wrote last and which partitions are incomplete from crs_products_probe in this card's metadata rather than downloading manifest.json. The Hub counts file downloads, so checks stay out of the download count; syncs, which download files, are in it.

GitHub starts scheduled jobs late, or drops them, when it is busy, and its documentation names the start of every hour, when this job is scheduled, as a busy time. On September 24, 2026, from 06:01 to 15:26 UTC, it started 1 of the 113 jobs an earlier 5-minute schedule asked for. So a new product can take more than 12 hours to appear, and the delay is not fixed. A dropped job loses nothing, because the next one reads whatever changed.

The job writes with Hugging Face Trusted Publishing, so no write token is stored anywhere. Only one writer fetches at a time, because www.congress.gov asks for at most 10 requests a minute in total: the manifest records who wrote last and when, a writer waits while another's record is under 45 minutes old, and every commit names its parent commit, so a second writer's commit is refused instead of merged. manifest.json names each run's writer: github-actions for this job, local for the same pipeline run from a computer.

Known gaps

—Only the current version of each product. Earlier versions are not kept.
—text is the PDF's text layer, so it keeps line breaks, page headers and footers, and has no figures or table structure. When the API lists only an HTML rendition, the PDF is tried at the path the HTML implies before falling back to the HTML.
—www.congress.gov sometimes answers CRS HTML with a Cloudflare bot challenge, which this pipeline does not try to get past. A product whose only rendition was challenged has null text, and products without text are fetched again every 7 days.
—The API's updateDate values end in Z but appear to be US Eastern time. updated_at keeps them as given; they are compared, not converted.
—A partition may lag a live listing by up to 10 products between syncs.

License

CRS products are works of the United States Government and are not subject to copyright in the United States (17 U.S.C. § 105). The notice in the products themselves adds: "However, as a CRS Report may include copyrighted images or material from a third party, you may need to obtain the permission of the copyright holder if you wish to copy or otherwise use copyrighted material."