Datasetpublic

MIT-WAL/ai-jobs-news-articles-abstracts

News articles and research abstracts on AI, labor, and jobs Dataset summary This file is a standalone CSV of news articles (full scraped text) and scholarly paper abstracts curated for research on artificial intelligence, work, and labor markets. Each row is one document: a stable id, publication date, normalized title and main text, and a small metadata dictionary. Rows: 53,526 document_class Rows Approx. date range (date column) news 29,857 Jan. 2025… See the full description on the dataset page: https://huggingface.co/datasets/MIT-WAL/ai-jobs-news-articles-abstracts.

sourceHugging Facemitupdated 2mo agoView on Hugging Face

1likes99downloads

Dataset Card

News articles and research abstracts on AI, labor, and jobs

Dataset summary

This file is a standalone CSV of news articles (full scraped text) and scholarly paper abstracts curated for research on artificial intelligence, work, and labor markets. Each row is one document: a stable id, publication date, normalized title and main text, and a small metadata dictionary.

Rows: 53,526

`document_class`	Rows	Approx. date range (`date` column)
`news`	29,857	Jan. 2025 → Jul. 2026
`paper`	23,669	Jan. 2020 → Sep. 2026

Use it for retrieval, RAG, classification, topic modeling, or qualitative sampling—without depending on any other repository layout or auxiliary files.

Changes in the 2026-07-20 release

This snapshot replaces the previous (April 2026) version of text_db.csv:

—The news outlet allowlist was revised. 12 outlets from the previous snapshot were removed and 22 were added, for a new total of 42 outlets (previously 32). The revision drops several regional and aggregator sources in favor of major English-language and technology-focused outlets.
—News coverage extends through July 2026 (previously March 2026).
—The paper collection grew from 19,063 to 23,669 abstracts.

<details> <summary>Outlets removed in this release (12 — expand)</summary>

—abc.net.au
—businesstoday.in
—chicagotribune.com
—chinadaily.com.cn
—dailymail.co.uk
—dw.com
—econotimes.com
—indianexpress.com
—indiatimes.com
—livemint.com
—manilatimes.net
—webpronews.com

</details>

<details> <summary>Outlets added in this release (22 — expand)</summary>

—arstechnica.com
—computerweekly.com
—forbes.com
—fortune.com
—foxnews.com
—govtech.com
—hbr.org
—independent.co.uk
—nature.com
—newscientist.com
—nypost.com
—reuters.com
—sciencedaily.com
—siliconangle.com
—technologyreview.com
—theconversation.com
—thehill.com
—theregister.com
—theverge.com
—venturebeat.com
—wired.com
—zdnet.com

</details>

<details> <summary>News sources in this snapshot (42 outlets — expand for full list)</summary>

These are the distinct values of metadata["SourceCommonName"] for rows with document_class == "news".

—aljazeera.com
—apnews.com
—arstechnica.com
—bbc.com
—bloomberg.com
—businessinsider.com
—cbsnews.com
—cnbc.com
—cnn.com
—computerweekly.com
—forbes.com
—fortune.com
—foxnews.com
—govtech.com
—hbr.org
—independent.co.uk
—nature.com
—nbcnews.com
—newscientist.com
—newsweek.com
—npr.org
—nypost.com
—nytimes.com
—reuters.com
—sciencedaily.com
—scmp.com
—siliconangle.com
—techcrunch.com
—technologyreview.com
—techradar.com
—theconversation.com
—theglobeandmail.com
—theguardian.com
—thehill.com
—theregister.com
—theverge.com
—time.com
—venturebeat.com
—wired.com
—wsj.com
—yahoo.com
—zdnet.com

</details>

What’s in each row

—News (document_class == "news"): doc_id is the SHA-256 (hex) of the article URL. text is the article body; metadata holds GDELT-derived source and content signals (see below).
—Papers (document_class == "paper"): doc_id is the Semantic Scholar `paperId`. text is the abstract only (not full PDF text). metadata includes bibliographic fields when available.

Column reference

Column	Description
`doc_id`	Stable id: URL hash (news) or `paperId` (papers).
`date`	`YYYY-MM-DD` publication date (or best available).
`document_class`	`"news"` or `"paper"`.
`metadata`	Python `repr` of a `dict` (use `ast.literal_eval` in Python).
`wordcount`	Word count (float; often aligned with GDELT for news).
`title`	Headline (news) or paper title.
`text`	Article body (news) or abstract (papers).

News metadata (GDELT-related keys)

For news rows, metadata may include these keys. They originate from the GDELT Global Knowledge Graph style exports; see the GKG codebook for full detail.

Key	Meaning
`SourceCommonName`	Human-readable outlet (usually the website domain, e.g. `reuters.com`).
`SourceCollectionIdentifier`	Type of document id: `1` means open web and `DocumentIdentifier` is a full URL; other values mean citations, DOIs, etc.
`DocumentIdentifier`	Canonical document id; for web news, the article URL (same URL used to compute `doc_id`).
`V2Locations`	Places mentioned in the article. Raw GDELT encodes geocoded blocks; in this release the string may be normalized (simplified place tokens, not the full raw GKG encoding).
`V2Tone`	Comma-separated numbers (in order): (1) overall tone −100…+100; (2) positive word %; (3) negative word %; (4) polarity (how emotionally charged the text is); (5) activity density; (6) self/group (pronoun) density; (7) GDELT word count for the text they analyzed.
`AllNames`	Proper names GDELT associated with the article (people, orgs, named events, etc.); often simplified to a deduplicated token list in this release, not raw offset-encoded blocks.
`wordcount`	Word count carried in metadata (may mirror the last `V2Tone` field or the `wordcount` column).

Paper metadata keys (when present)

Key	Meaning
`paperId`	Semantic Scholar identifier (matches `doc_id`).
`year`	Publication year.
`authors`	Author string or list as stored at export time.
`citationCount`	Citation count from Semantic Scholar.

Intended uses

—Build retrieval or RAG corpora on AI, automation, and labor themes.
—Train or evaluate models on mixed news + academic abstract text.
—Filter or join on dates, outlets, or tone; trace articles back via DocumentIdentifier.

Limitations

—Not factual ground truth—news reflects publishers; abstracts are summaries only.

File

—Format: CSV, UTF-8, header row.
—File name: text_db.csv

Quick start

Dataset hub id: `MIT-WAL/ai-jobs-news-articles-abstracts`.

With `load_dataset`

python

from datasets import load_dataset
ds = load_dataset("MIT-WAL/ai-jobs-news-articles-abstracts", split="train")

With pandas (`hf://`)

python

import pandas as pd
df = pd.read_csv("hf://datasets/MIT-WAL/ai-jobs-news-articles-abstracts/text_db.csv")

Citation

If you use this dataset, please cite the MIT Work Analytics Laboratory.

License

This dataset is released under the MIT License.

News articles and research abstracts on AI, labor, and jobs

Dataset summary

Changes in the 2026-07-20 release

What’s in each row

Column reference

News metadata (GDELT-related keys)

Paper metadata keys (when present)

Intended uses

Limitations

File

Quick start

With load_dataset

With pandas (hf://)

Citation

License

With `load_dataset`

With pandas (`hf://`)