Yigit-Karaman/open-jobs-daily

Open Jobs Daily 🌍💼

Commercial vendors often charge upwards of $1,000/month for firehose access to global job market data. This dataset democratizes that access.

The main creator of this dataset is Reddit user OminousLatinWord. For convenience, I converted the dataset to Parquet files and uploaded it to Hugging Face.

Source Data & Attribution

• —Creator: Created and originally open-sourced by Reddit user OminousLatinWord under a CC0 license.
• —Source Release: Announced in the r/opensource subreddit via the post "Vendors charge $1000+/mo for this data. I'm giving it away for free."
• —Repository: The original codebase and data pipeline can be found at https://github.com/elliottdehn/open-jobs.
• —Collection Method: Generated via a daily automated crawl of approximately 65,000 individual company career sites.
• —Scope: Contains a daily snapshot of over 3 million active job postings. The listings cover the global market but statistically skew toward US-based roles.
• —Primary Sources: Major Applicant Tracking Systems (ATS) including Greenhouse, Lever, and Workable.
• —Original Format: The data was initially published as static JSON files, accompanied by tools to download and query specific parts locally into Parquet formats.
• —Data Processing: Job descriptions are extracted as raw text (jd) and processed into 1536-dimensional float32 vector embeddings (vec_b64) to enable immediate semantic search and clustering capabilities.

Dataset Structure

The dataset is provided in highly compressed, sharded .parquet parts, separated into two configurations:

Default (Job Postings)

• —`ats` (string): The source Applicant Tracking System (e.g., greenhouse, lever).
• —`slug` (string): The company's ATS identifier.
• —`id` (string): The unique job posting ID from the source board.
• —`title` (string): Job title.
• —`company` (string): Hiring company name.
• —`location` (string): Raw location string (e.g., "Remote", "New York, NY").
• —`url` (string): Direct link to the active job posting.
• —`seen_ms` (int64): Epoch timestamp when the crawler first saw the job.
• —`jd` (string): The complete, raw text of the job description.
• —`leaf` (int32): Cluster group ID for the dataset manifest.
• —`sim` (float64): Pre-calculated similarity score (if applicable to the fetch context).
• —`vec_b64` (string): A 1536-dimensional float32 vector embedding of the job description, encoded in Base64 (useful for semantic search).
• —`pub_ms` (int64): Epoch timestamp of publication (if available).

Ledger

• —Contains the manifest and cluster tracking data associated with the scraping runs.

Usage

Hugging Face's datasets library automatically handles the sharded parts. Enable streaming=True to process the data without exhausting local disk space.

Load the main dataset:

from datasets import load_dataset
# Stream the main job postings directly from the hub
ds = load_dataset("Yigit-Karaman/open-jobs-daily", split="train", streaming=True)

Load the ledger subset:

# Specify the 'ledger' config name to load the manifest data
ledger_ds = load_dataset("Yigit-Karaman/open-jobs-daily", "ledger", split="train", streaming=True)