TheFinAI/hacker-news

Hacker News Dataset

Dataset Summary

This dataset is derived from the official Hacker News data provided via the Hacker News Firebase API. It contains user-generated content including stories, comments, and metadata from the Hacker News platform.

Hacker News is a social news website focusing on computer science, entrepreneurship, and technology. The dataset captures real-world discussions, technical conversations, and community interactions over time.

The data was programmatically collected using the official API endpoints and processed into structured JSONL format suitable for large-scale language modeling and analysis.

Dataset Structure

Each sample in the dataset is stored as a JSON object with the following fields:

• —Source: Dataset name (e.g., "HackerNews")
• —Date: Year extracted from the timestamp
• —Text: Main textual content (e.g., comment or story text)
• —Token_count: Number of tokens computed using tiktoken (cl100k_base)

Example

{
  "Source": "HackerNews",
  "Date": 2021,
  "Text": "This is an example comment from Hacker News.",
  "Token_count": 18
}

Data Collection

Data was collected using the official Hacker News API: https://github.com/HackerNews/API

Data Processing

The following preprocessing steps were applied:

Removed empty or null text entries Converted timestamps to year format Tokenized text using cl100k_base Split into multiple JSONL shards for efficient storage