nellaivijay/llm-research-daily

Research Collector Dataset

This dataset contains research results aggregated from multiple sources by the Research-Collector tool. Each item is enriched with comprehensive metadata, ML subfield classifications, quality scores, and temporal features.

Dataset Details

• —Topic: large language models OR LLM OR language models
• —Time Range: 2026-04-12T16:58:40.412069 to 2026-04-26T16:58:40.412076
• —Sources: pubmed, crossref, semantic_scholar, paperswithcode, arxiv, medium, kaggle, stackoverflow, github, reddit, hackernews, gdelt
• —Total Items: 41
• —Exported At: 2026-04-26T16:59:04.042590

Dataset Structure

Core Fields

• —id: Unique identifier
• —title: Title of the research item
• —source: Source platform (e.g., pubmed, arxiv, github, reddit, stackoverflow)
• —url: URL to original content
• —author: Author(s)
• —published_date: Publication date (ISO 8601 format)
• —citations: Number of citations (if available)
• —upvotes: Number of upvotes (if available)
• —downloads: Number of downloads (if available)
• —comments: Number of comments (if available)
• —content: Content/abstract/description
• —score: Relevance score

Enriched Metadata Fields

• —metadata_year: Publication year
• —metadata_month: Publication month
• —metadata_day: Publication day
• —metadata_week: Week of year
• —metadata_quarter: Quarter of year
• —metadata_days_since: Days since publication
• —metadata_ml_subfields: ML subfield classifications (JSON array)
• —metadata_subfield_count: Number of ML subfields
• —metadata_keywords: Extracted keywords (JSON array)
• —metadata_keyword_count: Number of keywords
• —metadata_quality_scores: Quality score metrics (JSON dict)
• —metadata_content_type: Content type (paper, preprint, repository, discussion, qa, news)
• —metadata_has_code: Whether item contains code
• —metadata_has_doi: Whether item has DOI
• —metadata_sentiment_polarity: Sentiment polarity score (-1 to 1)
• —metadata_sentiment_subjectivity: Sentiment subjectivity score (0 to 1)
• —metadata_sentiment_category: Sentiment category (positive, negative, neutral)
• —metadata_summary: Automatic summary of content (extractive)
• —metadata_summary_length: Length of summary in characters
• —metadata_data_quality: Data quality metrics (JSON dict)
• —completeness_score: Field completeness percentage (0-100)
• —consistency_score: Internal consistency score (0-100)
• —validity_score: Data validity score (0-100)
• —overall_quality_score: Overall data quality score (0-100)
• —metadata_trending_score: Engagement velocity score
• —metadata_trending_category: Trending category (hot, warm, cool, cold)
• —metadata_engagement_score: Raw engagement score
• —metadata_related_items: Related items with similarity scores (JSON array)
• —metadata_related_count: Number of related items

Source-Specific Metadata

• —PubMed: metadata_journal, metadata_doi, metadata_mesh_terms, metadata_publication_types, metadata_abstract_length
• —arXiv: metadata_arxiv_id, metadata_primary_category, metadata_categories, metadata_journal_ref
• —GitHub: metadata_stars, metadata_forks, metadata_language, metadata_license, metadata_topics, metadata_has_readme
• —Reddit: metadata_subreddit, metadata_link_flair_text, metadata_upvote_ratio, metadata_total_awards, metadata_is_gilded
• —Stack Overflow: metadata_tags, metadata_answer_count, metadata_has_accepted_answer, metadata_view_count, metadata_owner_reputation
• —Semantic Scholar: metadata_citation_count, metadata_influential_citation_count, metadata_fields_of_study, metadata_has_open_access
• —Medium: metadata_author, metadata_publication, metadata_read_time, metadata_claps
• —Kaggle: metadata_votes, metadata_usability_rating, metadata_file_count

Usage Examples

from datasets import load_dataset

# Load dataset
dataset = load_dataset("nellaivijay/llm-research-daily")
train_data = dataset["train"]

# Filter by source
pubmed_items = train_data.filter(lambda x: x["source"] == "pubmed")
github_items = train_data.filter(lambda x: x["source"] == "github")

# Filter by content type
papers = train_data.filter(lambda x: x.get("metadata_content_type") == "paper")
repositories = train_data.filter(lambda x: x.get("metadata_content_type") == "repository")

# Filter by ML subfield
cv_papers = train_data.filter(lambda x: "computer-vision" in x.get("metadata_ml_subfields", []))

# Filter by quality
high_quality = train_data.filter(lambda x: x.get("metadata_quality_scores", {}).get("overall_quality_score", 0) > 0.7)

# Sort by score
sorted_items = train_data.sort("score", reverse=True)

# Filter by date
recent_items = train_data.filter(lambda x: x.get("metadata_days_since", 999) < 30)

# Filter by trending category
trending_items = train_data.filter(lambda x: x.get("metadata_trending_category") == "hot")

# Filter by data quality
high_quality = train_data.filter(lambda x: x.get("metadata_data_quality", {}).get("overall_quality_score", 0) > 0.7)

# Filter by sentiment
positive_items = train_data.filter(lambda x: x.get("metadata_sentiment_category") == "positive")

# Get related items
item_with_related = train_data[0]
related_items = item_with_related.get("metadata_related_items", [])

Data Quality Features

• —Standardized Dates: All dates normalized to ISO 8601 format
• —ML Subfield Classification: Automatic classification into 15+ ML subfields
• —Quality Scoring: Multi-dimensional quality assessment (abstract length, code availability, DOI, engagement, recency)
• —Temporal Features: Year, month, week, quarter, days since publication
• —Keyword Extraction: Automatic extraction of technical keywords
• —Content Type Detection: Automatic classification of item type
• —Sentiment Analysis: Sentiment polarity, subjectivity, and category classification
• —Automatic Summarization: Extractive summaries for quick content overview
• —Data Quality Metrics: Completeness, consistency, and validity scores for each item
• —Trending Metrics: Engagement velocity analysis with trending categories
• —Cross-References: Related item detection based on shared subfields, keywords, and tags
• —Fuzzy Deduplication: Intelligent duplicate detection with metadata merging
• —Metadata Completeness: Fallback logic to infer missing metadata fields

Data Sources

This dataset aggregates research from:

• —Academic: PubMed, arXiv, Semantic Scholar, Crossref, Papers with Code
• —Professional: GitHub, Stack Overflow, Kaggle
• —Social: Reddit, Hacker News
• —News: GDELT
• —Blogs: Medium, Towards Data Science

Limitations

• —Data is limited to the specified time range
• —Some sources may have rate limits or API restrictions
• —Citation counts may vary between sources
• —ML subfield classification is based on keyword matching and may not be perfect

Source

Generated by Research-Collector, an educational multi-source research aggregation tool.

License

MIT License

Citation

If you use this dataset, please cite the repository URL: https://huggingface.co/datasets/nellaivijay/llm-research-daily