shash42/forecast-news-embeddings

Forecast News Embeddings

Precomputed LanceDB table used for keyword, semantic, and hybrid retrieval in forecast-sim and future-sim.

Snapshot

• —7,911,857 indexed source articles
• —16,207,764 text chunks
• —Coverage: 2023-01-11 through 2026-08-31
• —Snapshot published: 2026-09-18
• —Lance dataset version: 856
• —Total artifact size: approximately 303.2 GiB

Articles with empty searchable text are not represented. Long articles can produce multiple chunks, so the chunk count is larger than the article count.

Embeddings and indexes

• —Model: Qwen/Qwen3-Embedding-8B
• —Vector dimension: 4,096
• —Chunk size: 512 tokens
• —Scalar index: complete BTree indexes on date and date_publish
• —Vector index: complete cosine IVF-PQ index on vector
• —Full-text index: complete Tantivy index on content, including phrase positions

The downloaded repository is hybrid-search ready. The materialized Tantivy sidecar is stored under articles.lance/_indices/fts/.

Schema

~~~text chunkid, articleid, chunkindex, title, source, date, datepublish, content, url, vector ~~~

Usage

Inspect the Lance dataset remotely without downloading the full artifact:

~~~python import lance

dataset = lance.dataset( "hf://datasets/shash42/forecast-news-embeddings/articles.lance" ) print(dataset.count_rows()) ~~~

For full-text or hybrid search, download the snapshot so LanceDB can open the Tantivy sidecar:

~~~python from huggingfacehub import snapshotdownload import lancedb

repodir = snapshotdownload( repoid="shash42/forecast-news-embeddings", repotype="dataset", localdir="forecast-news-embeddings", ) db = lancedb.connect(repodir) table = db.open_table("articles")

rows = ( table.search("central bank interest rates", querytype="fts") .where("date <= timestamp '2026-08-31 23:59:59'", prefilter=True) .limit(10) .tolist() ) ~~~

Semantic and hybrid queries must be embedded with the same Qwen3 model and query instruction used by forecast-sim.