CoolFace
Apppublic

rshashank11/outlook

sourceHugging Faceupdated 3mo agoView on Hugging Face
0likes
App README

NewsGPT

A FastAPI-based hybrid RAG chatbot for searching and answering questions from indexed news archives.

![Python](https://www.python.org/) ![FastAPI](https://fastapi.tiangolo.com/) ![LangGraph](https://github.com/langchain-ai/langgraph) ![License: MIT](LICENSE)

NewsGPT answers questions grounded exclusively in retrieved news content — it never draws on model memory. It supports two sources out of the box:

SourceLanguageIndex
SakalMarathiPinecone
Bar & BenchEnglishPinecone + PostgreSQL

Table of Contents

How It Works

User question
     │
     ▼
 Planner — rewrites the question into a source-aware search query
     │
     ▼
 Retriever — dense embedding + BM25 sparse vectors → Pinecone hybrid search
     │
     ▼
 Context judge — checks whether retrieved chunks are sufficient
     │
     ▼
 Answer — generates a grounded response with inline citations
  1. 1.The planner turns the user's question into a source-aware search query.
  2. 2.The retriever embeds the query for semantic (dense) search.
  3. 3.The BM25 encoder converts the query into sparse keyword values.
  4. 4.Pinecone runs hybrid search combining dense and sparse vectors.
  5. 5.The context judge decides whether retrieved chunks are sufficient.
  6. 6.The answer step writes a source-grounded response with citations.

Architecture

[image]

The editable source diagram is at docs/newsgpt-architecture.drawio.

Prerequisites

  • —Python 3.11+
  • —`uv` (recommended) or pip
  • —Docker & Docker Compose (for containerised runs and ingestion)
  • —A Pinecone account with indexes created for each source
  • —An OpenAI API key or an Azure OpenAI deployment

Quick Start

1. Clone and install

bash
git clone https://github.com/your-org/news-rag.git
cd news-rag
uv sync

2. Configure environment

bash
cp .env.example .env
# Edit .env and fill in the required values (see Configuration below)

3. Start the API

bash
uv run uvicorn main:app --reload --host 0.0.0.0 --port 8000
EndpointURL
API roothttp://localhost:8000
Interactive docshttp://localhost:8000/docs
Health checkhttp://localhost:8000/health

Docker Compose

bash
docker compose up app

Configuration

Copy .env.example to .env and fill in real values. Never commit `.env`.

Required

VariableDescription
POSTGRESQL_URLPostgreSQL connection string
PINECONE_API_KEYPinecone API key
PINECONE_INDEX_HOSTDefault Pinecone index host
DEFAULT_NEWS_SOURCEsakal or barandbench
OPENAI_API_KEYOpenAI API key (or use Azure settings below)

Source-specific

VariableDescription
BARANDBENCH_PINECONE_INDEX_HOSTBar & Bench index host
BARANDBENCH_PINECONE_NAMESPACEBar & Bench namespace
BARANDBENCH_BM25_ENCODER_PATHPath to BM25 values JSON (default: bm25_barandbench_values.json)
SAKAL_PINECONE_INDEX_HOSTSakal index host
SAKAL_PINECONE_NAMESPACESakal namespace
SAKAL_BM25_ENCODER_PATHPath to BM25 values JSON (default: bm25_sakal_values.json)

OpenAI

VariableDescription
OPENAI_CHAT_MODELChat completion model (e.g. gpt-4o-mini)
OPENAI_EMBEDDING_MODELEmbedding model (e.g. text-embedding-3-small)
OPENAI_CONTEXT_JUDGE_MODELModel used for context sufficiency check
OPENAI_CONTENT_KIND_MODELOptional model for Bar & Bench article-type classification

Azure OpenAI (alternative to direct OpenAI)

VariableDescription
AZURE_OPENAI_API_KEYAzure OpenAI key
AZURE_OPENAI_ENDPOINTAzure endpoint URL
AZURE_OPENAI_API_VERSIONAPI version
AZURE_OPENAI_EMBEDDING_DEPLOYMENTEmbedding deployment name
AZURE_OPENAI_CHAT_DEPLOYMENTChat deployment name
AZURE_OPENAI_PLANNER_DEPLOYMENTPlanner deployment name
AZURE_OPENAI_CONTEXT_JUDGE_DEPLOYMENTContext judge deployment name
AZURE_OPENAI_CONTENT_KIND_DEPLOYMENTArticle-type classification deployment name

Retrieval (optional)

VariableDefaultDescription
HYBRID_ALPHA0.5Dense-vs-sparse balance (0 = sparse only, 1 = dense only)
RETRIEVAL_TOP_K10Number of chunks to retrieve
RERANK_MODEnonenone (Pinecone scores) or jina (Jina reranker)
JINA_API_KEY—Required when RERANK_MODE=jina
MAX_CONTEXT_JUDGE_CHARS_PER_SOURCE1500Max chars sent to the context judge per source

Ingestion

Sakal

bash
# 1. Convert XML dumps to JSON
python convert_sakal_xml_to_json.py

# 2. Chunk, embed, and upload to Pinecone
docker compose --profile ingest run ingest-sakal

Bar & Bench

bash
# Ingest story dumps into Postgres + Pinecone
docker compose --profile ingest run ingest-barandbench

# Backfill published_at metadata for existing Pinecone chunks (if needed)
python backfill_barandbench_pinecone_published_at.py

Tests

bash
uv run python -m unittest discover -s tests

Project Structure

news-rag/
├── app/
│   ├── agents/          # LangGraph agent definitions
│   ├── config.py        # Settings (pydantic-settings)
│   ├── embeddings.py    # Embedding helpers
│   ├── retrieval.py     # Hybrid retrieval logic
│   ├── sparse.py        # BM25 sparse encoder
│   └── vectorstore.py   # Pinecone client wrapper
├── docs/                # Architecture diagrams
├── tests/               # Unit tests
├── main.py              # FastAPI application entry point
├── ingest_sakal.py      # Sakal ingestion script
├── ingest_barandbench.py# Bar & Bench ingestion script
├── .env.example         # Environment variable template
├── docker-compose.yml
└── pyproject.toml