Apppublic

okidokiz/digital_doppelganger

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes

App README

Digital Doppelganger

A production-style FastAPI backend that ingests user writing, processes documents asynchronously, stores vectorized user-scoped knowledge in PostgreSQL/pgvector, and serves personalized RAG responses.

Stack: FastAPI, PostgreSQL/pgvector, Redis, Celery, Alembic, JWT auth, RBAC, Pytest, Prometheus, Grafana, Docker

Live API: https://okidokiz-digital-doppelganger.hf.space/

Interactive Docs: https://okidokiz-digital-doppelganger.hf.space/docs

Features

—Ingests User Writing: Accepts notes, documents, and plain text input
—Generates In-Style Responses: Answers questions using retrieved context in a user's tone
—Hybrid Retrieval: Combines semantic search with BM25 keyword matching
—Multi-Format Processing: Supports plain text, PDF, and DOCX ingestion
—User-Scoped Data: Stores and retrieves data per user with authentication and authorization

Quick Start (Docker)

Prerequisites

—Docker
—Docker Compose
—OpenRouter API Account (free tier available)

Setup

Clone and configure the project:

bash

git clone https://github.com/okidokizhhz/digital_doppelganger.git
cd digital_doppelganger
cp .env.example .env
# configure OPENROUTER_API_KEY, SECRET_KEY, and database settings in .env
docker compose up --build

API Documentation

Once running, visit http://localhost:8000/docs for interactive API documentation with full endpoint details and testing capabilities.

Usage Examples

API Endpoints

Swagger Docs:

bash

curl -X GET "http://localhost:8000/docs"

Health check:

bash

curl -X GET "http://localhost:8000/v1/health"

Services (DB, Redis, Celery) readiness check:

bash

curl -X GET "http://localhost:8000/v1/ready"

Register a new user:

bash

curl -X POST "http://localhost:8000/v1/register" \
  -H "Content-Type: application/json" \
  -d '{
  "username": "testuser123",
  "email": "test@example.com",
  "password": "testpassword123"
  }'

Login and get access token (JWT):

bash

curl -X POST "http://localhost:8000/v1/login" \
  -H "Content-Type: application/json" \
  -d '{
  "username": "testuser123", 
  "password": "testpassword123"
  }'

Feed text data (JWT, all permissions, Idempotency):

bash

curl -X POST "http://localhost:8000/v1/feed/text" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $IDEMPOTENCY_KEY" \
  -d '{
  "data_text": "I like eating chocolate ice cream."
  }'

Upload a file (JWT, all permissions, Idempotency):

bash

curl -i -X POST "http://localhost:8000/v1/upload/file" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -F "file=@/home/appuser/my_notes.txt" \
  -H "Idempotency-Key: $IDEMPOTENCY_KEY"

Get the task status for a specific task (JWT, all permissions):

bash

curl -i -X GET "http://localhost:8000/v1/tasks/$TASK_ID" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \

Ask a question (JWT, all permissions, Idempotency):

bash

curl -i -X POST "http://localhost:8000/v1/ask" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $IDEMPOTENCY_KEY" \
  -d '{
  "question": "Do I like chocolate ice cream?"
  }'

List Sources - data ingested/uploaded (JWT, all permissions):

bash

curl -X GET "http://localhost:8000/v1/sources" \
    -H "Authorization: Bearer $ACCESS_TOKEN"

Delete Source (JWT, all permissions):

bash

curl -X DELETE "http://localhost:8000/v1/sources/delete/$FILE_ID" \
    -H "Authorization: Bearer $ACCESS_TOKEN"

List all users + roles (JWT, admin permission only):

bash

curl -X GET "http://localhost:8000/v1/admin/users" \
    -H "Authorization: Bearer $ACCESS_TOKEN"

Promote user (JWT, admin permission only):

bash

curl -X PATCH "http://localhost:8000/v1/admin/promote/$USER_ID" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $ACCESS_TOKEN"

Demote user (JWT, admin permission only):

bash

curl -X PATCH "http://localhost:8000/v1/admin/demote/$USER_ID" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $ACCESS_TOKEN"

Renew access token (Refresh Token):

bash

curl -X POST "http://localhost:8000/v1/refresh" \
    --cookie "refresh_token=$REFRESH_TOKEN"

Delete/Invalidate refresh tokens for a user (JWT, admin permission only):

bash

curl -X DELETE "http://localhost:8000/v1/admin/refresh_tokens/delete/$USER_ID" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $ACCESS_TOKEN" \
    -d '{
    "confirmation": "revoke refresh tokens"
    }'

Delete/Revoke a user's Idempotency Keys (JWT, admin permission only):

bash

curl -sS -X DELETE "http://localhost:8000/v1/admin/idempotency_keys/delete/$USER_ID" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $ACCESS_TOKEN" \
    -d '{
    "confirmation": "delete idempotency keys"
    }'

Logout (JWT, Refresh Token):

bash

curl -X POST "http://localhost:8000/v1/logout" \
     -H "Authorization: Bearer $ACCESS_TOKEN" \
     --cookie "refresh_token=$REFRESH_TOKEN"

Delete own account with all data (JWT, all permissions):

bash

curl -X DELETE "http://localhost:8000/v1/delete/user" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $ACCESS_TOKEN" \
    -d '{
    "confirmation": "delete my account with all data"
    }'

Technical Architecture

mermaid

flowchart LR
    Client --> FastAPI
    FastAPI --> PostgreSQL
    FastAPI --> Redis
    FastAPI --> Celery
    Celery --> PostgreSQL
    FastAPI --> OpenRouter

Core Components

—API Layer: FastAPI with JWT authentication and async endpoints
—Search Engine: Hybrid search combining 70% semantic similarity and 30% BM25 keyword matching
—Vector Database: PostgreSQL with pgvector extension for efficient similarity search
—AI Models: SentenceTransformers for embeddings, OpenRouter for LLM responses
—Document Processing: Unstructured.io for multi-format file parsing with Celery

Key Technologies

—Backend: Python 3.11, FastAPI, SQLAlchemy, PostgreSQL 16, Redis, Celery
—AI/ML: Sentence Transformers, OpenRouter API, NLTK, rank-bm25
—Infrastructure: Docker, PgVector, async/await patterns
—Security: JWT tokens, RBAC, bcrypt hashing, user data isolation

Retrieval Notes

—Hybrid retrieval combines semantic similarity with BM25 keyword matching
—Embeddings use all-mpnet-base-v2
—Chunking is sentence-based and configurable
—Async endpoints and connection pooling are used to improve responsiveness

Project Structure

digital_doppelganger/
├── .github/
│   └── workflows/
│       ├── ci-tests.yml
│       ├── code-quality.yml
│       ├── docker-build.yml
│       └── hf-deployment.yml
├── alembic/
│   ├── env.py
│   ├── README
│   ├── script.py.mako
│   └── versions/
│       ├── bb277e2c752b_.py
│       └── 7139b974b7f1_.py
├── app/
│   ├── api/
│   │   ├── auth_dependencies.py
│   │   ├── jwt_blacklisting.py
│   │   ├── models.py
│   │   ├── password_validation.py
│   │   └── rate_limiting.py
│   ├── cache/
│   │   └── refresh_token_caching.py
│   ├── core/
│   │   ├── auth.py
│   │   ├── idempotency.py
│   │   ├── logger.py
│   │   ├── metrics.py
│   │   ├── rbac.py
│   │   └── security.py
│   ├── scripts/
│   │   └── create_demo_admin.py
│   ├── services/
│   │   ├── ask_service.py
│   │   ├── auth_service.py
│   │   ├── bm25.py
│   │   ├── document_processor.py
│   │   ├── embedding.py
│   │   ├── hybrid_search.py
│   │   ├── idempotency_service.py
│   │   ├── llm_integration.py
│   │   ├── response_generation.py
│   │   ├── upload_file_service.py
│   │   └── text_processing.py
│   ├── storage/
│   │   ├── connection.py
│   │   ├── crud.py
│   │   └── schema.py
│   ├── utils/
│   │   ├── dependencies.py
│   │   └── utils.py
│   ├── config.py
│   ├── main.py
│   └── middleware.py
├── logs/
├── tests/
│   ├── api/
│   │   └── test_api_endpoints.py
│   └── unit/
│       ├── test_bcrypt_hashing.py
│       ├── test_bm25.py
│       ├── test_crud.py
│       ├── test_document_processor.py
│       ├── test_embedding.py
│       ├── test_generate_llm_response.py
│       ├── test_password_validation.py
│       ├── test_security_service.py
│       └── test_text_processing.py
├── .dockerignore
├── .env.example
├── .gitignore
├── LICENSE
├── README.md
├── Dockerfile
├── alembic.ini
├── docker-compose.yml
├── prometheus.yml
├── pytest.ini
├── requirements.txt
└── server.py

How It Works

1.Data Ingestion: Text is split into chunks and converted to vector embeddings
2.Storage: Chunks and embeddings stored in PostgreSQL with pgvector
3.Query Processing: Questions are embedded and matched against stored data
4.Hybrid Search: Combines semantic similarity with keyword relevance
5.Response Generation: LLM synthesizes a response using retrieved context

Technical Decisions

—FastAPI for async request handling and automatic OpenAPI docs
—PostgreSQL + pgvector to keep structured data and vector search in one database
—Redis for rate limiting, token blacklisting, and idempotency caching
—Celery for asynchronous document processing
—Hybrid retrieval to combine semantic similarity with keyword matching

Known Limitations

—BM25 keyword retrieval is rebuilt per query and is not yet optimized for larger datasets
—CORS and deployment settings are currently oriented toward demo/local environments
—The project is optimized for portfolio/demo use, not hardened production deployment
—The Idempotency implementation is not safe when it comes to duplicate concurrent requests (both will be processed)

License

This project is open source and available under the MIT License.