mohamedhassan22/WHEC_ERI_experimental

🏥 US Army Medical Research RAG System

A Retrieval-Augmented Generation (RAG) system for querying US Army medical research papers using state-of-the-art AI models - completely free and open-source!

🌟 Features

• —🤖 LLM: Uses OpenAI GPT-3.5 Turbo for intelligent answers
• —🔍 Semantic Search: OpenAI text-embedding-3-small for accurate retrieval
• —📚 Document Support: Upload JSONL or TXT files
• —☁️ Cloud Powered: Leveraging OpenAI API for high performance
• —🔌 REST API: Full FastAPI implementation with automatic docs
• —💾 Persistent Index: Build once, query many times

🚀 Quick Start

Using the API

1. Check System Health

curl https://your-space-name.hf.space/health

2. Upload Documents

curl -X POST "https://your-space-name.hf.space/upload" \
  -F "files=@your_document.jsonl"

3. Build Index

curl -X POST "https://your-space-name.hf.space/build-index"

4. Query the System

curl -X POST "https://your-space-name.hf.space/query" \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What are the main causes of exertional injuries in US Army soldiers?",
    "top_k": 5
  }'

Using Python

import requests

# Base URL
BASE_URL = "https://your-space-name.hf.space"

# Query the RAG system
response = requests.post(
    f"{BASE_URL}/query",
    json={
        "question": "What injury prevention strategies are recommended?",
        "top_k": 5
    }
)

result = response.json()
print(f"Answer: {result['answer']}")
print(f"\nSources: {len(result['sources'])}")

Using JavaScript

const BASE_URL = "https://your-space-name.hf.space";

async function queryRAG(question) {
  const response = await fetch(`${BASE_URL}/query`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      question: question,
      top_k: 5
    })
  });
  
  const data = await response.json();
  return data;
}

// Use it
queryRAG("What are the risk factors for heat illness?")
  .then(result => console.log(result.answer));

📋 API Endpoints

Core Endpoints

Interactive Documentation

Visit /docs for full Swagger UI documentation with interactive testing.

📊 API Request/Response Examples

Query Request

POST /query
{
  "question": "What are the main causes of exertional injuries?",
  "top_k": 5
}

Query Response

{
  "answer": "The main causes of exertional injuries in US Army soldiers include...",
  "sources": [
    {
      "text": "Excerpt from the paper...",
      "score": 0.85,
      "metadata": {
        "title": "Risk factors for musculoskeletal injuries...",
        "journal": "Military Medical Research",
        "year": "2021",
        "authors": "Author1, Author2, Author3",
        "pmcid": "PMC123456"
      }
    }
  ],
  "question": "What are the main causes of exertional injuries?"
}

📁 Document Format

JSONL Format (Recommended)

{"pmcid": "PMC123456", "title": "Paper Title", "text": "Full paper text...", "metadata": {"authors": ["Author1", "Author2"], "journal": "Journal Name", "year": "2021", "doi": "10.1234/example", "keywords": ["keyword1", "keyword2"]}}

Plain Text Format

Simply upload .txt files containing your documents. Metadata will be extracted from filenames.

🛠️ Configuration

The system uses the following default configuration:

EMBEDDING_MODEL = "text-embedding-3-small"
LLM_MODEL = "gpt-3.5-turbo"
CHUNK_SIZE = 1024
CHUNK_OVERLAP = 200
TOP_K = 5
TEMPERATURE = 0.1

🔧 Local Development

Prerequisites

• —Python 3.10+
• —NVIDIA GPU with CUDA support (recommended)
• —16GB+ RAM
• —Docker (for containerized deployment)

Setup

1. 1.Clone the repository

git clone https://huggingface.co/spaces/your-username/your-space-name
cd your-space-name

1. 1.Install dependencies

pip install -r requirements.txt

1. 1.Run the application

uvicorn app:app --host 0.0.0.0 --port 7860

1. 1.Access the API
2. 2.API: http://localhost:7860
3. 3.Docs: http://localhost:7860/docs

Docker Build

docker build -t rag-system .
docker run -p 7860:7860 --gpus all rag-system

📦 Project Structure

.
├── Dockerfile              # Docker configuration
├── requirements.txt        # Python dependencies
├── app.py                 # FastAPI application
├── rag_pipeline.py        # Core RAG logic
├── README.md              # This file
├── data/                  # Document storage (created automatically)
└── rag_index/             # Vector index storage (created automatically)

🧠 How It Works

1. 1.Document Processing: Documents are split into chunks using sentence-based splitting
2. 2.Embedding: Each chunk is embedded using OpenAI's text-embedding-3-small
3. 3.Indexing: Embeddings are stored in a vector index
4. 4.Query: User questions are embedded and similar chunks are retrieved
5. 5.Generation: GPT-3.5 Turbo generates answers based on retrieved context

🎯 Use Cases

• —📖 Research paper analysis
• —🔍 Medical literature search
• —💡 Knowledge extraction from documents
• —🎓 Educational Q&A systems
• —📊 Data-driven insights

⚙️ System Requirements

Minimum

• —CPU: 4 cores
• —RAM: 16GB
• —Storage: 20GB

Recommended

• —GPU: NVIDIA GPU with 8GB+ VRAM
• —CPU: 8+ cores
• —RAM: 32GB+
• —Storage: 50GB+ SSD

🚨 Troubleshooting

Index Not Found

Issue: RAG system returns "not initialized" Solution: Upload documents and build the index using /build-index

API Key Missing

Issue: RAG system errors with authentication failure Solution: Ensure OPENAI_API_KEY is set in your environment variables or Space settings.

Rate Limits

Issue: Queries failing continuously Solution: Check your OpenAI API usage and limits.

📝 Example Questions

• —"What are the main causes of exertional injuries in US Army soldiers?"
• —"How does heat illness affect military readiness?"
• —"What injury prevention strategies are recommended for military training?"
• —"What are the risk factors for musculoskeletal injuries?"
• —"How can training programs reduce injury rates?"

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• —OpenAI for LLM and Embedding models
• —LlamaIndex for the RAG framework
• —Hugging Face for hosting and model infrastructure

📧 Contact

For questions or support, please open an issue on the repository.

Note: This system is designed for research and educational purposes. Always verify critical information with primary sources.