CultriX/Generate-Knowledge-Graphs

1---2license: mit3title: Generate Knowledge Graphs4sdk: streamlit5emoji: 📉6colorFrom: indigo7colorTo: pink8short_description: Use LLM to generate a knowledge graph from your input data.9---10# 🕸️ Knowledge Graph Extraction App11 12A complete knowledge graph extraction application using LLMs via OpenRouter, available in both Gradio and Streamlit versions.13 14## 🚀 Features15 16- **Multi-format Document Support**: PDF, TXT, DOCX, JSON files up to 10MB17- **LLM-powered Extraction**: Uses OpenRouter API with free models (Gemma-2-9B, Llama-3.1-8B)18- **Smart Entity Detection**: Automatically identifies people, organizations, locations, concepts, events, and objects19- **Importance Scoring**: LLM evaluates entity importance from 0.0 to 1.020- **Interactive Visualization**: Multiple graph layout algorithms with filtering options21- **Batch Processing**: Optional processing of multiple documents together22- **Export Capabilities**: JSON, GraphML, and GEXF formats23- **Real-time Statistics**: Graph metrics and centrality analysis24 25## 📁 Project Structure26 27```28knowledge-graphs/29├── app.py                    # Main Gradio application (legacy)30├── app_streamlit.py          # Main Streamlit application (recommended)31├── run_streamlit.py          # Simple launcher script32├── requirements.txt          # Python dependencies33├── README.md                # Project documentation34├── .env.example             # Environment variables template35├── config/36│   └── settings.py          # Configuration management37└── src/38    ├── document_processor.py # Document loading and chunking39    ├── llm_extractor.py      # LLM-based entity extraction40    ├── graph_builder.py      # NetworkX graph construction41    └── visualizer.py         # Graph visualization and export42```43 44## 🔧 Installation & Setup45 46### Option 1: Streamlit Version (Recommended)47 48The Streamlit version is more stable and has better file handling.49 50**Quick Start:**51```bash52python run_streamlit.py53```54 55**Manual Setup:**561. **Install dependencies**:57```bash58pip install -r requirements.txt59```60 612. **Run the Streamlit app**:62```bash63streamlit run app_streamlit.py --server.address 0.0.0.0 --server.port 850164```65 66The app will be available at `http://localhost:8501`67 68### Option 2: Gradio Version (Legacy)69 70The Gradio version may have some file caching issues but is provided for compatibility.71 721. **Install dependencies**:73```bash74pip install -r requirements.txt75```76 772. **Set up environment variables** (optional):78```bash79cp .env.example .env80# Edit .env and add your OpenRouter API key81```82 833. **Run the application**:84```bash85python app.py86```87 88The app will be available at `http://localhost:7860`89 90### HuggingFace Spaces Deployment91 92For **Streamlit deployment**:931. Create a new Space on [HuggingFace Spaces](https://huggingface.co/spaces)942. Choose "Streamlit" as the SDK953. Upload `app_streamlit.py` as `app.py` (HF Spaces expects this name)964. Upload all other project files maintaining directory structure97 98For **Gradio deployment**:991. Create a new Space with "Gradio" as the SDK1002. Upload `app.py` and all other files1013. Note: May experience file handling issues102 103## 🔑 API Configuration104 105### Getting OpenRouter API Key106 1071. Visit [OpenRouter.ai](https://openrouter.ai)1082. Sign up for a free account1093. Navigate to API Keys section1104. Generate a new API key1115. Copy the key and use it in the application112 113### Free Models Used114 115- **Primary**: `google/gemma-2-9b-it:free`116- **Backup**: `meta-llama/llama-3.1-8b-instruct:free`117 118These models are specifically chosen to minimize API costs while maintaining quality.119 120## 📖 Usage Guide121 122### Basic Workflow123 1241. **Upload Documents**: 125   - Select one or more files (PDF, TXT, DOCX, JSON)126   - Toggle batch mode for multiple document processing127 1282. **Configure API**:129   - Enter your OpenRouter API key130   - Key is stored temporarily for the session131 1323. **Customize Settings**:133   - Choose graph layout algorithm134   - Toggle label visibility options135   - Set minimum importance threshold136   - Select entity types to include137 1384. **Extract Knowledge Graph**:139   - Click "Extract Knowledge Graph" button140   - Monitor progress through the status updates141   - View results in multiple tabs142 1435. **Explore Results**:144   - **Graph Visualization**: Interactive graph with colored nodes by entity type145   - **Statistics**: Detailed metrics about the graph structure146   - **Entities**: Complete list of extracted entities with details147   - **Central Nodes**: Most important entities based on centrality measures148 1496. **Export Data**:150   - Choose export format (JSON, GraphML, GEXF)151   - Download structured graph data152 153### Advanced Features154 155#### Entity Types156- **PERSON**: Individuals mentioned in the text157- **ORGANIZATION**: Companies, institutions, groups158- **LOCATION**: Places, addresses, geographical entities159- **CONCEPT**: Abstract ideas, theories, methodologies160- **EVENT**: Specific occurrences, meetings, incidents161- **OBJECT**: Physical items, products, artifacts162 163#### Relationship Types164- **works_at**: Employment relationships165- **located_in**: Geographical associations166- **part_of**: Hierarchical relationships167- **causes**: Causal relationships168- **related_to**: General associations169 170#### Filtering Options171- **Importance Threshold**: Show only entities above specified importance score172- **Entity Types**: Filter by specific entity categories173- **Layout Algorithms**: Spring, circular, shell, Kamada-Kawai, random174 175## 🛠️ Technical Details176 177### Architecture Components178 1791. **Document Processing**: 180   - Multi-format file parsing181   - Intelligent text chunking with overlap182   - File size validation183 1842. **LLM Integration**:185   - OpenRouter API integration186   - Structured prompt engineering187   - Error handling and fallback models188 1893. **Graph Processing**:190   - NetworkX-based graph construction191   - Entity deduplication and standardization192   - Relationship validation193 1944. **Visualization**:195   - Matplotlib-based static graphs196   - Interactive HTML visualizations197   - Multiple export formats198 199### Configuration Options200 201All settings can be modified in `config/settings.py`:202 203- **Chunk Size**: Default 2000 characters204- **Chunk Overlap**: Default 200 characters  205- **Max File Size**: Default 10MB206- **Max Entities**: Default 100 per extraction207- **Max Relationships**: Default 200 per extraction208- **Importance Threshold**: Default 0.3209 210### Differences Between Versions211 212**Streamlit Version Advantages:**213- More reliable file handling214- Better progress indicators215- Cleaner UI with sidebar configuration216- More stable caching system217- Built-in download functionality218 219**Gradio Version Advantages:**220- Simpler deployment to HF Spaces221- More compact interface222- Familiar for ML practitioners223 224## 🔒 Security & Privacy225 226- API keys are not stored permanently227- Files are processed temporarily and discarded228- No data is retained between sessions229- All processing happens server-side230 231## 🐛 Troubleshooting232 233### Common Issues234 2351. **"OpenRouter API key is required"**:236   - Ensure you've entered a valid API key237   - Check the key has sufficient credits238 2392. **"No entities extracted"**:240   - Document may be too short or unstructured241   - Try lowering the importance threshold242   - Check if the document contains meaningful text243 2443. **File upload issues (Gradio version)**:245   - Known issue with Gradio's file caching system246   - Try the Streamlit version instead247   - Ensure files are valid and not corrupted248 2494. **Segmentation fault (local development)**:250   - Usually related to matplotlib backend251   - Try setting `MPLBACKEND=Agg` environment variable252   - Install GUI toolkit if running locally with display253 2545. **Module import errors**:255   - Ensure all requirements are installed: `pip install -r requirements.txt`256   - Check Python version compatibility (3.8+)257 258### Performance Tips259 260- Use batch mode for related documents261- Adjust chunk size for very long documents262- Lower importance threshold for sparse documents263- Use simpler layout algorithms for large graphs264 265## 🤝 Contributing266 2671. Fork the repository2682. Create a feature branch2693. Make your changes2704. Test with both Streamlit and Gradio versions if applicable2715. Add tests if applicable2726. Submit a pull request273 274## 📄 License275 276This project is licensed under the MIT License - see the LICENSE file for details.277 278## 🙏 Acknowledgments279 280- [OpenRouter](https://openrouter.ai) for LLM API access281- [Streamlit](https://streamlit.io) for the modern web interface framework282- [Gradio](https://gradio.app) for the ML-focused web interface283- [NetworkX](https://networkx.org) for graph processing284- [HuggingFace Spaces](https://huggingface.co/spaces) for hosting