CoolFace
Apppublic

CultriX/Generate-Knowledge-Graphs

sourceHugging Facemitupdated 1y agoView on Hugging Face
3likes
App README

🕸️ Knowledge Graph Extraction App

A complete knowledge graph extraction application using LLMs via OpenRouter, available in both Gradio and Streamlit versions.

🚀 Features

  • —Multi-format Document Support: PDF, TXT, DOCX, JSON files up to 10MB
  • —LLM-powered Extraction: Uses OpenRouter API with free models (Gemma-2-9B, Llama-3.1-8B)
  • —Smart Entity Detection: Automatically identifies people, organizations, locations, concepts, events, and objects
  • —Importance Scoring: LLM evaluates entity importance from 0.0 to 1.0
  • —Interactive Visualization: Multiple graph layout algorithms with filtering options
  • —Batch Processing: Optional processing of multiple documents together
  • —Export Capabilities: JSON, GraphML, and GEXF formats
  • —Real-time Statistics: Graph metrics and centrality analysis

📁 Project Structure

knowledge-graphs/
├── app.py                    # Main Gradio application (legacy)
├── app_streamlit.py          # Main Streamlit application (recommended)
├── run_streamlit.py          # Simple launcher script
├── requirements.txt          # Python dependencies
├── README.md                # Project documentation
├── .env.example             # Environment variables template
├── config/
│   └── settings.py          # Configuration management
└── src/
    ├── document_processor.py # Document loading and chunking
    ├── llm_extractor.py      # LLM-based entity extraction
    ├── graph_builder.py      # NetworkX graph construction
    └── visualizer.py         # Graph visualization and export

🔧 Installation & Setup

Option 1: Streamlit Version (Recommended)

The Streamlit version is more stable and has better file handling.

Quick Start:

bash
python run_streamlit.py

Manual Setup:

  1. 1.Install dependencies:
bash
pip install -r requirements.txt
  1. 1.Run the Streamlit app:
bash
streamlit run app_streamlit.py --server.address 0.0.0.0 --server.port 8501

The app will be available at http://localhost:8501

Option 2: Gradio Version (Legacy)

The Gradio version may have some file caching issues but is provided for compatibility.

  1. 1.Install dependencies:
bash
pip install -r requirements.txt
  1. 1.Set up environment variables (optional):
bash
cp .env.example .env
# Edit .env and add your OpenRouter API key
  1. 1.Run the application:
bash
python app.py

The app will be available at http://localhost:7860

HuggingFace Spaces Deployment

For Streamlit deployment:

  1. 1.Create a new Space on HuggingFace Spaces
  2. 2.Choose "Streamlit" as the SDK
  3. 3.Upload app_streamlit.py as app.py (HF Spaces expects this name)
  4. 4.Upload all other project files maintaining directory structure

For Gradio deployment:

  1. 1.Create a new Space with "Gradio" as the SDK
  2. 2.Upload app.py and all other files
  3. 3.Note: May experience file handling issues

🔑 API Configuration

Getting OpenRouter API Key

  1. 1.Visit OpenRouter.ai
  2. 2.Sign up for a free account
  3. 3.Navigate to API Keys section
  4. 4.Generate a new API key
  5. 5.Copy the key and use it in the application

Free Models Used

  • —Primary: google/gemma-2-9b-it:free
  • —Backup: meta-llama/llama-3.1-8b-instruct:free

These models are specifically chosen to minimize API costs while maintaining quality.

📖 Usage Guide

Basic Workflow

  1. 1.Upload Documents:
  2. 2.Select one or more files (PDF, TXT, DOCX, JSON)
  3. 3.Toggle batch mode for multiple document processing
  1. 1.Configure API:
  2. 2.Enter your OpenRouter API key
  3. 3.Key is stored temporarily for the session
  1. 1.Customize Settings:
  2. 2.Choose graph layout algorithm
  3. 3.Toggle label visibility options
  4. 4.Set minimum importance threshold
  5. 5.Select entity types to include
  1. 1.Extract Knowledge Graph:
  2. 2.Click "Extract Knowledge Graph" button
  3. 3.Monitor progress through the status updates
  4. 4.View results in multiple tabs
  1. 1.Explore Results:
  2. 2.Graph Visualization: Interactive graph with colored nodes by entity type
  3. 3.Statistics: Detailed metrics about the graph structure
  4. 4.Entities: Complete list of extracted entities with details
  5. 5.Central Nodes: Most important entities based on centrality measures
  1. 1.Export Data:
  2. 2.Choose export format (JSON, GraphML, GEXF)
  3. 3.Download structured graph data

Advanced Features

Entity Types
  • —PERSON: Individuals mentioned in the text
  • —ORGANIZATION: Companies, institutions, groups
  • —LOCATION: Places, addresses, geographical entities
  • —CONCEPT: Abstract ideas, theories, methodologies
  • —EVENT: Specific occurrences, meetings, incidents
  • —OBJECT: Physical items, products, artifacts
Relationship Types
  • —works_at: Employment relationships
  • —located_in: Geographical associations
  • —part_of: Hierarchical relationships
  • —causes: Causal relationships
  • —related_to: General associations
Filtering Options
  • —Importance Threshold: Show only entities above specified importance score
  • —Entity Types: Filter by specific entity categories
  • —Layout Algorithms: Spring, circular, shell, Kamada-Kawai, random

🛠️ Technical Details

Architecture Components

  1. 1.Document Processing:
  2. 2.Multi-format file parsing
  3. 3.Intelligent text chunking with overlap
  4. 4.File size validation
  1. 1.LLM Integration:
  2. 2.OpenRouter API integration
  3. 3.Structured prompt engineering
  4. 4.Error handling and fallback models
  1. 1.Graph Processing:
  2. 2.NetworkX-based graph construction
  3. 3.Entity deduplication and standardization
  4. 4.Relationship validation
  1. 1.Visualization:
  2. 2.Matplotlib-based static graphs
  3. 3.Interactive HTML visualizations
  4. 4.Multiple export formats

Configuration Options

All settings can be modified in config/settings.py:

  • —Chunk Size: Default 2000 characters
  • —Chunk Overlap: Default 200 characters
  • —Max File Size: Default 10MB
  • —Max Entities: Default 100 per extraction
  • —Max Relationships: Default 200 per extraction
  • —Importance Threshold: Default 0.3

Differences Between Versions

Streamlit Version Advantages:

  • —More reliable file handling
  • —Better progress indicators
  • —Cleaner UI with sidebar configuration
  • —More stable caching system
  • —Built-in download functionality

Gradio Version Advantages:

  • —Simpler deployment to HF Spaces
  • —More compact interface
  • —Familiar for ML practitioners

🔒 Security & Privacy

  • —API keys are not stored permanently
  • —Files are processed temporarily and discarded
  • —No data is retained between sessions
  • —All processing happens server-side

🐛 Troubleshooting

Common Issues

  1. 1."OpenRouter API key is required":
  2. 2.Ensure you've entered a valid API key
  3. 3.Check the key has sufficient credits
  1. 1."No entities extracted":
  2. 2.Document may be too short or unstructured
  3. 3.Try lowering the importance threshold
  4. 4.Check if the document contains meaningful text
  1. 1.File upload issues (Gradio version):
  2. 2.Known issue with Gradio's file caching system
  3. 3.Try the Streamlit version instead
  4. 4.Ensure files are valid and not corrupted
  1. 1.Segmentation fault (local development):
  2. 2.Usually related to matplotlib backend
  3. 3.Try setting MPLBACKEND=Agg environment variable
  4. 4.Install GUI toolkit if running locally with display
  1. 1.Module import errors:
  2. 2.Ensure all requirements are installed: pip install -r requirements.txt
  3. 3.Check Python version compatibility (3.8+)

Performance Tips

  • —Use batch mode for related documents
  • —Adjust chunk size for very long documents
  • —Lower importance threshold for sparse documents
  • —Use simpler layout algorithms for large graphs

🤝 Contributing

  1. 1.Fork the repository
  2. 2.Create a feature branch
  3. 3.Make your changes
  4. 4.Test with both Streamlit and Gradio versions if applicable
  5. 5.Add tests if applicable
  6. 6.Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments