aamanlamba/Lineage-graph-accelerator

Lineage Graph Accelerator 🔥

AI-powered data lineage extraction and visualization for modern data platforms

  

🎉 Built for the Gradio Agents & MCP Hackathon - Winter 2025 🎉 Celebrating MCP's 1st Birthday! This project demonstrates the power of MCP integration for enterprise data governance.

🌟 What is Lineage Graph Accelerator?

Lineage Graph Accelerator is an AI-powered tool that helps data teams:

• —Extract data lineage from dbt, Airflow, BigQuery, Snowflake, and more
• —Visualize complex data dependencies with interactive Mermaid diagrams
• —Export lineage to enterprise data catalogs (Collibra, Microsoft Purview, Alation)
• —Integrate with MCP servers for enhanced AI-powered processing

Why Data Lineage Matters

Understanding where your data comes from and where it goes is critical for:

• —Data Quality: Track data transformations and identify issues
• —Compliance: Document data flows for GDPR, CCPA, and other regulations
• —Impact Analysis: Understand downstream effects of schema changes
• —Data Discovery: Help analysts find and trust data assets

🎯 Key Features

Multi-Source Support

Export to Data Catalogs

Visualization Options

• —Mermaid Diagrams: Interactive, client-side rendering
• —Subgraph Grouping: Organize by data layer (raw, staging, marts)
• —Color-Coded Nodes: Distinguish sources, tables, models, reports
• —Edge Labels: Show transformation types

🚀 Quick Start

Try Online (HuggingFace Space)

1. 1.Visit Lineage Graph Accelerator on HuggingFace
2. 2.Click "Load Sample" to load example data
3. 3.Click "Extract Lineage" to see the visualization
4. 4.Explore the Demo Gallery for more examples

Run Locally

# Clone the repository
git clone https://github.com/YOUR_REPO/lineage-graph-accelerator.git
cd lineage-graph-accelerator

# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Run the app
python app.py

Open http://127.0.0.1:7860 in your browser.

📖 Usage Guide

1. Text/File Metadata Tab

Paste your metadata directly:

{
  "nodes": [
    {"id": "source_db", "type": "source", "name": "Source Database"},
    {"id": "staging", "type": "table", "name": "Staging Table"},
    {"id": "analytics", "type": "table", "name": "Analytics Table"}
  ],
  "edges": [
    {"from": "source_db", "to": "staging"},
    {"from": "staging", "to": "analytics"}
  ]
}

2. Sample Data

Load pre-built samples to explore different scenarios:

• —Simple JSON: Basic node/edge lineage
• —dbt Manifest: Full dbt project with 15+ models
• —Airflow DAG: ETL pipeline with 15 tasks
• —Data Warehouse: Snowflake-style multi-layer architecture
• —ETL Pipeline: Complex multi-source pipeline
• —Complex Demo: 50+ node e-commerce platform

3. Export to Data Catalogs

1. 1.Extract lineage from your metadata
2. 2.Expand "Export to Data Catalog"
3. 3.Select format (OpenLineage, Collibra, Purview, Alation)
4. 4.Click "Generate Export"
5. 5.Copy the JSON for import into your catalog

🔌 MCP Integration

Connect to MCP (Model Context Protocol) servers for enhanced processing:

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Lineage Graph  │────▶│   MCP Server    │────▶│   AI Model      │
│   Accelerator   │     │  (HuggingFace)  │     │   (Claude)      │
└─────────────────┘     └─────────────────┘     └─────────────────┘

Configuration

1. 1.Expand "MCP Server Configuration" in the UI
2. 2.Enter your MCP server URL
3. 3.Add API key (if required)
4. 4.Click "Test Connection"

Run Local MCP Server

uvicorn mcp_example.server:app --reload --port 9000

Then use http://localhost:9000/mcp as your server URL.

🏗️ Architecture

flowchart TD
    A[User Interface - Gradio] --> B[Input Parser]
    B --> C{Source Type}
    C -->|dbt| D[dbt Parser]
    C -->|Airflow| E[Airflow Parser]
    C -->|SQL| F[SQL Parser]
    C -->|JSON| G[JSON Parser]
    D & E & F & G --> H[LineageGraph]
    H --> I[Mermaid Generator]
    H --> J[Export Engine]
    I --> K[Visualization]
    J --> L[OpenLineage]
    J --> M[Collibra]
    J --> N[Purview]
    J --> O[Alation]

    subgraph Optional
        P[MCP Server] --> H
    end

Project Structure

lineage-graph-accelerator/
├── app.py                 # Main Gradio application
├── exporters/             # Data catalog exporters
│   ├── __init__.py
│   ├── base.py           # Base classes
│   ├── openlineage.py    # OpenLineage format
│   ├── collibra.py       # Collibra format
│   ├── purview.py        # Microsoft Purview format
│   └── alation.py        # Alation format
├── samples/               # Sample data files
│   ├── sample_metadata.json
│   ├── dbt_manifest_sample.json
│   ├── airflow_dag_sample.json
│   ├── sql_ddl_sample.sql
│   ├── warehouse_lineage_sample.json
│   ├── etl_pipeline_sample.json
│   └── complex_lineage_demo.json
├── mcp_example/          # Example MCP server
│   └── server.py
├── tests/                # Unit tests
│   └── test_app.py
├── memories/             # Agent configuration
├── USER_GUIDE.md         # Comprehensive user guide
├── BUILD_PLAN.md         # Development roadmap
└── requirements.txt

🧪 Testing

# Activate virtual environment
source .venv/bin/activate

# Run unit tests
python -m unittest tests.test_app -v

# Run setup validation
python test_setup.py

📋 Requirements

• —Python 3.9+
• —Gradio 5.49.1+
• —See requirements.txt for full dependencies

🎖️ Competition Submission

Track: Track 2 - MCP in Action (Productivity)

Team Members:

• —Aaman Lamba | HuggingFace | GitHub

Judging Criteria Alignment

Demo Video

📺 YouTube: Watch the Demo 🎥 Loom: Alternative Link

Highlights:

• —AI Assistant with Google Gemini generating lineage from natural language
• —MCP Integration with Local Demo server
• —Demo Gallery with 50+ node complex pipelines
• —Export to Collibra, Purview, and Apache Atlas
• —Interactive Mermaid visualizations with zoom and download

Social Media Post

📱 LinkedIn: View the announcement post

🔜 Roadmap

• —[x] Gradio 6 upgrade for enhanced UI components
• —[x] Agentic chatbot for natural language queries (Google Gemini)
• —[x] Apache Atlas export support
• —[ ] File upload functionality
• —[x] Graph export as PNG/SVG
• —[ ] Batch processing API
• —[ ] Column-level lineage

🤝 Contributing

Contributions welcome! Please:

1. 1.Fork the repository
2. 2.Create a feature branch
3. 3.Make your changes
4. 4.Submit a pull request

See CONTRIBUTING.md for guidelines.

📄 License

MIT License - see LICENSE for details.

🙏 Acknowledgments

• —Anthropic - MCP Protocol and Claude
• —Gradio Team - Amazing UI framework
• —HuggingFace - Hosting and community
• —dbt Labs - Inspiration for metadata standards
• —OpenLineage - Open lineage specification

📞 Support

• —Documentation: USER_GUIDE.md
• —Author Website: aamanlamba.com
• —Issues: GitHub Issues
• —Discussion: HuggingFace Community

<p align="center"> Built with ❤️ by <a href="https://aamanlamba.com"><strong>Aaman Lamba</strong></a> for the <strong>Gradio Agents & MCP Hackathon - Winter 2025</strong> <br> Celebrating MCP's 1st Birthday! 🎂 </p>