aamanlamba/Lineage-graph-accelerator

1# Local Setup Guide - Lineage Graph Extractor2 3This guide provides detailed instructions for setting up and running the Lineage Graph Extractor agent locally.4 5## Table of Contents61. [System Requirements](#system-requirements)72. [Installation Methods](#installation-methods)83. [Configuration](#configuration)94. [Usage Scenarios](#usage-scenarios)105. [Advanced Configuration](#advanced-configuration)116. [Troubleshooting](#troubleshooting)12 13## System Requirements14 15### Minimum Requirements16- **OS**: Windows 10+, macOS 10.15+, or Linux17- **Python**: 3.9 or higher18- **Memory**: 2GB RAM minimum19- **Disk Space**: 100MB for agent files20 21### Recommended Requirements22- **Python**: 3.10+23- **Memory**: 4GB RAM24- **Internet**: Stable connection for API calls25 26## Installation Methods27 28### Method 1: Standalone Use (Recommended)29 30This method uses the agent configuration files with any platform that supports the Anthropic API.31 321. **Download the agent**33   ```bash34   # If you have a git repository35   git clone <repository-url>36   cd local_clone37   38   # Or extract from downloaded archive39   unzip lineage-graph-extractor.zip40   cd lineage-graph-extractor41   ```42 432. **Set up environment**44   ```bash45   # Copy environment template46   cp .env.example .env47   ```48 493. **Edit .env file**50   ```bash51   # Edit with your preferred editor52   nano .env53   # or54   vim .env55   # or56   code .env  # VS Code57   ```58 59   Add your credentials:60   ```61   ANTHROPIC_API_KEY=sk-ant-your-key-here62   GOOGLE_CLOUD_PROJECT=your-gcp-project63   GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json64   ```65 664. **Install Python dependencies** (optional, for examples)67   ```bash68   pip install anthropic google-cloud-bigquery requests pyyaml69   ```70 71### Method 2: Claude Desktop Integration72 73If you're using Claude Desktop or similar platforms:74 751. **Locate your agent configuration directory**76   - Claude Desktop: `~/.config/claude/agents/` (Linux/Mac) or `%APPDATA%\claude\agents\` (Windows)77   - Other platforms: Check platform documentation78 792. **Copy the memories folder**80   ```bash81   # Linux/Mac82   cp -r memories ~/.config/claude/agents/lineage-extractor/83   84   # Windows85   xcopy /E /I memories %APPDATA%\claude\agents\lineage-extractor\86   ```87 883. **Configure API credentials** in your platform's settings89 904. **Restart the application**91 92### Method 3: Python Integration93 94To integrate into your own Python application:95 961. **Install dependencies**97   ```bash98   pip install anthropic python-dotenv99   ```100 1012. **Use the integration example**102   ```python103   from anthropic import Anthropic104   from dotenv import load_dotenv105   import os106   107   # Load environment variables108   load_dotenv()109   110   # Initialize client111   client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))112   113   # Load agent configuration114   with open("memories/agent.md", "r") as f:115       system_prompt = f.read()116   117   # Use the agent118   response = client.messages.create(119       model="claude-3-5-sonnet-20241022",120       max_tokens=4000,121       system=system_prompt,122       messages=[{123           "role": "user",124           "content": "Extract lineage from this metadata: ..."125       }]126   )127   128   print(response.content[0].text)129   ```130 131## Configuration132 133### API Keys Setup134 135#### Anthropic API Key1361. Go to https://console.anthropic.com/1372. Create an account or sign in1383. Navigate to API Keys1394. Create a new key1405. Copy to `.env` file141 142#### Google Cloud (for BigQuery)1431. Go to https://console.cloud.google.com/1442. Create a project or select existing1453. Enable BigQuery API1464. Create a service account:147   - Go to IAM & Admin → Service Accounts148   - Create service account149   - Grant "BigQuery Data Viewer" role150   - Create JSON key1515. Download JSON and reference in `.env`152 153#### Tavily (for web search)1541. Go to https://tavily.com/1552. Sign up for an account1563. Get your API key1574. Add to `.env` file158 159### Tool Configuration160 161Edit `memories/tools.json` to customize available tools:162 163```json164{165  "tools": [166    "bigquery_execute_query",      // Query BigQuery167    "read_url_content",             // Fetch from URLs168    "google_sheets_read_range",     // Read Google Sheets169    "tavily_web_search"             // Web search170  ],171  "interrupt_config": {172    "bigquery_execute_query": false,173    "read_url_content": false,174    "google_sheets_read_range": false,175    "tavily_web_search": false176  }177}178```179 180**Available Tools:**181- `bigquery_execute_query`: Execute SQL queries on BigQuery182- `read_url_content`: Fetch content from URLs/APIs183- `google_sheets_read_range`: Read data from Google Sheets184- `tavily_web_search`: Perform web searches185 186### Subagent Configuration187 188Customize subagents by editing their configuration files:189 190**Metadata Parser** (`memories/subagents/metadata_parser/`)191- `agent.md`: Instructions for parsing metadata192- `tools.json`: Tools available to parser193 194**Graph Visualizer** (`memories/subagents/graph_visualizer/`)195- `agent.md`: Instructions for creating visualizations196- `tools.json`: Tools available to visualizer197 198## Usage Scenarios199 200### Scenario 1: BigQuery Lineage Extraction201 202```python203from anthropic import Anthropic204import os205 206client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))207 208with open("memories/agent.md", "r") as f:209    system_prompt = f.read()210 211response = client.messages.create(212    model="claude-3-5-sonnet-20241022",213    max_tokens=4000,214    system=system_prompt,215    messages=[{216        "role": "user",217        "content": "Extract lineage from BigQuery project: my-project, dataset: analytics"218    }]219)220 221print(response.content[0].text)222```223 224### Scenario 2: File-Based Metadata225 226```python227# Read metadata from file228with open("dbt_manifest.json", "r") as f:229    metadata = f.read()230 231response = client.messages.create(232    model="claude-3-5-sonnet-20241022",233    max_tokens=4000,234    system=system_prompt,235    messages=[{236        "role": "user",237        "content": f"Extract lineage from this dbt manifest:\n\n{metadata}"238    }]239)240```241 242### Scenario 3: API Metadata243 244```python245response = client.messages.create(246    model="claude-3-5-sonnet-20241022",247    max_tokens=4000,248    system=system_prompt,249    messages=[{250        "role": "user",251        "content": "Extract lineage from API: https://api.example.com/metadata"252    }]253)254```255 256## Advanced Configuration257 258### Custom Visualization Formats259 260To add custom visualization formats, edit `memories/subagents/graph_visualizer/agent.md`:261 262```markdown263### 4. Custom Format264Generate a custom format with:265- Your specific requirements266- Custom styling rules267- Special formatting needs268```269 270### Adding New Metadata Sources271 272To support new metadata sources:273 2741. Add tool to `memories/tools.json`2752. Update `memories/agent.md` with source-specific instructions2763. Update `memories/subagents/metadata_parser/agent.md` if needed277 278### MCP Integration279 280To integrate with Model Context Protocol servers:281 2821. Check if MCP tools are available: `/tools` directory2832. Add MCP tools to `memories/tools.json`2843. Configure MCP server connection2854. See `memories/mcp_integration.md` (if available)286 287## Troubleshooting288 289### Common Issues290 291#### 1. Authentication Errors292 293**Problem**: API authentication fails294**Solutions**:295- Verify API key is correct in `.env`296- Check key hasn't expired297- Ensure environment variables are loaded298- Try regenerating the API key299 300```bash301# Test Anthropic API key302python -c "from anthropic import Anthropic; import os; from dotenv import load_dotenv; load_dotenv(); client = Anthropic(api_key=os.getenv('ANTHROPIC_API_KEY')); print('✓ API key works')"303```304 305#### 2. BigQuery Access Issues306 307**Problem**: Cannot access BigQuery308**Solutions**:309- Verify service account has BigQuery permissions310- Check project ID is correct311- Ensure JSON key file path is correct312- Test credentials:313 314```bash315# Test BigQuery access316gcloud auth activate-service-account --key-file=/path/to/key.json317bq ls --project_id=your-project-id318```319 320#### 3. Import Errors321 322**Problem**: `ModuleNotFoundError`323**Solutions**:324```bash325# Install missing packages326pip install anthropic google-cloud-bigquery requests pyyaml python-dotenv327 328# Or install all at once329pip install -r requirements.txt  # if you create one330```331 332#### 4. Environment Variables Not Loading333 334**Problem**: `.env` file not being read335**Solutions**:336```python337# Explicitly load .env338from dotenv import load_dotenv339load_dotenv()340 341# Or specify path342load_dotenv(".env")343 344# Verify loading345import os346print(os.getenv("ANTHROPIC_API_KEY"))  # Should not be None347```348 349#### 5. File Path Issues350 351**Problem**: Cannot find `memories/agent.md`352**Solutions**:353```python354# Use absolute path355import os356base_dir = os.path.dirname(os.path.abspath(__file__))357agent_path = os.path.join(base_dir, "memories", "agent.md")358 359# Or change working directory360os.chdir("/path/to/local_clone")361```362 363### Performance Issues364 365#### Slow Response Times366 367**Causes**:368- Large metadata files369- Complex lineage graphs370- Network latency371 372**Solutions**:373- Break large metadata into chunks374- Use filtering to focus on specific entities375- Increase API timeout settings376- Cache frequently used results377 378### Debugging Tips379 3801. **Enable verbose logging**381   ```python382   import logging383   logging.basicConfig(level=logging.DEBUG)384   ```385 3862. **Test each component separately**387   - Test API connection first388   - Test metadata retrieval389   - Test parsing separately390   - Test visualization separately391 3923. **Validate metadata format**393   - Ensure JSON is valid394   - Check for required fields395   - Verify structure matches expected format396 3974. **Check agent configuration**398   - Verify `memories/agent.md` is readable399   - Check `tools.json` syntax400   - Ensure subagent files exist401 402## Getting Help403 404### Documentation405- Agent instructions: `memories/agent.md`406- Subagent docs: `memories/subagents/*/agent.md`407- Anthropic API: https://docs.anthropic.com/408 409### Testing Your Setup410 411Run this complete test:412 413```python414from anthropic import Anthropic415from dotenv import load_dotenv416import os417 418# Load environment419load_dotenv()420 421# Test 1: API Connection422try:423    client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))424    print("✓ Anthropic API connection successful")425except Exception as e:426    print(f"✗ API connection failed: {e}")427    exit(1)428 429# Test 2: Load Agent Config430try:431    with open("memories/agent.md", "r") as f:432        system_prompt = f.read()433    print("✓ Agent configuration loaded")434except Exception as e:435    print(f"✗ Failed to load agent config: {e}")436    exit(1)437 438# Test 3: Simple Query439try:440    response = client.messages.create(441        model="claude-3-5-sonnet-20241022",442        max_tokens=1000,443        system=system_prompt,444        messages=[{445            "role": "user",446            "content": "Hello, what can you help me with?"447        }]448    )449    print("✓ Agent response successful")450    print(f"\nAgent says: {response.content[0].text}")451except Exception as e:452    print(f"✗ Agent query failed: {e}")453    exit(1)454 455print("\n✓ All tests passed! Your setup is ready.")456```457 458Save as `test_setup.py` and run:459```bash460python test_setup.py461```462 463## Next Steps464 4651. ✅ Complete setup4662. ✅ Test with sample metadata4673. 📊 Extract your first lineage4684. 🎨 Customize visualization preferences4695. 🔧 Integrate with your workflow470 471---472 473**Setup complete?** Try the usage examples in README.md or run your own lineage extraction!474 475