CoolFace
Apppublic

saranga2000/gaia-agent-for-evaluation

sourceHugging Faceupdated 1y agoView on Hugging Face
0likes
App README

🤖 GAIA Agent - Advanced Q&A Chatbot

🌟 Introduction

GAIA Agent is a sophisticated AI-powered chatbot system designed to handle complex questions and tasks through an intuitive Q&A interface. Built on top of the GAIA benchmark framework, this agent combines advanced reasoning, code execution, web search, document processing, and multimodal understanding capabilities. The system features both a user-friendly chatbot interface and a comprehensive evaluation runner for benchmark testing.

🚀 Key Features

  • —🔍 Multi-Modal Search: Web search, Wikipedia, and arXiv paper search
  • —💻 Code Execution: Support for Python, Bash, SQL, C, and Java
  • —🖼️ Image Processing: Analysis, transformation, OCR, and generation
  • —📄 Document Processing: PDF, CSV, Excel, and text file analysis
  • —📁 File Upload Support: Handle multiple file types with drag-and-drop
  • —🧮 Mathematical Operations: Complete set of mathematical tools
  • —💬 Conversational Interface: Natural chat-based interaction
  • —📊 Evaluation System: Automated benchmark testing and submission

🏗️ Project Structure

gaia-agent/
├── app.py                    # Main Q&A chatbot interface
├── evaluation_app.py         # GAIA benchmark evaluation runner
├── agent.py                  # Core agent implementation with tools
├── code_interpreter.py       # Multi-language code execution
├── image_processing.py       # Image processing utilities
├── system_prompt.txt         # System prompt for the agent
├── requirements.txt          # Python dependencies
├── metadata.jsonl           # GAIA benchmark metadata
├── explore_metadata.ipynb   # Data exploration notebook
└── README.md               # This file

🛠️ Tool Categories

🌐 Browser & Search Tools

  • —Wikipedia Search: Search Wikipedia with up to 2 results
  • —Web Search: Tavily-powered web search with up to 3 results
  • —arXiv Search: Academic paper search with up to 3 results

💻 Code Interpreter Tools

  • —Multi-Language Execution: Python, Bash, SQL, C, Java support
  • —Plot Generation: Matplotlib visualization support
  • —DataFrame Analysis: Pandas data processing
  • —Error Handling: Comprehensive error reporting

🧮 Mathematical Tools

  • —Basic Operations: Add, subtract, multiply, divide
  • —Advanced Functions: Modulus, power, square root
  • —Complex Numbers: Support for complex number operations

📄 Document Processing Tools

  • —File Operations: Save, read, and download files
  • —CSV Analysis: Pandas-based data analysis
  • —Excel Processing: Excel file analysis and processing
  • —OCR: Extract text from images using Tesseract

🖼️ Image Processing & Generation Tools

  • —Image Analysis: Size, color, and property analysis
  • —Transformations: Resize, rotate, crop, flip, adjust brightness/contrast
  • —Drawing Tools: Add shapes, text, and annotations
  • —Image Generation: Create gradients, noise patterns, and simple graphics
  • —Image Combination: Stack and combine multiple images

🎯 How to Use

Q&A Chatbot Interface (app.py)

  1. 1.Start the Chatbot:
bash
   python app.py
  1. 1.Access the Interface:
  2. 2.Open http://localhost:7860 in your browser
  3. 3.Upload files (images, documents, CSV, etc.) if needed
  4. 4.Ask questions in natural language
  5. 5.Get comprehensive answers with tool usage
  1. 1.Supported Interactions:
  2. 2.Text Questions: "What is the capital of France?"
  3. 3.Math Problems: "Calculate the square root of 144"
  4. 4.Code Requests: "Write a Python function to sort a list"
  5. 5.Image Analysis: Upload an image and ask "What do you see?"
  6. 6.Data Analysis: Upload a CSV and ask "What are the trends?"
  7. 7.Web Search: "What are the latest AI developments?"

Evaluation Runner (evaluation_app.py)

  1. 1.Run the Evaluation:
bash
   python evaluation_app.py
  1. 1.Benchmark Testing:
  2. 2.Log in with your Hugging Face account
  3. 3.Click "Run Evaluation & Submit All Answers"
  4. 4.Monitor progress as the agent processes GAIA benchmark questions
  5. 5.View results and scores automatically

🔧 Technical Architecture

LangGraph State Machine

START → Retriever → Assistant → Tools → Assistant
                     ↑              ↓
                     └──────────────┘
  1. 1.Retriever Node: Searches vector database for similar questions
  2. 2.Assistant Node: LLM processes question with available tools
  3. 3.Tools Node: Executes selected tools (web search, code, etc.)
  4. 4.Conditional Routing: Dynamically routes between assistant and tools

Vector Database Integration

  • —Supabase Vector Store: Stores GAIA benchmark Q&A pairs
  • —Semantic Search: Finds similar questions for context
  • —HuggingFace Embeddings: sentence-transformers/all-mpnet-base-v2

Multi-Modal File Support

  • —Images: JPG, PNG, GIF, BMP, WebP
  • —Documents: PDF, DOC, DOCX, TXT, MD
  • —Data: CSV, Excel, JSON
  • —Code: Python, Bash, SQL, C, Java

⚙️ Installation & Setup

1. Clone Repository

bash
git clone https://github.com/fisherman611/gaia-agent.git
cd gaia-agent

2. Install Dependencies

bash
pip install -r requirements.txt

3. Environment Variables

Create a .env file with your API keys:

env
SUPABASE_URL=your_supabase_url
SUPABASE_SERVICE_ROLE_KEY=your_supabase_key
GROQ_API_KEY=your_groq_api_key
TAVILY_API_KEY=your_tavily_api_key
HUGGINGFACEHUB_API_TOKEN=your_hf_token
LANGSMITH_API_KEY=your_langsmith_key

LANGSMITH_TRACING=true
LANGSMITH_PROJECT=ai_agent_course
LANGSMITH_ENDPOINT=https://api.smith.langchain.com

4. Database Setup (Supabase)

Execute this SQL in your Supabase database:

sql
-- Enable pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create match function for documents2 table
CREATE OR REPLACE FUNCTION public.match_documents_2(
  query_embedding vector(768)
)
RETURNS TABLE(
  id         bigint,
  content    text,
  metadata   jsonb,
  embedding  vector(768),
  similarity double precision
)
LANGUAGE sql STABLE
AS $$
  SELECT
    id,
    content,
    metadata,
    embedding,
    1 - (embedding <=> query_embedding) AS similarity
  FROM public.documents2
  ORDER BY embedding <=> query_embedding
  LIMIT 10;
$$;

-- Grant permissions
GRANT EXECUTE ON FUNCTION public.match_documents_2(vector) TO anon, authenticated;

🚀 Running the Application

Chatbot Interface

bash
python app.py

Access at: http://localhost:7860

Evaluation Runner

bash
python evaluation_app.py

Access at: http://localhost:7860

Live Demo

Try it online: Hugging Face Space

🔗 Resources

🤝 Contributing

Contributions are welcome! Areas for improvement:

  • —New Tools: Add specialized tools for specific domains
  • —UI Enhancements: Improve the chatbot interface
  • —Performance: Optimize response times and accuracy
  • —Documentation: Expand examples and use cases

📄 License

This project is licensed under the MIT License.