zzzt19/gaia-baseline-agent

GAIA Agent Harness — Final Project

This is a working baseline for the Hugging Face Agents Course Final Assignment. It keeps the course's framework-free agent-loop and registry approach while adding the pieces needed by the official 20-question GAIA Level-1 evaluation.

What is implemented

• —basic OpenAI-compatible Chat Completions tool loop with bounded turns
• —transport-level LLM request retries (reliability only, not an agent recovery policy)
• —real web search, page reading, and YouTube caption retrieval
• —guarded calculation-oriented Python execution
• —task-scoped attachment access for text, Python, CSV, PDF, DOCX, and XLSX
• —image input and OpenAI audio transcription
• —official /questions, /random-question, /files/{task_id}, and /submit flows
• —per-task JSON traces containing latency, token counts, tool calls, and errors
• —incremental answer caching, so an interrupted 20-question run still leaves evidence
• —separate run and submit buttons to prevent accidental leaderboard submissions

The official grader compares answers using exact match, so the system prompt requires answer-only output and normalize_answer() removes common wrapper text.

Baseline boundary

This project intentionally separates five layers:

• —Agent core: basic loop, tool calling, and Tool Registry
• —External capabilities: web/page search, Python, YouTube, and attachment readers
• —GAIA adapter: questions, files, exact-match cleanup, and submission
• —Evaluation infrastructure: traces, latency, token counts, and saved answers
• —Deployment: Gradio, Hugging Face OAuth, and the local CLI

The baseline intentionally has no memory compression, explicit planning, reflection/critic loop, parallel tool execution, or multi-agent orchestration. Tool errors are returned as ordinary observations, but there is no retry policy that changes the agent's execution strategy. Network-level request retries and maximum-turn limits are operational safeguards, not advanced reasoning mechanisms.

Local setup

From this directory:

py -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
Copy-Item .env.example .env

Set OPENAI_API_KEY in your shell (the Python code deliberately does not auto-load secret files). To use the GAIA attachment fallback, first accept the dataset terms for gaia-benchmark/GAIA, then also set HF_TOKEN:

$env:OPENAI_API_KEY = "your-key"
$env:HF_TOKEN = "your-read-token"
python cli.py random

Other useful commands:

python cli.py ask "What is 17 * 23? Return only the number."
python cli.py all
python app.py
pytest -q

Artifacts are written to .runs/:

.runs/
  answers.json
  <task-id>/
    files/
    trace.json

Deploy to Hugging Face Spaces

1. 1.Duplicate the official Final Assignment Space or create a public Gradio Space.
2. 2.Copy the contents of this directory to the Space repository root.
3. 3.In Settings → Variables and secrets, add OPENAI_API_KEY as a secret. Optionally add OPENAI_MODEL and OPENAI_BASE_URL.
4. 4.Wait for the Space to build. Log in through the UI.
5. 5.Run one random question first, inspect the answer and .runs/<task-id>/trace.json, then run all 20.
6. 6.Only click Submit Cached Answers after reviewing the outputs.

The leaderboard expects a public https://huggingface.co/spaces/<user>/<space>/tree/main code URL. The Space runtime supplies SPACE_ID, which the app uses to build it.

Current scoring-service caveat

During local verification on 2026-08-30, the official /questions and /random-question endpoints worked, but /files/{task_id} returned HTTP 404 for all five questions that advertised attachments. On a 404, the client now uses an authenticated fallback to the official gated gaia-benchmark/GAIA dataset. Accept that dataset's terms and provide HF_TOKEN; without a token, the question still runs without the unavailable attachment.

Security boundary

run_python rejects dangerous imports, common dynamic-execution builtins, dunder access, long programs, and runs in an isolated child process with a timeout. This is a useful educational guardrail, not a mathematically secure Python sandbox. Keep the Space scoped to the public GAIA benchmark and do not expose arbitrary anonymous prompts without stronger container isolation.

fetch_url also rejects private, loopback, and link-local destinations to reduce SSRF risk.

Suggested experiment sequence

Keep the 20 questions fixed and record each version in source control:

1. 1.v0: current baseline
2. 2.v1: memory / context compression
3. 3.v2: explicit planning before tool use
4. 4.v3: reflection / answer critic
5. 5.v4: safe parallel search/fetch calls
6. 6.v5: specialized researcher / file analyst orchestration

Compare score, latency, input/output tokens, tool-call count, and failure categories from the saved traces. Do not optimize on hidden answers; improve the general mechanism and rerun the same harness.