mf-heijoe/final-assignment

GAIA Agent — Agents Course Unit 4

A smolagents CodeAgent running on Claude, answering the 20 GAIA questions from the course scoring API and submitting them for a leaderboard score. Pass mark is 30% (6/20).

This Space hosts the code, it does not run it. Running a Gradio Space requires a payment method on file, so the agent runs locally via run_local.py and submits from there. app.py is the Gradio front end and works as-is if you deploy it to a Gradio Space — see Deploying for the graded run below.

Layout

Why run and submit are separate

The course template does fetch → run → submit behind one button. A full run is slow and any question can fail, so this version caches each answer to answers.json the moment it completes. Re-running skips what's already cached (--force overrides), and submitting reads from the cache. A bug in the submission path costs a retry, not another full run.

Local development

pip install -r requirements.txt
export ANTHROPIC_API_KEY=sk-ant-...        # PowerShell: $env:ANTHROPIC_API_KEY = "sk-ant-..."

python run_local.py --random               # one random question — cheapest loop
python run_local.py --limit 3              # first 3 of the real set
python run_local.py --all                  # full run, cached as it goes
python run_local.py --show                 # dump the cache
python run_local.py --task-id <id> --force # re-run one question after a fix

GAIA_MODEL_ID overrides the model (default anthropic/claude-opus-5); any LiteLLM model string works.

Submitting without a Space

Creating a Space needs a payment method on file, so this repo can also submit directly from a local run:

python run_local.py --all          # fill answers.json
python run_local.py --submit \
    --username <hf-username> \
    --agent-code https://huggingface.co/<user>/<repo>/tree/main

agent_code only needs to be a public link (≥10 chars) to the code you actually ran — the scoring API declares no URL validation on it. A static Space or a plain model repo both work and neither needs compute. Submitting fewer answers than there are questions is blocked unless you pass --yes, so a half-filled cache can't post a bad score by accident.

python app.py also runs locally — the login button and submission are hidden (there's nothing to submit to off-Space), but you can watch a run and inspect the cache. Day-to-day iteration is faster through run_local.py.

Behind a TLS-inspecting corporate proxy, requests rejects the intercepted certificate. truststore (in requirements.txt, imported opportunistically by gaia_api.py) fixes this by using the Windows trust store, which already has the corporate root CA. It's a no-op elsewhere and never disables verification.

Deploying for the graded run

The submission has to come from a public Space — /submit records agent_code as a link to your repo, and the leaderboard verifies it.

1. 1.Duplicate the course template Space, or create a new Gradio Space.
2. 2.Push these files to it.
3. 3.Settings → Variables and secrets → add secret ANTHROPIC_API_KEY.
4. 4.Open the Space, log in with the button, run, then submit.

hf_oauth: true in the front matter above is what makes gr.LoginButton() work — without it there is no username and /submit will refuse.

Known gaps

`/files/{task_id}` is currently broken server-side. All 20 questions are Level 1; 5 declare an attachment (1 png, 2 mp3, 1 py, 1 xlsx) and every one of those returns:

404 {"detail":"No file path associated with task_id <id>."}

So attachments cannot be fetched at all right now, whatever the client does. The runner detects this and tells the agent the file is unavailable instead of letting it hallucinate contents. That caps a realistic ceiling at 15/20 — still well clear of the 6/20 pass mark. Re-check the endpoint before assuming a low score is your agent's fault.

Other gaps, only worth closing if you're chasing a score well above the bar:

• —Audio (.mp3) has no transcription tool wired up — moot while the files endpoint is down.
• —YouTube questions need a transcript tool; web search alone rarely gets them.