ruby56/Citation-Benchmark

Citation-Agent: OpenEnv Peer Review Benchmark

Citation-Agent is an OpenEnv-compliant environment for evaluating LLM agents as Academic Peer Reviewers.

Given a Target Paper Abstract, the agent must parse the text, navigate the SQLite research database to traverse its citation graph, and read contextual intents from citing papers. If the citations validate the abstract conceptually, the agent must ACCEPT the paper. If the citations contradict the abstract (indicating a hallucination or decoy task), the agent must REJECT it.

The Challenge

Most agent benchmarks focus on web scraping or bash terminal navigation. This environment tests an agent's ability to reason over complex scholarly graphs. The agent must:

• —Differentiate between ArXiv IDs and Semantic Scholar Corpus IDs.
• —Follow citation intents.
• —Run SQL-backed queries to find core papers.
• —Avoid getting stuck in infinite search loops.

How to Run

Option 1: Running Locally (Recommended)

1. 1.Install Dependencies: Requires Python 3.10+.

   pip install -r requirements.txt

1. 1.Set Environment Variables: We use the Hugging Face Router with the google/gemma-4-31B-it model.
2. 2.PowerShell:

      $env:HF_TOKEN="your_huggingface_api_key"
      $env:MODEL_NAME="google/gemma-4-31B-it"
      $env:API_BASE_URL="https://router.huggingface.co/v1"

• —Bash/Zsh:

      export HF_TOKEN="your_huggingface_api_key"
      export MODEL_NAME="google/gemma-4-31B-it"
      export API_BASE_URL="https://router.huggingface.co/v1"

1. 1.Run the Inference Script:

   python inference.py

This runs the agent through the 50 tasks and outputs the final scores.

Option 2: Running via Docker (Hugging Face Spaces validation)

To test if your environment works with the automated Hugging Face Spaces pipeline:

docker build -t citation-agent .
docker run --env HF_TOKEN="your_token" --env MODEL_NAME="google/gemma-4-31B-it" --env API_BASE_URL="https://router.huggingface.co/v1" citation-agent

OpenEnv Compliance Checklist

1. 1.Real-world task: Accurately simulates literature review and academic source verification.
2. 2.OpenEnv Spec Compliance:
3. 3.Contains a valid openenv.yaml.
4. 4.environment.py uses Pydantic Models (Observation, Action, Reward) and implements the required reset(), state(), and step(action) methods.
5. 5.Observation Space: JSON containing current_claim, search_results, last_abstract, citations_data, message, and step_count.
6. 6.Action Space: action_type (search, read_abstract, get_citations, submit), query, and paper_id.
7. 7.50 Distinct Tasks with Graders:
8. 8.tasks.py loads 50 distinct Easy, Medium, and Hard tasks from the SQLite database.
9. 9.Includes a deterministic Grader class that checks the submitted paper_id against the ground truth.
10. 10.Database Architecture:
11. 11.To stay under the 1GB limit for free HF Spaces, environment.py uses huggingface-hub to automatically pull the citation_db.sqlite database from the ruby56/Citation-Database dataset on startup.
12. 12.Continuous Reward Function:
13. 13.Uses dense shaping. The agent gets a -0.05 step penalty to encourage efficiency, and -0.1 to -0.2 penalties for formatting errors. It gets the final grader score (+1.0 max) on submit.
14. 14.Inference Script Auto-Retry:
15. 15.Uses the OpenAI Python client pointed at HF Serverless endpoints. Implements basic exponential backoff to handle HTTP 429 rate limits safely without crashing the evaluation loop.
16. 16.Production Dockerfile:
17. 17.Standard Dockerfile that installs dependencies and runs inference.py directly.