f0rsworN/agentic-sysadmin

Agentic Sysadmin: The OS-Level AI Benchmark

 ![OpenEnv Compliant]() ![uv]() 

Most AI coding agents are excellent at writing Python scripts or React components. But if you drop them into a broken Linux server with a memory leak, shadowed binaries, and ruined file descriptors, they panic and spam whoami.

Agentic Sysadmin is an adversarial evaluation framework designed to test if Large Language Models (LLMs) can actually troubleshoot low-level Operating System issues, or if they just hallucinate high-level fixes. We don't test if an agent can build a service; we test if it can save one at 3 AM.

The Origin Story (Why this exists)

This project was born out of a 3-month diagnostic nightmare. While setting up Gentoo Linux on a Lenovo laptop, the audio stack was completely silent. Despite having the correct ALSA drivers and USE flags, the system refused to produce sound.

State-of-the-art LLMs were completely useless. They repeatedly suggested high-level, generic fixes like "reinstall PulseAudio" or "use apt-get" (on a Gentoo system!). The actual fix required tracing kernel modules, diagnosing ALSA state binaries, and understanding low-level hardware constraints.

It became clear that current AI agents lack deep, lateral Site Reliability Engineering (SRE) skills. They don't check the physical layer, and they trust user-space abstractions too much. Agentic Sysadmin turns those impossible debug sessions into automated, standardized benchmarks.

Novelty: System Deception & Epistemic Uncertainty

Most benchmarks tell the AI exactly what is broken and provide a perfectly clean environment to fix it. We test Epistemic Uncertainty—whether an AI can realize its own diagnostic tools are lying to it.

Across the Gauntlet, agents must navigate active deception:

• —Shadowed Utilities: In ls_cat_trivia, core binaries are hijacked by malicious wrappers. If an agent tries to verify a fix using a tool it hasn't restored yet, the false output will gaslight it into a debugging death spiral.
• —Libc Hijacking: In 2k_vs_200k, network syscalls are intercepted via LD_PRELOAD. Standard diagnostic tools will fail in ways that look like network outages, forcing the AI to question the integrity of the OS itself.

This benchmark proves whether an agent blindly trusts user-space stdout, or if it can dynamically reason about the physical integrity of its environment using absolute paths and strace.

Core Architecture Challenges

• —Native-Root OpenEnv Execution: We completely eliminated Docker-in-Docker. To comply with OpenEnv's strict spec, we run a fully isolated, native-root Ubuntu sandbox directly on Hugging Face Spaces using Uvicorn, serving the standard step()/reset()/state() API.
• —State-Based Partial Rewards: We don't grade the agent on the commands it types; we grade it on the final state of the machine using partial reward shaping (e.g., +0.1 for fixing a tool, +0.4 for restoring network).
• —Immune Graders: The grading scripts use pure Python and absolute pathing to bypass hijacked $PATH variables, ensuring the agent cannot "assassinate" the judge.
• —"History Brain" Context: Agents are fed their own execution history (Stdout, Stderr, Exit Codes, and CWD), forcing them to pivot their logic rather than repeating failed commands.

Repository Structure

The framework is highly modular, managed by the ultra-fast uv package manager.

• —`pyproject.toml` & `uv.lock`: Modern dependency management ensuring strict OpenEnv compliance.
• —`server/app.py`: The FastAPI/Gradio backend handling the OpenEnv API spec.
• —`inference.py`: The main execution loop utilizing the official OpenAI client and an exponential backoff 429-retry loop.
• —`/tasks`: The Gauntlet. Each subfolder contains:
• —setup.py: The instructions to poison the live Linux state.
• —grader.py: The immune evaluation script.
• —task_brief.txt: The initial mission given to the agent.

Spoiler Warning: The /tasks directory contains a task_explanations.txt file. This acts as a master key, fully breaking down the intended human solution and logic for every trap. Read it only if you want to see how the magic trick is done!

The Gauntlet (Task Registry)

We evaluate agents across 6 strict scenarios. There is no partial credit for simply running strace. The system state must be healed.

Baseline Scoreboard (Zero-Shot)

Evaluated using a 30-step maximum limit with Temperature = 0.0. Models were tested on their ability to autonomously navigate the terminal and fix the system. The updated graders use an exclusive scoring range, so untouched tasks now begin near 0.50 and fully solved tasks land near 0.99.

Observation: Weaker models can still solve surface-level issues, but they tend to stall on the harder tasks. Under the updated graders, failures no longer collapse to 0.0; instead, they remain near the neutral baseline until the model makes real progress. The hardest tasks still expose brittle reasoning, especially when the model hallucinates accounts, misreads diagnostic output, or fixes the wrong absolute path.

Quickstart & Usage

1. Installation (Using uv)

git clone https://huggingface.co/spaces/f0rsworN/agentic-sysadmin
cd agentic-sysadmin
uv venv
source .venv/bin/activate
uv sync

2. Testing a Single Task (run_task.py)

To quickly test an agent against a specific task without launching the full server, run the standalone runner. This is the best way to debug or validate a specific scenario:

python run_task.py <folder_name_of_task>

Example: python run_task.py 2k_vs_200k

3. Running the Full Evaluation (inference.py)

To run the complete benchmark evaluation (useful for reproducing baseline scores across all tasks):

python inference.py

Note: This requires an `OPENAI_API_KEY` set in your `.env` file.

4. Starting the OpenEnv Server

The server acts as an OpenEnv spec-compliant endpoint. It doesn't run testing itself but exposes the sandbox to external SRE agent runners:

uvicorn server.app:app --host 0.0.0.0 --port 8000

5. Video Tutorial

For a complete step-by-step walkthrough of the project, including how tasks work under the hood, you can view or download the tutorial video here: https://drive.google.com/file/d/1IQbm7iBbbdvCD_F5PEqzEUhRcJn56cGA/view?usp=sharing