bothari01/secops-env

1---2title: SecOps Environment3emoji: 🔐4colorFrom: purple5colorTo: blue6sdk: docker7app_port: 80008pinned: false9license: bsd-3-clause10---11# SecOps Environment - OpenEnv12 13**Security Operations Environment for AI Agent Training and Evaluation**14 15[![OpenEnv Compatible](https://img.shields.io/badge/OpenEnv-0.2.0+-blue.svg)](https://github.com/meta-pytorch/OpenEnv)16[![Python 3.10+](https://img.shields.io/badge/python-3.10+-green.svg)](https://www.python.org/)17 18---19 20## Overview21 22**SecOps Environment** is a real-world security operations environment built on the OpenEnv framework. It simulates common DevOps/SecOps tasks that security analysts perform daily, providing a standardized benchmark for evaluating AI agents in security operations.23 24### Why SecOps?25 26Security operations is a critical domain with:27- **High-stakes decisions**: Misclassified security issues can lead to data breaches28- **Deterministic success states**: Unlike open-ended tasks, security fixes can be verified programmatically29- **Real-world applicability**: Training agents on actual security tasks has immediate practical value30- **Clear evaluation metrics**: Success can be measured precisely (PII removed, buckets fixed, users disabled)31 32---33 34## Tasks35 36### 1. PII Redaction (Easy)37 38**Objective**: Identify and redact Personally Identifiable Information from text.39 40**PII Types Detected**:41- Social Security Numbers (SSN: XXX-XX-XXXX)42- Email addresses (user@domain.com)43- Phone numbers (XXX-XXX-XXXX)44- Credit card numbers (XXXX-XXXX-XXXX-XXXX)45- IP addresses (XXX.XXX.XXX.XXX)46 47**Success Criteria**: All PII replaced with `[REDACTED]`, no false positives48 49**Reward Structure**:50- +0.1 per correctly identified PII51- +0.5 bonus for 100% completion52- -0.2 per false positive53 54---55 56### 2. Fix Public Access (Medium)57 58**Objective**: Identify S3 buckets with public access and fix their permissions.59 60**Task**: Analyze cloud storage resources and identify buckets with overly permissive access policies.61 62**Success Criteria**: All public buckets identified and fixed to block public access63 64**Reward Structure**:65- +0.15 per correctly identified public resource66- +0.4 bonus for fixing all public resources67- -0.1 per missed public resource68- -0.2 per false positive69 70---71 72### 3. Disable Ghost User (Hard)73 74**Objective**: Identify orphaned/inactive user accounts and disable them.75 76**Ghost User Criteria**:77- No login in 90+ days78- No active cloud resources79- No recent deployments80- Created >1 year ago and never active81 82**Success Criteria**: All ghost users correctly identified and disabled, no active users disabled83 84**Reward Structure**:85- +0.1 per correctly identified ghost user86- +0.3 bonus for correctly disabling all87- -0.2 per incorrect disable (active user marked as ghost)88 89---90 91### 4. Log Analysis (Medium)92 93**Objective**: Analyze security logs and classify security events with appropriate severity.94 95**Classifications**:96- `MALWARE`: Confirmed malware activity97- `TRUE_POSITIVE`: Confirmed security threat98- `FALSE_POSITIVE`: Benign event misidentified as threat99- `NEEDS_INVESTIGATION`: Uncertain, requires further analysis100- `LATERAL_MOVEMENT`: Attacker moving through network101- `DATA_EXFILTRATION`: Unauthorized data transfer102- `UNAUTHORIZED_ACCESS`: Access without proper credentials103- `BENIGN`: Normal, safe activity104 105**Severity Levels**: LOW, MEDIUM, HIGH, CRITICAL106 107**Success Criteria**: Correct classification, accurate severity, clear reasoning108 109**Reward Structure**:110- +0.5 for correct classification111- +0.25 for correct severity112- +0.25 for adequate reasoning113 114---115 116### 5. Configuration Hardening (Hard)117 118**Objective**: Review YAML/JSON configurations for security misconfigurations.119 120**Common Issues Detected**:121- Privileged containers (privileged: true)122- Running as root (runAsUser: 0)123- Overly permissive IAM policies124- Plaintext secrets in config125- Public S3 bucket access126- Missing TLS/SSL configuration127- Overly permissive network policies128- Exposed services (LoadBalancer, hostPort)129 130**Success Criteria**: All security issues identified with correct severity, proper fixes applied131 132**Reward Structure**:133- +0.4 for correct issue identification134- +0.3 for appropriate remediation135- +0.3 for correct configuration fixes136 137---138 139## Quick Start140 141### Installation142 143```bash144# Clone the repository145git clone https://github.com/yourusername/secops_env.git146cd secops_env147 148# Install dependencies149pip install -e .150 151# Or with uv152uv pip install -e .153```154 155### Running the Server156 157```bash158# Using the installed script159secops-env-server160 161# Or directly162python -m secops_env.server.app163```164 165### Using the Environment166 167```python168from secops_env import SecOpsEnv, SecOpsAction169from secops_env.models import TaskType, ActionType170 171# Sync usage (recommended for simple scripts)172with SecOpsEnv(base_url="http://localhost:8000").sync() as env:173    # Reset for PII redaction task174    result = env.reset(task="pii_redaction")175    print(f"Objective: {result.observation.objective}")176    print(f"Text to redact: {result.observation.context.get('text')}")177    178    # Execute actions179    action = SecOpsAction(180        task_type=TaskType.PII_REDACTION,181        action_type=ActionType.FINALIZE,182        redacted_text="Customer [REDACTED]..."183    )184    result = env.step(action)185    print(f"Reward: {result.reward}")186    print(f"Done: {result.done}")187 188# Async usage189import asyncio190 191async def main():192    async with SecOpsEnv(base_url="http://localhost:8000") as env:193        result = await env.reset(task="public_access")194        # ... interact with environment195        196asyncio.run(main())197```198 199### Running Baseline Inference200 201```bash202# Set environment variables203export API_BASE_URL="https://router.huggingface.co/v1"204export MODEL_NAME="Qwen/Qwen2.5-7B-Instruct"  # Default model205export HF_TOKEN="your_token_here"206 207# Run baseline evaluation208python inference.py209```210 211---212 213## Environment API214 215### `reset(task=None, difficulty=None, seed=None)`216 217Reset the environment for a new episode.218 219**Parameters**:220- `task` (str, optional): Task type ("pii_redaction", "public_access", "ghost_user")221- `difficulty` (str, optional): Difficulty level ("easy", "medium", "hard")222- `seed` (int, optional): Random seed for reproducibility223 224**Returns**: `StepResult` with initial observation225 226### `step(action)`227 228Execute an action in the environment.229 230**Parameters**:231- `action` (SecOpsAction): Action to execute232 233**Returns**: `StepResult` with observation, reward, and done flag234 235### `state()`236 237Get current environment state.238 239**Returns**: Dictionary with episode metadata240 241---242 243## Action Space244 245```python246class SecOpsAction(BaseModel):247    task_type: TaskType                    # Current task248    action_type: ActionType                # Type of action249    250    # PII Redaction251    redacted_text: Optional[str] = None252    253    # Public Access254    public_resources: Optional[List[str]] = None255    fixed_resources: Optional[List[str]] = None256    257    # Ghost User258    ghost_users: Optional[List[str]] = None259    disabled_users: Optional[List[str]] = None260    261    confidence: Optional[float] = None262    reasoning: Optional[str] = None263```264 265## Observation Space266 267```python268class SecOpsObservation(BaseModel):269    task_type: TaskType270    task_difficulty: TaskDifficulty271    objective: str                         # Clear objective272    273    context: Dict[str, Any]                # Scenario data274    available_actions: List[str]275    current_state: Dict[str, Any]276    277    partial_progress: float                # 0.0-1.0278    step_count: int279    max_steps: int280    281    feedback: Optional[str] = None282    detected_issues: List[str] = []283    fixed_issues: List[str] = []284    285    reward_accumulated: float286    done: bool287    success: bool288```289 290---291 292## Project Structure293 294```295secops_env/296├── __init__.py                 # Package exports297├── models.py                   # Pydantic models298├── client.py                   # EnvClient implementation299├── openenv.yaml               # Environment manifest300├── pyproject.toml             # Dependencies301├── Dockerfile                 # Container build302├── inference.py              # Baseline inference script303├── README.md                  # This file304└── server/305    ├── __init__.py306    ├── app.py                 # FastAPI application307    ├── secops_environment.py  # Core environment logic308    ├── tool_simulator.py     # Mock AWS CLI execution309    ├── tasks/310    │   ├── pii_redaction.py   # Easy task311    │   ├── public_access.py   # Medium task312    │   ├── ghost_user.py      # Hard task313    │   ├── log_analysis.py    # Medium task314    │   └── config_hardening.py # Hard task315    └── graders/316        ├── pii_grader.py317        ├── access_grader.py318        ├── user_grader.py319        ├── log_grader.py320        └── config_grader.py321```322 323---324 325## Baseline Scores326 327Expected baseline performance on Qwen/Qwen2.5-7B-Instruct:328 329| Task | Difficulty | Avg Reward | Success Rate |330|------|------------|-----------|--------------|331| PII Redaction | Easy | ~0.70 | ~60% |332| Fix Public Access | Medium | ~0.80 | ~70% |333| Disable Ghost User | Hard | ~0.75 | ~60% |334| Log Analysis | Medium | ~0.70 | ~55% |335| Config Hardening | Hard | ~0.65 | ~50% |336| **Overall** | - | ~0.72 | ~59% |337 338*Note: Actual scores may vary based on model capabilities and prompting strategies.*339 340---341 342## Docker Deployment343 344### Building the Image345 346```bash347docker build -t secops-env:latest .348```349 350### Running the Container351 352```bash353docker run -p 8000:8000 secops-env:latest354```355 356### Deploying to Hugging Face Spaces357 358```bash359# Install OpenEnv CLI360pip install openenv-cli361 362# Login to Hugging Face363huggingface-cli login364 365# Push to Spaces366openenv push --repo-id yourusername/secops-env367```368 369---370 371## Development372 373### Running Tests374 375```bash376# Install dev dependencies377pip install -e ".[dev]"378 379# Run tests380pytest tests/ -v381```382 383### Testing Individual Tasks384 385```bash386# Test PII Redaction387python -c "388from secops_env.server.tasks.pii_redaction import PIIRedactionTask389from secops_env.server.graders.pii_grader import PIIGrader390 391task = PIIRedactionTask()392data = task.generate_scenario()393print('Scenario:', data)394"395```396 397---398 399## API Reference400 401### Environment Variables402 403| Variable | Description | Default |404|----------|-------------|---------|405| `API_BASE_URL` | LLM API endpoint | `https://router.huggingface.co/v1` |406| `MODEL_NAME` | Model identifier | Required |407| `HF_TOKEN` | API key | Required |408| `MAX_STEPS` | Max steps per episode | `10` |409| `TEMPERATURE` | Model temperature | `0.2` |410 411---412 413## Contributing414 415Contributions are welcome! Please:416 4171. Fork the repository4182. Create a feature branch4193. Add tests for new functionality4204. Submit a pull request421 422---423 424## License425 426BSD 3-Clause License - see LICENSE file for details.427 428---429 430## Acknowledgments431 432- Built on [OpenEnv](https://github.com/meta-pytorch/OpenEnv) framework433- Inspired by real-world security operations workflows434- Developed for AI safety and agent evaluation research435