bothari01/secops-env
0
1# Comprehensive Hackathon Plan: SecOps Environment
2
3**Created:** March 28, 2026
4**Goal:** Qualify for Round 2 with maximum score
5**Timeline:** 11 days (until April 8)
6**Mode:** Solo, simulation-based (no AWS/external dependencies)
7
8---
9
10## Executive Summary
11
12This plan transforms `secops_env` from a basic 3-task environment into a polished, impressive submission that demonstrates:
13
141. **Realistic tool simulation** - Agents execute simulated commands
152. **Diverse security domains** - 5 tasks covering different SecOps scenarios
163. **Sophisticated grading** - F1-based with partial credit and error handling
174. **Production-ready** - HF Space deployment, Docker, comprehensive tests
18
19---
20
21## Constraints
22
23| Constraint | Implication | Solution |
24|------------|-------------|----------|
25| No AWS account | Can't make real API calls | **Mock command execution** with simulated responses |
26| External APIs unavailable | Can't call real services | **All external deps simulated** |
27| Testing infrastructure | Hackathon may restrict access | **Self-contained Docker** |
28| Solo work | Limited bandwidth | **Prioritize high-impact tasks** |
29
30---
31
32## Phase 0: Preparation (Today - March 28)
33
34### Task 0.1: Environment Setup Checklist
35```
36□ HF Account ready (for Space deployment)
37□ Docker Desktop running
38□ Python 3.10+ available
39□ Git configured
40□ IDE ready (VS Code recommended)
41```
42
43### Task 0.2: Repository Structure Review
44```
45secops_env/
46├── __init__.py
47├── models.py # Pydantic models
48├── client.py # HTTP client
49├── inference.py # Baseline script
50├── openenv.yaml # OpenEnv manifest
51├── pyproject.toml # Dependencies
52├── Dockerfile
53├── README.md
54├── server/
55│ ├── app.py # FastAPI server
56│ ├── secops_environment.py
57│ ├── tasks/
58│ │ ├── pii_redaction.py
59│ │ ├── public_access.py
60│ │ └── ghost_user.py
61│ └── graders/
62│ ├── pii_grader.py
63│ ├── access_grader.py
64│ └── user_grader.py
65└── tests/
66 ├── test_api.py
67 ├── test_graders.py
68 └── test_tasks.py
69```
70
71---
72
73## Phase 1: Core Improvements (Days 1-3)
74
75**Goal:** Establish solid foundation with tool simulation
76
77### Day 1: Tool Simulation Architecture
78
79#### 1.1.1 Create Tool Simulator Module
80**File:** `server/tool_simulator.py` (NEW)
81
82```python
83class ToolSimulator:
84 """Simulates command execution without external dependencies."""
85
86 def __init__(self):
87 self.execution_log = []
88 self.cloud_state = {
89 "s3_buckets": {},
90 "iam_users": {},
91 "ec2_instances": {},
92 "security_groups": {}
93 }
94
95 def execute_aws_command(self, command: str, args: dict) -> dict:
96 """
97 Parse and execute simulated AWS CLI commands.
98 Returns: {"success": bool, "output": str, "error": str | None}
99 """
100 # Parse command type and execute mock logic
101 pass
102
103 def get_state(self, resource_type: str, resource_id: str) -> dict:
104 """Get current state of a resource."""
105 pass
106
107 def update_state(self, resource_type: str, resource_id: str, changes: dict):
108 """Update simulated resource state."""
109 pass
110
111 def simulate_delay(self):
112 """Add realistic delay to command execution."""
113 pass
114
115 def generate_audit_log(self) -> list:
116 """Return log of all executed commands."""
117 pass
118```
119
120#### 1.1.2 Update Existing Tasks with Tool Simulation
121
122**PII Redaction Enhancement:**
123```
124Before: Agent identifies PII → Grader scores → Done
125After:
126 1. Agent analyzes text
127 2. Agent executes: python /tools/redact.py --input <text>
128 3. Simulator runs mock script → Returns redacted output
129 4. Agent verifies output
130 5. Agent finalizes
131 6. Grader scores + execution log checked
132```
133
134**Public Access Enhancement:**
135```
136Before: Agent identifies buckets → Done
137After:
138 1. Agent analyzes buckets
139 2. Agent executes: aws s3api put-public-access-block --bucket <name>
140 3. Simulator updates cloud_state
141 4. Agent verifies: aws s3api get-public-access-block --bucket <name>
142 5. Agent executes for each bucket
143 6. Grader scores based on final state + execution log
144```
145
146**Ghost User Enhancement:**
147```
148Before: Agent identifies users → Done
149After:
150 1. Agent analyzes user accounts
151 2. Agent executes: aws iam update-user --user-name <name> --status disabled
152 3. Simulator marks user as disabled in cloud_state
153 4. Agent verifies with: aws iam get-user --user-name <name>
154 5. Grader scores based on disabled users + execution log
155```
156
157### Day 2: Add 4th Task - Log Analysis
158
159**File:** `server/tasks/log_analysis.py` (NEW)
160
161#### Task Definition
162```
163Task: Security Log Triage
164Difficulty: Medium
165Objective: Analyze firewall/SIEM logs and classify security alerts
166```
167
168#### Scenarios Pool (20+ examples)
169```python
170SCENARIOS = [
171 {
172 "logs": """
1732026-03-28 10:15:23 FIREWALL BLOCK 192.168.1.100 → 8.8.8.8:443 PROTO:TCP
1742026-03-28 10:15:24 FIREWALL BLOCK 192.168.1.100 → 45.33.32.156:22 PROTO:TCP
1752026-03-28 10:15:25 FIREWALL ALLOW 192.168.1.100 → 10.0.0.5:443 PROTO:TCP
176""",
177 "classification": "LATERAL_MOVEMENT",
178 "severity": "HIGH",
179 "reasoning": "Multiple blocked connection attempts to external IPs followed by internal communication"
180 },
181 # ... 19 more scenarios
182]
183```
184
185#### Action Types
186| Action | Description |
187|--------|-------------|
188| ANALYZE | Parse and understand log entries |
189| CLASSIFY | Assign classification (Malware/True Positive/False Positive/Needs Investigation) |
190| PRIORITIZE | Rank alerts by severity |
191| FINALIZE | Submit classification report |
192
193#### Grading Criteria
194```
195Correct classification: +0.4
196Correct severity: +0.2
197Correct reasoning: +0.2
198Partial credit for partial matches
199-0.2 per false positive (misclassifying benign as malicious)
200```
201
202### Day 3: Add 5th Task - Config Hardening
203
204**File:** `server/tasks/config_hardening.py` (NEW)
205
206#### Task Definition
207```
208Task: Security Configuration Review
209Difficulty: Hard
210Objective: Review YAML/JSON configs for security misconfigurations
211```
212
213#### Scenarios Pool (15+ examples)
214```python
215SCENARIOS = [
216 {
217 "config_type": "yaml",
218 "content": """
219apiVersion: v1
220kind: Pod
221metadata:
222 name: myapp
223spec:
224 containers:
225 - name: app
226 image: nginx:latest
227 securityContext:
228 privileged: true
229 runAsUser: 0
230---
231apiVersion: networking.k8s.io/v1
232kind: NetworkPolicy
233metadata:
234 name: allow-all
235spec:
236 podSelector: {}
237 ingress:
238 - {}
239""",
240 "issues": [
241 {"line": 10, "severity": "CRITICAL", "type": "privileged_container"},
242 {"line": 11, "severity": "HIGH", "type": "run_as_root"},
243 {"line": 19, "severity": "HIGH", "type": "allow_all_policy"}
244 ],
245 "fixes": [
246 "Set privileged: false",
247 "Set runAsUser: 1000",
248 "Restrict NetworkPolicy to specific pods"
249 ]
250 },
251 # ... 14 more scenarios
252]
253```
254
255#### Action Types
256| Action | Description |
257|--------|-------------|
258| REVIEW | Analyze configuration file |
259| IDENTIFY_ISSUES | Find security problems |
260| SUGGEST_FIXES | Propose remediation |
261| APPLY_FIXES | Generate hardened config |
262| FINALIZE | Submit review report |
263
264---
265
266## Phase 2: Polish & Tests (Days 4-6)
267
268### Day 4: Comprehensive Testing
269
270#### 4.1.1 Add Tests for New Tasks
271```python
272# tests/test_log_analysis.py
273def test_log_analysis_scenario_generation():
274 task = LogAnalysisTask()
275 scenario = task.generate_scenario()
276 assert "logs" in scenario
277 assert "classification" in scenario
278 assert "severity" in scenario
279
280def test_log_classification():
281 task = LogAnalysisTask()
282 task.generate_scenario()
283 grader = LogGrader()
284
285 action = SecOpsAction(
286 task_type=TaskType.LOG_ANALYSIS,
287 action_type=ActionType.CLASSIFY,
288 reasoning="Malicious traffic pattern detected"
289 )
290
291 reward, feedback, done, success = task.execute_action(action, grader, {})
292 assert 0.0 <= reward <= 1.0
293
294# tests/test_config_hardening.py
295def test_config_hardening_scenario_generation():
296 # Similar structure
297 pass
298
299def test_config_issue_identification():
300 # Test grading logic
301 pass
302```
303
304#### 4.1.2 Add Integration Tests
305```python
306# tests/test_integration.py
307def test_full_pii_redaction_workflow():
308 """Test agent completes full workflow with tool simulation."""
309 # Reset → Analyze → Execute Tool → Verify → Finalize
310 pass
311
312def test_full_ghost_user_workflow():
313 """Test agent disables ghost users with simulated AWS."""
314 # Reset → Analyze → Execute Disable → Verify → Finalize
315 pass
316```
317
318### Day 5: Documentation & README
319
320#### 5.1.1 Update README Structure
321```markdown
322# SecOps Environment
323
324## Quick Start
325## Architecture
326## Tasks
327 ### PII Redaction (Easy)
328 ### Public Access (Medium)
329 ### Ghost User (Hard)
330 ### Log Analysis (Medium)
331 ### Config Hardening (Hard)
332## Tool Simulation
333## API Reference
334## Development
335## Deployment
336```
337
338#### 5.1.2 Add Architecture Diagram
339```
340┌─────────────────────────────────────────────────────────────┐
341│ Client (Agent) │
342└─────────────────────────────────────────────────────────────┘
343 │
344 ▼
345┌─────────────────────────────────────────────────────────────┐
346│ SecOps Environment │
347│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
348│ │ Tasks │ │ Tool │ │ Graders │ │
349│ │ │ │ Simulator │ │ │ │
350│ │ • PII │◄─┤ │──► • PIIGrader │ │
351│ │ • Public │ │ • AWS Mock │ │ • AccessGrader │ │
352│ │ • Ghost │ │ • Shell │ │ • UserGrader │ │
353│ │ • Log │ │ • Audit │ │ • LogGrader │ │
354│ │ • Config │ │ Log │ │ • ConfigGrader │ │
355│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
356└─────────────────────────────────────────────────────────────┘
357```
358
359### Day 6: Error Handling & Edge Cases
360
361#### 6.1.1 Add Error Scenarios
362```python
363# Modify tool simulator to return errors
364ERROR_SCENARIOS = [
365 {
366 "command": "aws s3api put-public-access-block",
367 "error": "AccessDenied",
368 "simulate_error": True
369 },
370 {
371 "command": "aws iam update-user",
372 "error": "NoSuchEntity",
373 "simulate_error": True
374 }
375]
376
377# Tasks should handle errors gracefully
378def execute_action(self, action, grader, task_data):
379 try:
380 result = self.tool_simulator.execute(command)
381 except SimulationError as e:
382 # Partial credit for correct approach
383 return 0.3, f"Command failed: {e}", False, False
384```
385
386---
387
388## Phase 3: Deployment & Validation (Days 7-9)
389
390### Day 7: HF Space Deployment
391
392#### 7.1.1 Create HF Space
3931. Go to https://huggingface.co/new-space
3942. Select Docker template
3953. Name: `secops-env` (or your choice)
3964. Visibility: Public
397
398#### 7.1.2 Configure Space
399```yaml
400# README.md header for Space
401---
402title: SecOps Environment
403emoji: 🔒
404colorFrom: blue
405colorTo: green
406sdk: docker
407app_port: 8000
408---
409```
410
411#### 7.1.3 Push to HF
412```bash
413git init
414git add .
415git commit -m "Initial SecOps Environment submission"
416git remote add origin https://huggingface.co/spaces/<username>/secops-env
417git push -u origin main
418```
419
420### Day 8: Validation & Testing
421
422#### 8.1.1 Pre-Submission Checklist
423```
424□ openenv validate passes
425□ docker build succeeds
426□ docker run works locally
427□ HF Space responds to /health
428□ HF Space responds to /reset
429□ inference.py runs without errors
430□ All pytest tests pass
431□ No hardcoded credentials
432□ README complete
433□ License included (BSD-3-Clause)
434```
435
436#### 8.1.2 Baseline Run
437```bash
438HF_TOKEN="your_token" MODEL_NAME="Qwen/Qwen2.5-7B-Instruct" python inference.py
439```
440
441Expected output:
442```
443======================================================================
444SECOPS ENVIRONMENT BENCHMARK RESULTS
445======================================================================
446Task Difficulty Avg Reward Success Rate Max Reward
447----------------------------------------------------------------------
448pii_redaction easy ~0.70 ~60% 1.000
449public_access medium ~0.80 ~70% 1.000
450ghost_user hard ~0.75 ~60% 1.000
451log_analysis medium ~0.70 ~55% 1.000
452config_hardening hard ~0.65 ~50% 1.000
453----------------------------------------------------------------------
454OVERALL ~0.72 ~59%
455======================================================================
456```
457
458### Day 9: Final Polish
459
460- [ ] Clean up any debug output
461- [ ] Verify README is comprehensive
462- [ ] Check all file headers
463- [ ] Update benchmark_results.json with final scores
464- [ ] Take screenshots/demo video (optional)
465
466---
467
468## Phase 4: Submission & Buffer (Days 10-11)
469
470### Day 10: Submission
4711. Double-check all requirements
4722. Submit on hackathon portal
4733. Save submission confirmation
474
475### Day 11: Buffer
476- Fix any issues discovered
477- Prepare backup submission
478- Rest
479
480---
481
482## Implementation Details
483
484### New Files to Create
485
486| File | Purpose | Complexity |
487|------|---------|------------|
488| `server/tool_simulator.py` | Mock AWS CLI execution | Medium |
489| `server/tasks/log_analysis.py` | Log triage task | Low |
490| `server/tasks/config_hardening.py` | Config review task | Medium |
491| `server/graders/log_grader.py` | Log analysis grader | Low |
492| `server/graders/config_grader.py` | Config hardening grader | Medium |
493| `tests/test_log_analysis.py` | Log task tests | Low |
494| `tests/test_config_hardening.py` | Config task tests | Low |
495| `tests/test_integration.py` | Integration tests | Medium |
496
497### Files to Modify
498
499| File | Changes | Complexity |
500|------|---------|------------|
501| `models.py` | Add TaskType.LOG_ANALYSIS, TaskType.CONFIG_HARDENING | Low |
502| `secops_environment.py` | Register new tasks | Low |
503| `server/app.py` | Update task list | Low |
504| `inference.py` | Add prompts for new tasks | Low |
505| `openenv.yaml` | Update task definitions | Low |
506| `README.md` | Document new tasks | Low |
507| `pyproject.toml` | May need updates | Low |
508
509### Estimated Effort
510
511| Component | Hours | Total |
512|-----------|-------|-------|
513| Tool Simulator | 4-5 | 4-5 |
514| Log Analysis Task | 3-4 | 7-9 |
515| Config Hardening Task | 4-5 | 11-14 |
516| Graders (2) | 2-3 | 13-17 |
517| Tests | 3-4 | 16-21 |
518| Documentation | 2-3 | 18-24 |
519| HF Space Deployment | 2-3 | 20-27 |
520| Buffer/Fixes | 3-4 | 23-31 |
521
522**Total estimated: 23-31 hours**
523
524---
525
526## Success Metrics
527
528| Metric | Target | Stretch Goal |
529|--------|--------|--------------|
530| Tasks implemented | 5 | 5 |
531| Test coverage | 80%+ | 90%+ |
532| Baseline score | 0.70+ | 0.80+ |
533| OpenEnv validate | Pass | Pass with warnings |
534| HF Space uptime | 95%+ | 99%+ |
535| Documentation | Complete | Comprehensive with examples |
536
537---
538
539## Risk Mitigation
540
541| Risk | Likelihood | Mitigation |
542|------|------------|------------|
543| HF Space deployment fails | Medium | Test locally first with Docker |
544| LLM API rate limits | Medium | Use fallback actions gracefully |
545| Time overrun | High | Cut features, not polish |
546| Grader bugs | Low | Comprehensive tests |
547| Environment issues in testing | Low | Self-contained Docker |
548
549---
550
551## Notes
552
5531. **Start early:** Don't wait until day 10 to deploy
5542. **Test incrementally:** Run tests after each feature
5553. **Document as you go:** Saves time later
5564. **Submit early:** Beat the deadline rush
5575. **Keep it simple:** Don't over-engineer
558
559---
560
561*Plan saved for later implementation. Ready to execute when you give the go-ahead.*