Saptak225/multiturn_technical_interviewer

Multi-turn Technical Interviewer

An OpenEnv environment that simulates a realistic technical interview. The agent receives a coding or system-design problem, submits a solution, then navigates 5–7 structured follow-up questions covering time/space complexity, edge cases, concurrency, and distributed systems design.

Each answer is graded with a deterministic keyword-matching rubric, providing dense per-turn reward signals throughout the episode. A bad early answer compounds — the environment is designed to stress multi-turn reasoning under natural difficulty escalation across five tasks of increasing difficulty.

Environment Overview

Action Space

class MultiturnTechnicalInterviewerAction(Action):
    response: str   # The agent's complete answer to the current question

The agent's response should be a free-form text string containing reasoning, code, Big-O analysis, trade-off discussion, etc. — whatever best answers the interviewer's current question.

Observation Space

class MultiturnTechnicalInterviewerObservation(Observation):
    question: str                    # Interviewer's current question
    turn: int                        # Current turn (0 = problem statement)
    max_turns: int                   # Total graded turns in this episode
    task_name: str                   # "two_sum" | "lru_cache" | "median_stream" | "rate_limiter" | "message_queue"
    task_difficulty: str             # "easy" | "medium" | "hard"
    task_display_name: str           # Human-readable task name
    conversation_history: List[str]  # Full conversation so far
    turn_score: Optional[float]      # Score for the most recent answer [0,1]
    hint: str                        # Interviewer feedback on last answer
    done: bool                       # True when episode is complete
    reward: Optional[float]          # Reward for the most recent step

Tasks

Task 1 — Two Sum (Easy, 5 turns)

Problem: Given nums and target, return indices of two numbers that sum to target. Solve and discuss the design.

Grading keywords: hash map, O(n), complement, space trade-off, edge cases, partitioned, distributed, coordinator, network overhead.

Task 2 — LRU Cache (Medium, 6 turns)

Problem: Design a Least Recently Used cache with O(1) get and put.

Grading keywords: doubly linked list, hash map, O(1), mutex/lock, consistent hashing, Redis, write-through, TTL, eventual consistency.

Task 3 — Median of a Data Stream (Medium, 6 turns)

Problem: Design a data structure supporting addNum(num) and findMedian() on a continuous stream of integers. findMedian() must be as fast as possible.

Grading keywords: max heap, min heap, O(log n), O(1) findMedian, rebalance, edge cases, reservoir sampling / t-digest / histogram (for scale), lazy deletion.

Task 4 — Distributed Rate Limiter (Hard, 7 turns)

Problem: Design a rate limiter enforcing 100 req/user/min at scale (50+ servers).

Grading keywords: token bucket, sliding window, Redis, Lua script, atomic, centralized store, fail open, circuit breaker, OAuth/API key.

Task 5 — Distributed Message Queue (Hard, 7 turns)

Problem: Design a distributed message queue (Kafka-style) supporting millions of producers/consumers, durable storage, and per-partition ordering.

Grading keywords: partition, append-only log, write-ahead log, consumer group, offset, ISR, leader election, exactly-once, idempotent producer, consumer lag, retention policy.

Reward Function

Each turn produces a reward in [0.0, 1.0] computed as:

score = required_matched/total_required * 0.75
      + bonus_matched/total_bonus       * 0.25
      * length_factor                         (penalty for <30-char answers)
      + 0.04 if len(response) > 300 chars

Score bands: | Score | Meaning | |-------|---------| | 0.0 | Empty / dismissive response | | 0.1–0.3 | Thin or off-topic | | 0.3–0.6 | Partial credit — some relevant concepts | | 0.6–0.8 | Solid answer, most key terms covered | | 0.8–1.0 | Excellent — required + bonus topics addressed |

Episode score = mean reward across all turns. Success = episode score ≥ 0.40.

Episode Flow

reset()              → Observation(question=<problem>, turn=0, done=False, reward=0.0)
step(response_1)     → Observation(question=<follow_up_1>, turn=1, reward=r1)
step(response_2)     → Observation(question=<follow_up_2>, turn=2, reward=r2)
…
step(response_N)     → Observation(question=<closing>, turn=N, done=True, reward=rN)

The environment auto-cycles through tasks on consecutive reset() calls: two_sum → lru_cache → median_stream → rate_limiter → message_queue → two_sum → …

Set INTERVIEW_TASK=median_stream (or any task name) to pin the starting task.

Quick Start

Running locally

# Install dependencies
pip install openenv-core uvicorn fastapi

# Start the server
uvicorn server.app:app --reload --host 0.0.0.0 --port 8000

Running with Docker

# Build the image
docker build -t multiturn_technical_interviewer-env:latest .

# Run the server
docker run -p 8000:8000 multiturn_technical_interviewer-env:latest

Running inference

Hugging Face Router (default `inference.py`):

export HF_TOKEN=hf_...
export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
export API_BASE_URL=https://router.huggingface.co/v1

# Against a running server:
export BASE_URL=http://localhost:8000
python inference.py

# Against a Docker image:
export IMAGE_NAME=multiturn_technical_interviewer-env:latest
python inference.py

After a successful run, episode metrics are written to `outputs/baseline_scores.json` (override directory with `OUTPUT_DIR`). The same applies to inference_nvidea.py and inference_gemini.py.

NVIDIA NIM API (`inference_nvidea.py`) — Nemotron 3 Super:

export NVIDIA_API_KEY=nvapi_...
# Optional: NEMOTRON_THINKING=off|low|full  TEMPERATURE TOP_P MAX_TOKENS
export BASE_URL=http://localhost:8000
python inference_nvidea.py

Defaults: API_BASE_URL=https://integrate.api.nvidia.com/v1, MODEL_NAME=nvidia/nemotron-3-super-120b-a12b.

Expected output (five tasks, one block each):

[START] task=two_sum env=multiturn_technical_interviewer model=Qwen/Qwen2.5-72B-Instruct
[STEP] step=1 action='...' reward=0.72 done=false error=null
[STEP] step=2 action='...' reward=0.68 done=false error=null
...
[END] success=true steps=5 score=0.693 rewards=0.72,0.68,0.71,0.65,0.66

[START] task=lru_cache env=multiturn_technical_interviewer model=Qwen/Qwen2.5-72B-Instruct
...

[START] task=median_stream env=multiturn_technical_interviewer model=Qwen/Qwen2.5-72B-Instruct
...

[START] task=rate_limiter env=multiturn_technical_interviewer model=Qwen/Qwen2.5-72B-Instruct
...

[START] task=message_queue env=multiturn_technical_interviewer model=Qwen/Qwen2.5-72B-Instruct
...

Project Structure

multiturn_technical_interviewer/
├── __init__.py            # Module exports
├── README.md              # This file
├── openenv.yaml           # OpenEnv manifest with task definitions
├── pyproject.toml         # Project metadata and dependencies
├── Dockerfile             # Container image definition
├── validate-submission.sh # Pre-submission validation script
├── client.py              # MultiturnTechnicalInterviewerEnv HTTP/WS client
├── models.py              # Action and Observation Pydantic models
├── inference.py           # Baseline inference (HF Router / OpenAI-compatible)
├── inference_nvidea.py    # NVIDIA NIM — Nemotron 3 Super
├── inference_gemini.py    # Google Gemini
├── baseline_scores_output.py  # Writes outputs/baseline_scores.json after inference
└── server/
    ├── __init__.py
    ├── app.py             # FastAPI application
    └── multiturn_technical_interviewer_environment.py  # Core environment logic

Baseline Scores

Episode score is the mean per-turn reward (see Reward Function). Success is episode score ≥ 0.40.

NVIDIA Nemotron 3 Super 120B-A12B (NIM API)

Run: inference_nvidea.py with model nvidia/nemotron-3-super-120b-a12b, endpoint https://integrate.api.nvidia.com/v1, against a local server (BASE_URL=http://localhost:8000). One full pass over all five tasks.

Aggregate: mean episode score over five tasks ≈ 0.835 (all tasks succeeded).

Sync LLM calls run in asyncio.to_thread so the WebSocket client can handle keepalive pings during long completions; episode metrics are also saved to outputs/baseline_scores.json (see Quick Start).

Reference — Qwen2.5-72B-Instruct (Hugging Face Inference Router)

Approximate baseline with inference.py:

Scores reflect the keyword-graded rubric. Frontier models often land in the 0.75–0.90 range on episode means. The hard tasks (rate_limiter, message_queue) stress multi-layered distributed-systems answers.

OpenEnv API Endpoints