ChandraPrakashBathula/test-time-compute

Test-Time Compute: Same Compute, Different Allocation

An interactive laboratory for studying a simple question:

Given the same inference budget, is it better to sample multiple answers or spend the compute revising one answer?

This project compares parallel sampling and sequential revision on 100 real GSM8K questions using real Together AI completions. The experiment uses two models and 4,000 billed API calls. Nothing is simulated, and no experimental number is typed by hand.

NeurIPS 2026 Education Track submission.

Run the App

git clone <this-repo-url> && cd Test-time

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
(cd frontend && npm install)

Start the backend:

.venv/bin/python -m uvicorn backend.app.main:app \
  --host 127.0.0.1 --port 8010

Start the frontend:

cd frontend
BACKEND_URL=http://127.0.0.1:8010 PORT=3001 npm run dev

Open:

http://localhost:3001

The application includes three completed 100-question runs in fixtures/, so the full lesson, charts, and results work without an API key or additional spending.

The browser only communicates with the Next.js frontend. API requests are proxied to FastAPI, so the Together API key is never exposed to the client.

Run Your Own Experiment

cp backend/.env.example backend/.env

Set:

TOGETHER_API_KEY=your-together-api-key-here

Restart the backend.

Without a key, the application still works with the bundled experiments. New runs are simply disabled rather than producing fake results.

What the App Shows

Lesson

The lesson walks through the problem step by step:

1. 1.A question the model gets wrong.
2. 2.The two ways to spend additional inference compute.
3. 3.A prediction before seeing the results.
4. 4.Sample distributions and majority voting.
5. 5.The coverage ceiling.
6. 6.Matched-budget comparison.
7. 7.Adaptive stopping.
8. 8.Budget allocation.
9. 9.Exercises.
10. 10.What the experiments do not establish.

Map

The map shows the progression from one-shot inference to adaptive compute allocation and places the implemented strategies in that larger picture.

Lab

The lab lets you select the number of questions, estimate the cost, launch a run, inspect per-question traces, and compare token-normalized compute.

Reproducing the Results

All reported numbers are generated from the experiment records.

python3 experiments/run_all.py --config experiments/config.yaml
python3 experiments/verify_claims.py

The pipeline is:

fixtures/ + run_cache/
        |
        v
experiments/export_records.py
        |
        v
results/records.csv
        |
        v
experiments/analyse.py
        |
        ├── results/matched_budget.csv
        ├── results/adaptive.csv
        ├── results/summary.json
        └── results/tables.md
        |
        v
experiments/make_figures.py
        |
        v
figures/*.png

Each record contains:

run_key,n_questions,model,strategy,param,budget,
problem_id,seed,subset,correct,calls,tokens,depth,latency_ms

verify_claims.py traces the reported claims back to the raw experiment records and checks the calculations. It does not make API calls.

Parallel vs Sequential

The experiment compares two basic strategies.

Parallel Sampling

Generate multiple independent answers and choose the majority answer.

Question
   ├── sample 1
   ├── sample 2
   ├── sample 3
   ├── ...
   └── sample N
          ↓
     majority vote

Sequential Revision

Generate an answer and repeatedly ask the model to revise it.

Question
   ↓
Answer 1
   ↓
Revision 1
   ↓
Revision 2
   ↓
...
   ↓
Answer N

Both strategies use the same overall inference budget. The difference is how that compute is allocated.

Findings

The bundled experiments use the same 100 questions and seed 42 across two models.

1. Coverage Is the Ceiling

Parallel methods can only choose from the answers they sampled.

If the correct answer never appears in the sample pool, majority voting cannot recover it.

Sequential revision does not have the same limitation because it can generate a new answer based on the previous one.

In two gpt-oss examples, the correct answer appeared in 0 of 16 independent samples, while sequential revision eventually reached the correct answer.

So the two strategies fail differently:

• —Parallel sampling: limited by whether the model samples the correct answer.
• —Sequential revision: limited by whether the model can recognize and correct its own mistakes.

2. The Best Allocation Depends on the Model

Hard-subset accuracy at matched budgets:

The result is model-dependent.

• —gpt-oss-120b benefits more from sequential revision.
• —gemma-3n-E4B-it performs better with parallel sampling.

The confidence intervals overlap substantially, so these results are evidence for further investigation, not a claim that one strategy is universally better.

Sequential revision also saturates quickly. For both models, performance is mostly flat from B=4 to B=16, while parallel sampling continues to improve slightly.

3. Sampling Variance Matters at Low Budgets

The parallel strategy was evaluated across five different orderings of the same sample pool.

For gpt-oss-120b:

At small budgets, the ordering of samples can change the majority vote enough to matter.

At B=16, every sample is used, so the variance disappears.

4. Revision Does Not Automatically Know When It Is Wrong

Sequential revision is not the same as having a verifier.

For gemma, the model left its answer unchanged on 85 of 88 questions it had already answered correctly. On incorrect answers, it often either confirmed the mistake or changed to another incorrect answer.

This suggests that the missing component is an error signal.

Additional prompt experiments showed that even small changes in how revision is framed can produce very different results:

• —gemma: 2/12 to 5/12
• —gpt-oss: 10/17 to 6/17
• —adversarial error framing caused gpt-oss to fall to 1/17

The model can revise, but it does not reliably know when revision is necessary.

5. Adaptive Stopping Can Reduce Compute

Instead of always using the full sample budget, the experiment also tests early stopping.

The adaptive policy generates answers one at a time and stops when the answers agree enough. It never sees the correct answer while making the decision.

For the bundled runs, this reduced the number of calls by about 86% for gemma while keeping the same accuracy.

This is a result of this particular stopping rule and evaluation set, not a general claim about adaptive inference.

6. A Third Model Shows the Same Pattern

A third model, gpt-oss-20b, was tested after the original two-model comparison above, to check whether the sequential advantage seen on gpt-oss-120b reflects the model family rather than one model.

Hard-subset accuracy at matched budgets:

It does: sequential leads from B=2 onward, peaking at 14/18 (77.8%) at B=8 before a small non-monotonic dip to 13/18 (72.2%) at B=16, the same saturate-then-wobble pattern seen on gpt-oss-120b. This is two data points from one family, not a family-level claim.

What These Experiments Do Not Establish

These experiments are deliberately limited.

• —They do not show that sequential revision is always better.
• —They do not show that parallel sampling is always better.
• —They do not establish that adaptive inference generally saves 86% compute.
• —Agreement does not guarantee correctness. One question had all 16 samples agree on the wrong answer.
• —Equal compute does not mean equal latency. Sequential inference has greater depth, while actual wall-clock time depends on batching and serving.
• —The results do not establish generalization beyond these models and this evaluation.
• —GSM8K's short numerical answers are particularly favorable to majority voting.

There is also at least one apparent labeling issue in the GSM8K data: one question uses 2280 as the gold answer even though 2180 is arithmetically correct. The experiment keeps the dataset label rather than silently changing it.

Cost Controls

Live experiments can become expensive quickly, so the application includes several safeguards.

Identical configurations are served from cache instead of being billed again.

Changing the model, prompts, sampling parameters, or question set creates a new experiment.

Replacing a Deprecated Model

Together AI's serverless model catalog changes over time. A model can be deprecated with a few weeks' notice, and its replacement is not guaranteed to behave the same way.

This already happened once: Together AI issued a deprecation notice for openai/gpt-oss-20b (effective 2026-09-15), the model used for the run in Finding 6 above. That run is unaffected: it is a completed, frozen 100-question run shipped in fixtures/, not a live dependency. The model currently pinned in backend/ttc_core.py for new runs launched from the Lab is meta-llama/Llama-3.3-70B-Instruct-Turbo, chosen and validated after the deprecation, not one of the three models the Findings above report on. Every run records which model produced it (config.model), so this is never ambiguous.

If the model currently pinned in backend/ttc_core.py stops working in turn, replace it the same way:

1. Check what is currently available.

https://api.together.xyz/v1/models

2. Test the candidate model directly, before changing any code. A 200 response is not enough on its own: some models spend the full token budget on internal reasoning and return an empty content field on harder questions. Confirm content is non-empty and ends with a real Final Answer: <number> line.

import httpx

resp = httpx.post(
    "https://api.together.xyz/v1/chat/completions",
    headers={"Authorization": f"Bearer {TOGETHER_API_KEY}"},
    json={
        "model": "candidate/model-name",
        "messages": [{"role": "user", "content": "<a real, multi-step GSM8K question>"}],
        "temperature": 0.0,
        "max_tokens": 512,
    },
)
data = resp.json()
content = data["choices"][0]["message"]["content"]
print(len(content), content[-100:])

Also send a handful of concurrent requests to check the model holds up under this app's actual concurrency, not just a single call.

3. Update the model in two places.

# backend/ttc_core.py
MODEL = "candidate/model-name"

# experiments/config.yaml
models:
  - candidate/model-name

4. Restart the backend and launch a new run from the Lab. The run is saved under its own cache key, so it does not overwrite or get confused with a previous model's results.

5. Fold the new run into the tables and figures.

python3 experiments/run_all.py --config experiments/config.yaml

Every record stores which model produced it (config.model in each run, model in results/records.csv), so old and new results stay distinguishable even after the pinned model changes again. Old results are never overwritten or deleted when a model is replaced.

API

GET  /api/health
GET  /api/ttc/estimate
POST /api/ttc/run
GET  /api/ttc/run/{run_id}
GET  /api/ttc/cached/{cache_key}
GET  /api/ttc/lessons
GET  /api/ttc/lesson/{cache_key}
GET  /api/ttc/adaptive/{cache_key}
GET  /api/ttc/compare
GET  /api/ttc/summary

Runs are polled by the frontend. A POST request returns a run_id, and the frontend checks the run approximately every 900 ms.

Project Structure

backend/
  ttc_core.py
  app/
    runner
    cache
    adaptive
    routes

frontend/
  Next.js lesson, map, and lab

src/
  records.py
  metrics.py

experiments/
  config.yaml
  run_all.py
  verify_claims.py

tools/
  verify_port.py
  extend_sequential.py
  verifier_arm.py
  revision_diagnostic.py

fixtures/
  completed experiment runs

results/
  generated tables and summaries

figures/
  generated figures

The Main Takeaway

The experiment is not trying to prove that one test-time compute strategy wins.

The main point is that the same inference budget can behave very differently depending on how the compute is allocated.

Parallel sampling spends compute on breadth.

Sequential revision spends compute on depth.

Which one works better depends on the model, the task, and whether the model has a reliable way to recognize its own errors.

That is the question the interactive lab is designed to let students explore directly.