Anvit25/meta-signal

Meta-Signal: Privacy-Constrained Ad Budget Optimisation

Live demo: huggingface.co/spaces/Anvit25/meta-signal Dataset: huggingface.co/datasets/Anvit25/meta-signal-expert-demos Trained model: huggingface.co/Anvit25/meta-signal-q4-agent Blog post: huggingface.co/Anvit25/meta-signal-q4-agent Demo video: youtube.com/watch?v=M4gHED62yyQ

An OpenEnv-compliant RL environment where an AI agent manages advertising budget across three campaigns but can only observe noisy, aggregated conversion data — exactly how Meta's real ad system works after iOS signal loss. Includes the full Q4 Gauntlet extension: a 100-day narrative episode across four operational phases.

Why This Matters

On October 26, 2022, Meta reported its third-quarter earnings. Revenue had fallen year-over-year for the second consecutive quarter. The stock dropped 24% in after-hours trading. $232 billion in market capitalisation was erased in a single session — the largest single-day destruction of market value for any US company in history.

Zuckerberg named two causes. One was the metaverse. The other was signal loss.

Apple's App Tracking Transparency (ATT) prompt shipped in iOS 14.5. Roughly 80% of users opted out. Overnight, the deterministic, pixel-level conversion signals that Meta's ad auction had been trained on for a decade were replaced by aggregated counts, delayed postbacks, and Apple's coarse-grained SKAdNetwork attribution.

Meta's response was Aggregated Event Measurement (AEM) — a differential-privacy API that adds calibrated Laplace noise to conversion counts. It preserved some signal, but introduced a new constraint: signal quality degrades the more you query it. Budget allocation decisions that had been made on clean, dense data now had to be made on a finite, depletable information budget.

That is precisely the problem this environment models.

A trained RL agent on Meta-Signal is directly applicable to Meta's Advantage+ signal recovery pipeline.

Architecture

HTTP Client / LLM Agent
        │
        ▼
FastAPI Server (app/main.py)
        │
  ┌─────┴──────┐
  │            │
MetaSignalEnv  PrivacyEngine
(app/env.py)   (app/privacy.py)
  │
  ├── PhaseController  (Q4 Gauntlet phases)
  ├── MarketTrendGen   (100-day market signal)
  ├── DataLoader       (Criteo snapshot)
  └── TaskGraders      (per-task scoring)

Campaigns

Privacy Mechanic

The agent has a finite epsilon (ε) budget (differential privacy):

As epsilon depletes, Laplace noise scale grows through four regimes:

ATT structural noise (Q4 Phase 2): iOS App Tracking Transparency fires a 3× noise multiplier that epsilon budget cannot fix. The only counter is CAPI.

Tasks

Core Tasks (1–4)

Q4 Gauntlet Tasks (5–7)

Q4 Gauntlet — Four-Phase Narrative

Task 7 runs a 100-day episode across four distinct operational phases. Each phase changes the hidden mechanics the agent must adapt to.

Phase 1 — The Setup (Days 1–20)

Signal is clean. Use these steps to identify which campaign has the best ROAS. Progressive budget shift toward the leader. Stay below 70% to avoid the correlation penalty (>70% concentration drops other campaigns' CTR by 15%).

Phase 2 — ATT Blackout (Days 21–50)

iOS App Tracking Transparency fires. Noise is 3× higher — epsilon budget cannot fix this. The only counter is CAPI (use_capi=True in the action):

• —Costs 2.0ε per call
• —Returns true (noise-free) conversion counts
• —Ration carefully: 1 call every 3–5 steps is the optimal cadence
• —Between calls: hold Phase 1 allocation, do not chase the corrupted signal

Phase 3 — Andromeda Glitch (Days 51–80)

The Andromeda algorithm update is live. Any allocation change exceeding 20% of total budget in a single step triggers a 7-day learning reset — CVR drops to 30% of normal. The observation's learning_status field reports the state:

• —Optimized — normal performance
• —Learning — ramping up after a reset
• —Reset — just triggered, do not change allocations for 7 steps

Phase 4 — Black Friday Peak (Days 81–100)

Maximum traffic, doubled noise volatility. Setting pacing_speed > 1.5 in the action triggers a 30% chance per step of a midnight overspend event — the remaining budget is consumed in a single step. Set pacing_speed=1.0 and hold.

Self-Improvement Mechanic

If an agent beats ROAS > 3.0 for 5 consecutive steps, difficulty escalates on the next episode. The environment adapts to strong agents.

Action Space

{
  "allocations": {
    "camp_feed":    500.0,
    "camp_reels":   300.0,
    "camp_stories": 200.0
  },
  "attribution":      "last_click",
  "feature_mask":     ["I1"],
  "halted_campaigns": [],
  "legal_reason_code": null,
  "use_capi": false,
  "pacing_speed": 1.0,
  "apply_safety_cap": true
}

Observation Space

{
  "step": 25,
  "day": 25,
  "campaigns": [
    {
      "campaign_id": "camp_feed",
      "placement": "feed",
      "impressions": 35,
      "spend": 100.0,
      "noisy_conversions": 2.1,
      "estimated_roas": 1.43,
      "ctr": 0.0857,
      "confidence_interval": [0.8, 3.4]
    }
  ],
  "total_budget_remaining": 7500.0,
  "epsilon_remaining": 14.2,
  "privacy_regime": "high_noise",
  "available_features": ["I1", "I2", "I3"],
  "platform_health": "Signal_Loss",
  "learning_status": "Optimized",
  "market_trend": "Rising",
  "regulatory_violation": false,
  "audit_active": false,
  "flagged_campaign": null,
  "warning": null
}

API Endpoints

POST /hint — Expert-in-the-Loop

Inspired by Snorkel AI's expert annotation mechanic. Returns context-aware advice for the current episode phase — situation, strategy, what to watch for, CAPI advice, and live epsilon/budget stats.

curl -X POST http://localhost:7860/hint

{
  "phase": 2,
  "title": "Phase 2 — ATT Blackout (Days 21–50)",
  "situation": "iOS ATT has fired. Noise is 3× higher than normal.",
  "advice": "Use CAPI calls (use_capi=True, costs 2.0ε each). Ration carefully...",
  "watch_for": "Epsilon exhaustion: below 0.5 you enter high_noise regime.",
  "capi_advice": "Use CAPI now. This is what it is for.",
  "current_day": 24,
  "epsilon_remaining": 14.2,
  "epsilon_pct": 71.0,
  "budget_remaining": 7800.0,
  "budget_pct": 78.0,
  "learning_resets": 0,
  "overspend_events": 0,
  "capi_calls_used": 2
}

POST /simulate — no-code exploration

{
  "task_id": 7,
  "strategy": "conservative",
  "seed": 42
}

Baseline Scores

Scores from the deterministic ExpertBot (training/expert_bot.py, seed=42):

LLM baseline (llama-3.3-70b-versatile via Groq, Tasks 1–3): 0.43 / 0.54 / 0.72

Results

[image]

Left: ExpertBot baseline across all 7 tasks. Right: Reward improvement — Base model (no fine-tuning) → ExpertBot → Fine-tuned Llama-3.1-8B on Q4 Gauntlet tasks (3 seeds each).

Fine-Tuned Model Evaluation

training/evaluate_finetuned.ipynb — 9 episodes (3 seeds × Tasks 5/6/7) against the live environment.

Reward Improvement

Key findings:

• —Task 5: Fine-tuned model scores +67% above the base model (0.800 vs 0.479) — CAPI rationing strategy fully learned
• —Task 6: Fine-tuned model scores +82% above the base model (0.949 vs 0.522) and beats ExpertBot by +8.5 points — learned a superior freeze strategy
• —Task 7: Fine-tuned model scores +56% above the base model (0.850 vs 0.545) — 4-phase strategy fully learned
• —Overall: fine-tuned model beats ExpertBot by +3.3% on the Q4 Gauntlet
• —Training: 1 epoch on ~41k expert demos, loss 0.1080, 2,563 steps (~166 min on A10G)

Training Pipeline

A complete supervised fine-tuning pipeline is included in training/.

1. Expert Bot

python -m training.expert_bot --task 7 --seed 42 --verbose

Deterministic 4-phase strategy: explore (Phase 1) → CAPI ration (Phase 2) → freeze (Phase 3) → hold (Phase 4). Scores ~0.85 on Task 7.

2. Dataset Generation

python -m training.generate_dataset --tasks 5 6 7 --episodes 200 --out data/expert_demos.jsonl

Generates Alpaca-format JSONL with one record per step:

• —instruction: phase-specific strategy description
• —input: serialised observation (step, day, phase, campaigns, budget, epsilon)
• —output: expert action as JSON
• —metadata: task/seed/score for quality filtering

Published dataset: ~41,000 records (200 episodes × 3 tasks) at huggingface.co/datasets/Anvit25/meta-signal-expert-demos

3. Unsloth Fine-Tune (A10G, ~166 min)

training/unsloth_finetune.ipynb — fine-tunes Llama-3.1-8B-Instruct with 4-bit QLoRA (rank=16) on the expert demonstrations. Loads dataset from HF Hub, pushes trained adapter to Anvit25/meta-signal-q4-agent.

Actual training stats (A10G Small, ~41k records, 1 epoch):

• —Loss: 0.1080 — model correctly learns CAPI rationing, freeze, and hold strategies
• —Runtime: ~166 min (2,563 steps with sequence packing at maxseqlen=2048)
• —Inference validation: correctly sets use_capi=true for Phase 2 without explicit instruction

Trained model: huggingface.co/Anvit25/meta-signal-q4-agent

Setup

Local

cd meta-signal-env
pip install -r requirements.txt
uvicorn app.main:app --host 0.0.0.0 --port 7860

Docker

docker build -t meta-signal .
docker run -p 7860:7860 meta-signal

Run inference script

export API_BASE_URL=https://router.huggingface.co/v1
export MODEL_NAME=meta-llama/Llama-3.3-70B-Instruct
export HF_TOKEN=your_hf_token

python inference.py

Run tests

pytest tests/ -q   # 47 tests, all passing

Project Structure

meta-signal-env/
├── app/
│   ├── data_loader.py      Criteo loader + MarketTrendGenerator
│   ├── env.py              Core environment + Q4 phase controller
│   ├── main.py             FastAPI server (12 endpoints)
│   ├── models.py           Pydantic types (7 tasks, Q4 fields)
│   ├── privacy.py          Epsilon budget + Laplace noise + CAPI + ATT
│   ├── tasks.py            Task 1–7 configs + graders
│   └── static/index.html   Terminal-style dashboard UI
├── data/
│   ├── ad_logs_sampled.csv       Criteo-schema snapshot (10k rows)
│   └── expert_demos.jsonl        10,250 expert demonstration records
├── training/
│   ├── expert_bot.py             Deterministic 4-phase expert strategy
│   ├── generate_dataset.py       Alpaca-format JSONL dataset generator
│   ├── unsloth_finetune.ipynb    QLoRA fine-tune notebook (A10G, ~12 min)
│   └── push_dataset_to_hub.py    Upload dataset to HF Hub
├── tests/
│   └── test_server.py            47 end-to-end tests
├── inference.py                  LLM inference script
├── baseline.py                   Baseline runner
├── openenv.yaml                  OpenEnv competition manifest
├── Dockerfile
└── requirements.txt

Tags

openenv advertising differential-privacy reinforcement-learning budget-optimisation signal-loss q4-gauntlet att capi unsloth lora