CoolFace
Apppublic

flavianv/reco-protocol-a-leaderboard

sourceHugging Faceupdated 6mo agoView on Hugging Face
0likes
App README

Reco Leaderboard

Minimal playground for recommendation benchmarking. The active HF-deployable target is SmartBench Lite Arena: a locked-tool benchmark with a shared ReAct-lite agent loop.

Submitted model support:

  • —OpenAI model names like gpt-4.1-mini
  • —Hugging Face chat models like meta-llama/Llama-3.1-8B-Instruct
  • —Optional explicit prefixes:
  • —openai:gpt-4.1-mini
  • —hf:meta-llama/Llama-3.1-8B-Instruct

Setup

bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install .

SmartBench Lite: what this repo implements

Minimal HF-only setup

  • —Tiny public/private holdout query files or an HF-hosted Musical Instruments benchmark pack
  • —Locked local tools:
  • —semantic_search using the actual base E5 retrieval assets from iaouali/hybrid-reco
  • —perf_search using the fine-tuned item embedding space from iaouali/hybrid-reco
  • —get_complementary_products
  • —add_candidates
  • —rm_candidates
  • —score_candidates
  • —OpenAI used only for:
  • —deciding the next tool action inside the shared loop
  • —final 4-dimension judge
  • —The judge remains benchmark-owned and fixed via SMARTBENCH_LITE_JUDGE_MODEL
  • —Deterministic fallback if OPENAI_API_KEY is absent
  • —Repeated runs per query (k) and aggregate leaderboard scoring

What was implemented in this iteration

  1. 1.Added a separate smartbench_lite benchmark module.
  2. 2.Added the smallest dataset and local tool assets needed to run a locked-tool recommendation arena.
  3. 3.Added an OpenAI-backed planner/judge with deterministic fallback.
  4. 4.Added a Streamlit app that can launch evaluations directly inside an HF Space.
  5. 5.Added tests for the no-secret fallback path.
  6. 6.Added an HF asset loader that downloads and caches Musical Instruments data from iaouali/hybrid-reco.
  7. 7.Added a shared agent_react_lite.txt prompt and a step-by-step candidate-management loop inspired by SmartBench DeepShopper.
  • —Dataset files:
  • —hf_dataset/smartbench_lite_public.jsonl
  • —hf_dataset/smartbench_lite_private.jsonl
  • —Locked tool assets:
  • —space_assets/smartbench_lite_catalog.jsonl
  • —space_assets/smartbench_lite_complements.json
  • —Harness:
  • —src/reco_leaderboard/smartbench_lite/*
  • —Evaluation script:
  • —scripts/evaluate_smartbench_lite.py
  • —HF app:
  • —app_smartbench_lite.py
  • —Leaderboard artifact:
  • —data/smartbench_lite_leaderboard.csv

This is the lightest coherent SmartBench-style benchmark that still keeps planning and candidate selection agentic.

Run SmartBench Lite locally

Evaluate on public split:

bash
python scripts/evaluate_smartbench_lite.py --model-id gpt-4.1-mini --split public --runs-per-query 2 --asset-source hf-musical

Evaluate on private split:

bash
python scripts/evaluate_smartbench_lite.py --model-id gpt-4.1-mini --split private --runs-per-query 3 --asset-source hf-musical

Open the HF-target app locally:

bash
streamlit run app_smartbench_lite.py

Files to map directly to HF

  • —HF Space app: app_smartbench_lite.py
  • —Eval backend in Space: scripts/evaluate_smartbench_lite.py
  • —Locked tools: src/reco_leaderboard/smartbench_lite/*
  • —Shared loop prompt: prompts/agent_react_lite.txt
  • —HF-hosted Musical Instruments assets are cached under .cache/hf_assets/musical_instruments
  • —Local fallback mini dataset: hf_dataset/smartbench_lite_*.jsonl
  • —Leaderboard table: data/smartbench_lite_leaderboard.csv

Retrieval note:

  • —semantic_search now uses the real base retrieval assets from the SmartBench Light repo.
  • —perf_search uses the fine-tuned item embedding space from the repo with a base E5 query encoder in the HF Space path.
  • —This keeps the Space deployable on lightweight hardware without loading the full multi-GB fine-tuned query checkpoint.

Evaluation contract

  • —Submitter controls: model_id, temperature
  • —Benchmark controls: tasks, tools, prompt shape, scoring, repeated runs
  • —Leaderboard metrics:
  • —overall_score
  • —performance_score
  • —need_fit
  • —completeness
  • —redundancy_control
  • —explanation_quality
  • —avg_tool_calls
  • —avg_latency_ms

Notes for real HF deployment

  • —Keep the real hidden split out of the public repo. The included private file is only for demo deployment.
  • —Set OPENAI_API_KEY for OpenAI submissions.
  • —Set HF_TOKEN for Hugging Face chat-model submissions through the HF router.
  • —Set SMARTBENCH_LITE_JUDGE_MODEL if you want to override the fixed benchmark judge model.
  • —Without the key, the app falls back to deterministic heuristic evaluation.
  • —Keep runs_per_query and dataset size small if you want HF-only execution.
  • —The first hf-musical run downloads and converts the remote assets; later runs reuse the local cache.

Tests

bash
python -m pytest -q

Deploy as a Hugging Face Space

  1. 1.Create a Streamlit Space or sync this repo to an existing one.
  2. 2.Set secret OPENAI_API_KEY if you want model-driven planning and judging.
  3. 3.HF will launch app_smartbench_lite.py.
  4. 4.Use the sidebar in the app to run benchmark submissions directly.