FlyaiLab/ecommerce_last_exam

E-Commerce Last Exam

<iframe src="https://FlyaiLab-ecommercelastexam_leaderboard.hf.space" frameborder="0" width="100%" height="600"

</iframe>

A benchmark for evaluating LLM agents on 120 real-world travel planning and e-commerce tool-use tasks. Each task runs in an isolated Docker container with domain-specific CLI tools and SQLite databases. Agents must search, analyze, and produce structured recommendations.

• —Repository: alibaba-flyai/ecommerce_last_exam
• —Evaluation CLI: flyai-bench (pip install flyai-bench)
• —Leaderboard: FlyaiLab/ecommerce_last_exam_leaderboard

Dataset Summary

E-Commerce Last Exam consists of 120 tasks across two configs:

Each task provides a natural-language user question and a pre-built Docker image containing the environment (CLI tools, databases, test harness). Agents interact with the environment via tool calls and produce a structured answer scored 0.00 - 1.00.

Data Fields

Usage

from datasets import load_dataset

# Load all 120 tasks
ds = load_dataset("FlyaiLab/ecommerce_last_exam", split="test")

# Load only travel tasks (77)
ds_travel = load_dataset("FlyaiLab/ecommerce_last_exam", "travel", split="test")

# Load only e-commerce tasks (43)
ds_ecom = load_dataset("FlyaiLab/ecommerce_last_exam", "e_commerce", split="test")

Evaluation

pip install flyai-bench

# Run evaluation on travel config
flyai-bench run --dataset-config travel --limit 5 --dry-run
flyai-bench run --dataset-config travel

# Generate report and submit
flyai-bench report
flyai-bench submit --model your-model --provider your-provider

See the GitHub repository for full documentation.

Citation

If you use this benchmark, please cite:

@misc{ecommerce_last_exam,
  title={E-Commerce Last Exam: A Benchmark for LLM Agent Evaluation on Real-World Tool-Use Tasks},
  author={FlyaiLab},
  year={2026},
  url={https://huggingface.co/datasets/FlyaiLab/ecommerce_last_exam}
}