CoolFace
Datasetpublic

Lux1997/eMCR

eMCR: A Benchmark for Multi-Condition Product Retrieval in Chinese E-Commerce This is the official dataset and evaluation code for the paper "eMCR: A Benchmark for Multi-Condition Product Retrieval in Chinese E-Commerce". Overview Product search increasingly involves queries that combine multiple requirements — product attributes, brands, prices, exclusions, and visual descriptions. Existing retrieval benchmarks provide limited support for diagnosing which… See the full description on the dataset page: https://huggingface.co/datasets/Lux1997/eMCR.

sourceHugging Faceapache-2.0updated 16d agoView on Hugging Face
0likes887downloads
Dataset Card

eMCR: A Benchmark for Multi-Condition Product Retrieval in Chinese E-Commerce

This is the official dataset and evaluation code for the paper "eMCR: A Benchmark for Multi-Condition Product Retrieval in Chinese E-Commerce".

Overview

Product search increasingly involves queries that combine multiple requirements — product attributes, brands, prices, exclusions, and visual descriptions. Existing retrieval benchmarks provide limited support for diagnosing which conditions cause failures and how performance changes when conditions are combined.

eMCR is a diagnostic benchmark for multi-condition product retrieval in Chinese e-commerce. Built from a large-scale real-world Taobao product catalog, eMCR adopts a search-log-informed taxonomy of 15 atomic conditions spanning expression variation, intent type, and constraint signals. Controlled LLM-assisted generation composes these conditions into 48 task types with one to three condition signals.

Key Statistics

ItemCount
Queries1,399
Products36,722
Relevance Judgments42,022
Relevance Levels4 (0–3)
Atomic Conditions15
Task Types48

[image] Figure 1: Overview of the eMCR benchmark construction pipeline, including product sampling from a real-world e-commerce catalog, query generation with multi-condition constraints, candidate pool construction, and graded relevance annotation.

Evaluated Systems

We evaluate 16 systems across four paradigms:

  • —Sparse retrieval: BM25
  • —Dense retrieval: BGE-M3, E5-Mistral, GritLM, NV-Embed-v2, Qwen3-Embedding (0.6B/4B/8B)
  • —Multimodal: Qwen3-VL (2B/8B), VLM2Vec, MM-Embed
  • —Reranking: Cross-Encoder, Qwen3-Reranker, Qwen3-VL-Reranker
  • —Agentic: LLM-driven retrieval agent with tool use

[image] Figure 3: Cross-system P@1 performance breakdown by atomic condition.

Getting Started

Installation

bash
pip install -r requirements.txt

Data Structure

data/
├── emcr_benchmark.jsonl    # Benchmark queries and relevance labels
└── images/                 # Product images
    ├── part_0/
    ├── part_1/
    ├── part_2/
    └── part_3/

Run Evaluation

Dense Retrieval (e.g., BGE-M3):

bash
python -m mcr.cli \
  --benchmark configs/benchmarks/emcr.yaml \
  --model configs/models/bge_m3.yaml

Reranking (e.g., Qwen3-Reranker):

bash
python -m mcr.cli \
  --benchmark configs/benchmarks/emcr.yaml \
  --model configs/models/qwen3_reranker_8b.yaml

Agentic Retrieval:

bash
export DASHSCOPE_API_KEY=your_key_here
python run_agentic_v2.py \
  --benchmark configs/benchmarks/emcr.yaml \
  --model configs/models/agentic_qwen3_7_max.yaml

Configuration

  • —Benchmark configs: configs/benchmarks/
  • —Model configs: configs/models/

Each model config specifies the encoder/reranker class and parameters. Set API keys via environment variables where needed.

Citation

If you find this benchmark useful, please cite:

bibtex
@article{lu2025emcr,
  title={eMCR: A Benchmark for Multi-Condition Product Retrieval in Chinese E-Commerce},
  author={Lu, Xuan and Li, Qiang and Du, Xiaoxiong and Huang, Haohang and Zheng, Junjun and Kong, Xiangheng and Ou, Dan and Shen, Xiaoyu},
  year={2025}
}

License

This dataset is released under the Apache 2.0 License.

Lux1997/eMCR · CoolFace