Datasetpublic

guangliangliu/mewl-repro

MEWL Inference Reproduction (ICML'23 benchmark, modern models) Full-test-set reproduction of inference on MEWL (MachinE Word Learning, Jiang et al., ICML 2023) — 9 word-learning tasks x 600 test episodes, evaluated with 4 modern models (zero-shot). Each row = one episode: 6 context images (each labeled with a novel-word utterance), the query image, 5 candidate answers, the ground-truth answer, and each model's prediction + correctness. Subsets = the 9 task categories. Splits =… See the full description on the dataset page: https://huggingface.co/datasets/guangliangliu/mewl-repro.

sourceHugging Facecc-by-4.0updated 3mo agoView on Hugging Face

0likes156downloads

Dataset Card

MEWL Inference Reproduction (ICML'23 benchmark, modern models)

Full-test-set reproduction of inference on MEWL (MachinE Word Learning, Jiang et al., ICML 2023) — 9 word-learning tasks x 600 test episodes, evaluated with 4 modern models (zero-shot).

Each row = one episode: 6 context images (each labeled with a novel-word utterance), the query image, 5 candidate answers, the ground-truth answer, and each model's prediction + correctness.

Subsets = the 9 task categories. Splits = `work` / `not_work`: an episode is in work if >= 2 of the 4 models answered it correctly, else in not_work (chance that two independent models are both correct by luck is 4%). Use the n_models_correct column (0-4) to re-slice with a stricter or looser criterion.

Protocols

Route	Protocol	Models
Text	oracle caption + likelihood forced-choice — byte-identical reproduction of the paper's `gpt3.py` prompt (verified on all 5,400 episodes)	Llama-3.1-8B-Instruct, Qwen2.5-32B-Instruct
Vision	6 labeled context images + query image -> generative 5-way choice	Qwen2.5-VL-7B-Instruct, Qwen2.5-VL-32B-Instruct

Accuracy (test, n=600/task, chance = 20%)

Task	Llama-8B text	Qwen-32B text	VL-7B	VL-32B	Paper GPT-3.5	Human
object	.982	.980	.297	.730	98.2	79.1
shape	.888	.870	.827	.902	96.8	92.4
color	.872	.695	.557	.697	82.3	87.2
composite	.832	.843	.490	.817	88.3	63.5
material	.775	.667	.380	.558	87.0	72.7
bootstrap	.412	.428	.335	.498	45.8	71.0
pragmatic	.307	.275	.218	.217	26.7	54.8
number	.258	.242	.407	.492	22.7	93.9
relation	.213	.258	.225	.222	20.0	48.7

Key findings: cross-situational induction tasks (object/shape/color/ composite/material) work; relation fails at chance for all four models; pragmatic fails even when the pointing target is stated in text; number dissociates (text ~.25 vs vision ~.49 with a U-shaped count curve).

Full analysis: raw_results/REPORT.md; per-episode records: raw_results/<model>/<task>.jsonl; aggregates: raw_results/summary.json.

Fields

—context_images / context_labels: the 6 example scenes and their utterances (info.json order)
—question_image, choices (5), answer, answer_idx
—word_concept: JSON map novel word -> ground-truth concept
—pred_<model>, correct_<model> for the 4 models above

Attribution & license

Images and episode structure come from the MEWL dataset (Zenodo, CC-BY-4.0), (c) the MEWL authors. This repackaging adds model predictions and is released under the same CC-BY-4.0 terms.

bibtex

@inproceedings{jiang2023mewl,
  title={MEWL: Few-shot multimodal word learning with referential uncertainty},
  author={Jiang, Guangyuan and Xu, Manjie and Xin, Shiji and Liang, Wei and
          Peng, Yujia and Zhang, Chi and Zhu, Yixin},
  booktitle={ICML},
  year={2023}
}