chchenhui/mlrbench-tasks

This repository contains the benchmark dataset of MLR-Bench. We collect 201 tasks from ICLR/NeurIPS/ICML workshops over the past three years. The followings record the metadata of our collection.

Workshop without an official website or deleted

• —icml2024_fminwild
• —neurips2024attriblate
• —neurips2024_gsai
• —neurips2024_rlfm
• —iclr2023_ai4abm
• —iclr2023_ml4iot
• —iclr2023_mldd
• —iclr2023NeSyGeMs
• —neurips2023newin_ml
• —neurips2023_ai4mat
• —icml2023_esfomo

Non-general workshops

• —neurips2024_queerinai
• —iclr2024_africanlp
• —iclr2023_africanlp
• —iclr2023IndabaXRwanda
• —icml2023_lxai

Repeated workshops

• —iclr2023_dl4c
• —iclr2023_mefomo
• —iclr2023realign and iclr2024_realign (TODO: check the details and delete one)
• —icml2024_fminwild potentially repeat with other workshops

Missing workshop for unknown reason

• —ICML 2023 The Second Workshop on New Frontiers in Adversarial Machine Learning (https://advml-frontier.github.io/)

Workshop information to be updated

• —iclr2025_nfam (too many citations without link, should we keep or delete them? https://nfam.vizhub.ai/)

Workshop Links

• —ICML 2024 (33 workshops) https://openreview.net/group?id=ICML.cc/2024/Workshop
• —We keeped 24 workshops
• —Workshops not included
• —without an official website or deleted
• —Agentic Markets Workshop at ICML 2024
• —Workshop on Efficient Systems for Foundation Models II @ ICML2024
• —not included with unknown reason
• —ICML 2024 AI for Science Workshop (website: https://ai4sciencecommunity.github.io/icml24.html)
• —First Workshop on Controllable Video Generation @ICML24 (https://openreview.net/group?id=ICML.cc/2024/Workshop/CVG#tab-recent-activity / https://sites.google.com/view/cvgicml2024/home)
• —ICML 2024 Workshop Differentiable Almost Everything (https://openreview.net/group?id=ICML.cc/2024/Workshop/DifferentiableAlmostEverything#tab-accept / https://differentiable.xyz/)
• —ICML 2024 Workshop on Models of Human Feedback for AI Alignment (https://openreview.net/group?id=ICML.cc/2024/Workshop/MFHAIA#tab-accept-oral / https://sites.google.com/view/mhf-icml2024)
• —ICML 2024 Workshop on Mechanistic Interpretability (https://openreview.net/group?id=ICML.cc/2024/Workshop/MI#tab-accept-oral)
• —repeated page
• —Challenge of ICML 2024 TiFA Workshop
• —MLLM Attack Challenge of ICML 2024 TiFA Workshop
• —Non-general workshops
• —LatinX in AI (LXAI) Research
• —ICML 2024 Joint Workshop Queer in AI and {Dis}Ability in AI
• —Workshop out of nowhere
• —imcl2024_modelofhf??.md

• —NeurIPS 2023 Workshop (https://openreview.net/group?id=NeurIPS.cc/2023/Workshop)
• —In total 53 workshops
• —42 workshops have been kepped
• —Not included
• —not included with unknown reason
• —AI for Accelerated Materials Design - NeurIPS 2023 Workshop (https://openreview.net/group?id=NeurIPS.cc/2023/Workshop/AI4Mat#tab-accept-spotlight)
• —Associative Memory & Hopfield Networks in 2023 (https://openreview.net/group?id=NeurIPS.cc/2023/Workshop/AMHN#tab-accept-oral)
• —Socially Responsible Language Modelling Research (https://openreview.net/group?id=NeurIPS.cc/2023/Workshop/SoLaR#tab-accept-spotlight)
• —without an official website or deleted
• —NeurIPS Workshop on Attributing Model Behavior at Scale
• —NeurIPS 2023 Workshop on Backdoors in Deep Learning - The Good, the Bad, and the Ugly
• —Intrinsically-Motivated and Open-Ended Learning Workshop @NeurIPS2023
• —Multi-Agent Security Workshop @ NeurIPS'23
• —Not general
• —2nd Annual Global South in AI Meetup
• —Muslims in Machine Learning Workshop @ NeurIPS2023
• —New in Machine Learning Workshop, NeurIPS 2023
• —NeurIPS 2023 Queer in AI Workshop

• —Things to notice
• —neurips2023_mlncp.md has no topics -> maybe too general for the LLMs to generate ideas?
• —neurips2023_want.md has been repeated but not deleted

• —NeurIPS 2024 Workshop
• —54 workshops, 51 are included
• —delete
• —3rd Annual Global South in AI Meetup 2024
• —Latinx in AI @ NeurIPS 2024
• —Muslims in ML Workshop co-located with NeurIPS 2024
• —No website or delted
• —Workshop on Generalization and Safety from Reinforcement Learning to Foundation Models
• —Things to notice: without an official website or deleted
• —NeurIPS 2024 Workshop Machine Learning with new Compute Paradigms (still with an abstract here)
• —NeurIPS 2024 Third Table Representation Learning Workshop
• —UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models