giantfish-fly/LLMs-First-Task

Super easy task for humans that All SOTA LLM fail to retrieve the correct answer from context. Including SOTA models: GPT5, Grok4, DeepSeek, Gemini 2.5PRO, Mistral, Llama4...etc Update: Accepted to COLM 2026 (San Francisco). AAAI 2026 Worshop Oral: Jan/2026 LaMAS (LLM-based Multi-Agent Systems: Towards Responsible, Reliable, and Scalable Agentic Systems) Jan/2026 Singapole ICML 2025 Long-Context Foundation Models Workshop Accepted.(https://arxiv.org/abs/2506.08184) Update: This dataset… See the full description on the dataset page: https://huggingface.co/datasets/giantfish-fly/LLMs-First-Task.