NoeFlandre/benchmark-llms-landuse-relevance
Land-use relevance benchmark v3-multilingual · 85 languages x 300 items/language · 25,500 items · binary yes/no labels. Code Task and prompt Does a sentence describe a place's land or environment in ways visible to satellites? English prompt · greedy decoding · seed 0 · max_new_tokens=4096 · bfloat16 · batch varies by model. unsloth/Qwen3.8-27B-GGUF@UD-IQ2_XXS runs the UD-IQ2_XXS GGUF quant through llama.cpp (same prompt, template, greedy decoding and budget).… See the full description on the dataset page: https://huggingface.co/datasets/NoeFlandre/benchmark-llms-landuse-relevance.
Land-use relevance benchmark
v3-multilingual · 85 languages x 300 items/language · 25,500 items · binary yes/no labels.
Task and prompt
Does a sentence describe a place's land or environment in ways visible to satellites?
English prompt · greedy decoding · seed 0 · max_new_tokens=4096 · bfloat16 · batch varies by model. unsloth/Qwen3.8-27B-GGUF@UD-IQ2_XXS runs the UD-IQ2_XXS GGUF quant through llama.cpp (same prompt, template, greedy decoding and budget).
Prompt text
Replace {} with the target sentence.
Classify whether the TARGET SENTENCE contains information about the target place that could help characterize its land use, land cover, or geographic environment from remote sensing, either directly or through observable proxies.
Return exactly one token: yes or no.
Answer yes for information about vegetation, agriculture, forests, water, soil or surface, terrain, buildings, settlements, infrastructure, transport networks, mining, managed land, or other human or natural features with a spatial or remotely detectable signature.
Answer no for information only about history, administration, people, events, demographics, economy, navigation, or activities with no meaningful land-use, land-cover, or remotely detectable implication.
Output only the lowercase token yes or no.
TARGET SENTENCE: {}Aggregate scores
Per-model macro averages across languages. Full metrics: `aggregates.csv`. Bold = best; underline = second best in each metric column.
Scoring models
Scores are normalized to [0, 1]. Best thresholds are selected on this benchmark (an upper bound); ROC-AUC needs no threshold. Full sweep: `threshold_sweep.csv`.
Scoring setup
Scoring prompts
Alibaba-NLP/gte-multilingual-reranker-base, Qwen/Qwen3-Reranker-0.6B, Qwen/Qwen3-Reranker-4B, convaiinnovations/laya-multilingual, mixedbread-ai/mxbai-rerank-base-v2:
<Instruct>: Judge whether the Document carries information about its target place that could help characterize land use, land cover, or the geographic environment from remote sensing, either directly or through observable proxies. Answer yes for vegetation, agriculture, forests, water, soil or surface, terrain, buildings, settlements, infrastructure, transport networks, mining, managed land, or other human or natural features with a spatial or remotely detectable signature. Answer no for information only about history, administration, people, events, demographics, economy, navigation, or activities with no meaningful land-use, land-cover, or remotely detectable implication.
<Query>: Does this sentence carry land-use, land-cover, or geographic-environment signal observable from remote sensing?
<Document>: {}BalaRajesh1/mmbert-small-nli, MoritzLaurer/bge-m3-zeroshot-v2.0, MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7, fastino/gliner2.5-multi-v1, knowledgator/gliclass-multilang-mini:
Zero-shot classification input for NLI and label-matching models. The premise is the TARGET SENTENCE; the hypothesis / positive label is the line after HYPOTHESIS, derived from the LLM prompt.
TARGET SENTENCE: {}
HYPOTHESIS: This sentence contains information useful for inferring the land use or land cover of the associated geographic place.LiquidAI/LFM2.5-2.6B@logprob uses the task prompt above.
Best thresholded scoring metrics
LFM2.5-2.6B: log-probabilities vs generation
Same model, prompt and languages. Generation can reason inside LFM's open <think> block before parsing its final yes/no; log-probs close the block empty and compare immediate next-token P(yes) vs. P(no), so the decision contexts differ.
