anhaidgroup/polaris-lter-v2

LTER v2

LTER is one of six datasets in [Polaris: Learning to Generate Table Descriptions from Retrieval Feedback](https://arxiv.org/abs/2608.17171), alongside aw, arctic, ecir, wikitables, and wtr.

It holds 2,015 tables from the top-downloaded collections of the Long-Term Ecological Research sites in the Environmental Data Initiative (EDI) — bird surveys, forest phenology, reef colonisation, cattle records — and 15 keyword queries over them. For each query–table pair, a person decided whether that table answers that query; those decisions are the relevance judgments, and they live in qrels.csv. It draws on the same repository as arctic, but a larger and more focused sample.

Each Polaris dataset comes in two versions. v2, this one, adds the tuples — the rows of each table, in tuples.zip — on top of the table metadata, the queries, and the relevance judgments. v1 is the same without the tuples and is at polaris-lter-v1.

Files

`queries.csv` — one row per query.

query_id,query
q1,birdwatching
q4,beef exportation

`qrels.csv` — one row per relevant pair, here three of the tables relevant to q1.

query_id,table_id,relevance_score
q1,305,1
q1,306,1
q1,307,1

Judging was done by pooling: for each query, several retrievers proposed candidate tables and a person labelled each one, 1 for relevant and 0 for not. Only the 1s are listed here, so a table missing for a query counts as not relevant.

`metadata.csv` — one row per table, here those same three with their column lists cut short.

table_id,table_name,column_names
305,23_birds_1.csv,"[""id"", ""survey_id"", ""site_id"", ""direction"", ""species_id"", ...]"
306,23_surveys_1.csv,"[""survey_id"", ""site_id"", ""direction"", ""transect_len"", ""survey_date"", ...]"
307,23_sites_1.csv,"[""site_id"", ""description""]"

These three come from one bird-survey package, split into observations, surveys, and sites. Only the first has anything bird-like in its name, and the survey and site tables are relevant because of what they belong to rather than what they are called.

`tuples.zip` — one CSV per table, at Tuples/<table_id>.csv. 305 is 23_birds_1.csv, one of the tables relevant to q1. Its first rows:

id,survey_id,site_id,direction,species_id,obs_type,count_10m,count_20m
1,418,Twin Knol,N,VERD,calling,,
2,418,Twin Knol,N,WEKI,visual,,
3,419,Twin Knol,S,ANHU,singing,1.0,

A table appears in tuples.zip only if its tuples were available, so a few tables in metadata.csv have no CSV in the archive.

Each file's header matches that table's column_names. Tables were capped at 500,000 rows; 216 of them hit that cap.

Statistics

A table counts as gold if it scores above 0 for at least one query.

Download

The repo is about 1.86 GB, almost all of it tuples.zip, which unpacks to 13.2 GB. You do not need a Hugging Face account to download it.

Option 1 — click the Files tab at the top of this page and save each file.

Option 2 — command line (recommended):

pip install huggingface_hub
hf download anhaidgroup/polaris-lter-v2 --repo-type dataset --local-dir lter

Polaris has six datasets in total and LTER is one of them. Each sits in its own repository, so to download all six quickly — the v2 repositories, with tuples, about 4.8 GB in total:

for d in aw arctic lter ecir wikitables wtr; do
  hf download anhaidgroup/polaris-$d-v2 --repo-type dataset --local-dir polaris_v2/$d
done

Usage

column_names is an array, so it needs parsing when you load the file. For example:

import ast
import pandas as pd

metadata = pd.read_csv("lter/metadata.csv")
queries = pd.read_csv("lter/queries.csv")
qrels = pd.read_csv("lter/qrels.csv")

metadata["columns"] = metadata["column_names"].apply(ast.literal_eval)

relevant = qrels.loc[qrels.query_id == "q1", "table_id"].tolist()
# [305, 306, 307, ...]

To read one table's tuples without unpacking the archive:

import zipfile

with zipfile.ZipFile("lter/tuples.zip") as z:
    with z.open("Tuples/305.csv") as f:
        rows = pd.read_csv(f)

How is this dataset created?

The tables are a sample of 2,015 tables drawn from the top-downloaded collections of the Environmental Data Initiative, a scientific data repository that stores long-term ecological and environmental research data. Taking the most-downloaded collections focuses the sample on frequently used scientific datasets rather than the repository as a whole.

The Polaris authors wrote 15 queries and manually labelled the table–query pairs, giving a binary score of 0 or 1.

Citation

@misc{cai2026polaris,
  title         = {Polaris: Learning to Generate Table Descriptions from Retrieval Feedback},
  author        = {Cai, Ting and Phan, Tuan Minh and Doan, AnHai},
  year          = {2026},
  eprint        = {2608.17171},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  doi           = {10.48550/arXiv.2608.17171},
  url           = {https://arxiv.org/abs/2608.17171}
}

Please also cite the source of the tables:

@misc{edi,
  author = {Paul Hanson},
  title  = {Environmental Data Initiative},
  year   = {2025},
  url    = {https://edirepository.org},
}

License

Contact

Email minhrua@cs.wisc.edu, valid until May 2029. After that, email anhai@cs.wisc.edu.