k8-dmi3eva/spiderology

What is it?

• —Dataset for evaluation text-to-sql models based on <a href="https://yale-lily.github.io/spider">Spider dataset</a>.
• —It is divided into 32 groups (testsets) of samples. Each group checks some models ability.
• —For each group there is control group (simplyset) - the same samples, but without considering features.
• —This is very similar to A/B-testing technic.
• —All the simplified samples (from control group) are new.
• —Test samples contains:
• —Examples extracted from Spider (not only from "dev", but also from "train" and "train_other") Be careful, if you use it for training!
• —Examples extracted from Spider, but corrected from ambiguity and errors
• —Manually created (new) examples

Structure

Folders and files

• —[\[all\]](/all) - the hole dataset in one place
• —``all/testset_all.json`` - the samples with features from all testsets
• —``all/simplyset_all.json`` - the samples without features from all testsets
• —``all/spiderology.json`` - the samples with features from all testsets and simplysets
• —``all/tables_all.json`` - the schemes of all tables required for requests
• —``all/databases`` - the SQLite-databases for all requests in testsets and simplysets
• —[\[testsets\]](/testsets) - folder with testsets calledby their name
• —``testsets/X/testset_X.json`` - the samples with feature X. Structure of samples is below
• —``testsets/X/simplyset_X.json`` - the samples from testset without feature X. Structure of samples is the same
• —``testsets/X/tables_X.json`` - the schemes of all tables for testset and simplyset
• —``testsets/X/databases_X.json`` - the SQLite-databases for all requests in testsets and simplysets X
• —``testsets/X/markup_X.json`` - the simplysets and testsets in one table (in the form convenient for comparing)
• —[\[experiments/predictions\]](/experiments/predictions) - folder with testsets calledby their name
• —``/experiments/predictions/spiderology_with_predictions.json`` - all samples with predictions of two considering models

Structure of testsets and simplysets

{
  "id": Unique ID of sample
  "db_id": Name od database, corresponds to folder with DB in the databases_X
  "query": SQL-query
  "query_toks": Tokens of SQL-query
  "query_toks_no_value": Tokens of SQL-query, filter value replaced by token 'value'
  "question": Natural language question  
  "question_toks": Tokens of natural language question 
  "sql": Structured SQL-query (in Spider format
  "tags": List of tags (features an sub-characteristics)
  "source": Variant of sources (see below)
  "parents_id": None for samples in testset or ID of simplified sample for simplification
  "is_simplification": 'true' for simplifications, 'false' for other samples
  "simplifications_tags": None for samples from testset, list of named simplified features for simplyset,
  "predictions": Dictionary with SQL-queries predicted by models (exists only in the folder "experiments"),
  "evaluation": Results of exact_matching, execution and human evaluation (exists only in the folder "experiments")
}

Sources

• —``SIMPLIFIED`` = Simplification of samples
• —``SPIDER_DEV`` = Samples from Spider test set ('dev')
• —``SPIDER_TRAIN`` = Samples from Spider train set ('train')
• —``SPIDER_TRAIN_OTHERS`` = Samples from Spider train set ('train_other')
• —``ADDED`` = Created samples with features (not from Spider)

Test-set description

Here: https://github.com/rology/spiderology?tab=readme-ov-file#test-set-description