cruciverb-it/evalita2026

[image]

This repository contains the data release for the [Cruciverb-IT](https://sites.google.com/view/cruciverbit2026) shared task on automatic crossword solving in Italian, as part of the [2026 EVALITA campaign](https://www.evalita.it/campaigns/evalita-2026/). Refer to the task website for more details.

The data from both tasks can be downloaded from the 'Files and versions' tab.

Updates:

• —Minor update to both task*scorer.py in order to convert accented letters to their non-accented counterpart during evaluation
• —# Test data is out!!
• —The test data of both Cruciverb-IT tasks have been published along with the baselines results (see "Baseline scores" sections).

The test data can be found at: first task and second task.

Task 1: Clues Answering

Overall Description

The first task consists of answering clues extracted from Italian crosswords. Specifically, participants are presented with a set of clues C={c₁, c₂, ..., cₙ} and are asked to build a system that for a given clue cᵢ is able to produce one or multiple candidate solutions S={s₁, s₂, ..., sₙ}, possibly containing the correct answer sᵢ. To simulate a more realistic crossword solving scenario and to further guide the systems towards the correct answer space, each clue cᵢ is paired with the character length of the target answer sᵢ. For example: given the clue and the target character length “Sono un fiore di straordinaria bellezza, 4”, the systems should produce a list of one or more candidates, i.e. {iris, rosa, rose, yuzu, fior, ...} eventually containing the correct answer rosa.

Released Data Format

The folder task_1 contains two .csv files: datasets/train.csv and datasets/val.csv. Both files are structured with the following columns: clue, answer, answer_length. These correspond to the clue, the related answer and the number of charcaters of the related answer, respectively.

Scorer and Predicition Format

Together with the datasets, the folder "task1" contains a python file, "task1_scorer.py", that can be run to evaluate the predictions. This scorer will be the same used during the systems evaluation. The script expects two arguments: the path of the predictions and the path of the original dataset. For example:

python task_1_scorer.py val_preds.csv datasets/val.csv

The script will compute the metrics and save a .json file containing the results, for example:

{
    "acc@1": 0.11,
    "acc@10": 0.22,
    "mrr": 0.03,
    "num_examples": 20821
}

Metrics are:

• —acc@1: the accuracy in predicting the correct solution in the top 1 candidate
• —acc@10: the accuracy in predicting the correct solution in the top 10 candidates
• —mrr: Mean Reciprocal Rank, that is the average of the reciprocal ranks of the first correct candidate across the canidate list (Top10)

The predictions must be a .csv file structured exactly as the released datasets with an additional column "candidates" containing a list of one or more predictions for each clue. For example:

clue,answer,answer_length,candidates
Giorni di metà mese nell'antica Roma,idi,3,"['mid', 'poi', 'idi']"
...

Baseline scores

For clues-answering, our baseline is obtained by approaching the task as an information retrieval problem: given a clue cᵢ from the test set Cₜₑₛₜ = {c₁, ..., cₙ}, our system ranks the most similar clues by computing a similarity score between cᵢ and each clue in the training set Cₜᵣₐᵢₙ = {c₁, ..., cₘ}. After selecting the top ten most similar clues, we extract the corresponding ten answers. The similarity scores between clues are estimated using the BM25 algorithm, a well-established ranking function in the field of Information Retrieval. These are the baseline results:

{
    "acc@1": 0.40156572691033093,
    "acc@10": 0.6155323951779453,
    "mrr": 0.46273750250624707,
    "num_examples": 20821
}

Task 2: Grid Filling

Overall Description

The second task consists of autonomously solving Italian crossword grids. The participants are presented with a set of empty crossword grids G={G₁, G₂, ..., Gₖ} where each grid Gᵢ is paired with a list of clues, each one annotated with the coordinates of the square where the corresponding solution starts in the grid, the direction, either down (verticale) or across (orizzontale), and the answer length in characters. A crossword grid consists of a matrix Gᵢ of size Rⁿˣⁿ and each square is either blank or a black square. The developed systems should autonomously fill the grid with the appropriate solutions, yielding a fully or partially filled crossword grid that ensures a consistent overlap between the characters of crossing words and maximizes the number of appropriate solutions correctly placed in the grid.

Released Data Format

The "crosswords_datasets" folder contains the released data for the second task. Each split comes with three files. Importantly, all of them represents each crossword as a single line, that is they all have the same number of lines and each line refer to the same specifc crossword (i.e., the first line of each file is related to the first crossword, the second line of each file to the second crossword and so on):

• —"traingridsempty.txt": the flatted grids layout. Each line in the .txt is an empty crossword grid represented as a matrix, i.e. a list of lists, where each square is either blank (noted as a whitespace ' ') or a black square (noted as a dot '.'). For example:

[[' ', ' ', ' ', ' ', ' '], [' ', '.', ' ', ' ', ' '], [' ', ' ', ' ', ' ', ' '], [' ', ' ', ' ', '.', ' '], [' ', ' ', ' ', ' ', ' ']]
...

• —"traincrossclues.jsonl": a .jsonl file where each line is a list of json dictionaries representing the clues and the related informations for a specific crossword.
• —"clue": the clue
• —"row": row index
• —"col": column index
• —"direction": the direction in which the answer should be placed, either "A" (Across) or "D" (Down)
• —"target": the answer to the clue
• —"lenght": the answer character lenght

For example:

    [{"target": "EVANS", "clue": "Un Bill del jazz", "row": 0, "col": 0, "direction": "A", "length": 5}, {"target": "ONC", "clue": "Sigla dell'Opera Nazionale Combattenti", "row": 1, "col": 2, "direction": "A", "length": 3}, {"target": "CANEE", "clue": "Mute abbaianti", "row": 2, "col": 0, "direction": "A", "length": 5}, {"target": "HEI", "clue": "Un grido di richiamo", "row": 3, "col": 0, "direction": "A", "length": 3}, {"target": "EREDE", "clue": "Un discendente", "row": 4, "col": 0, "direction": "A", "length": 5}, {"target": "ESCHE", "clue": "Le valutano i pescatori", "row": 0, "col": 0, "direction": "D", "length": 5}, {"target": "AER", "clue": "L'aria dei Latini", "row": 2, "col": 1, "direction": "D", "length": 3}, {"target": "AONIE", "clue": "Appellativo delle Muse", "row": 0, "col": 2, "direction": "D", "length": 5}, {"target": "NNE", "clue": "Punto della bussola opposto a SSO", "row": 0, "col": 3, "direction": "D", "length": 3}, {"target": "SCEME", "clue": "Stolte, scarse di cervello", "row": 0, "col": 4, "direction": "D", "length": 5}]

• —"traingridsgold.txt": the solved flatted grids. Each line in the .txt is a correctly solved crossword represented as a matrix, i.e. a list of lists, where each square is either single uppercase character (i.e., 'B') or a black square (noted as a dot '.'). For example:

[['E', 'V', 'A', 'N', 'S'], ['S', '.', 'O', 'N', 'C'], ['C', 'A', 'N', 'E', 'E'], ['H', 'E', 'I', '.', 'M'], ['E', 'R', 'E', 'D', 'E']]
...

Scorer and Predicition Format

Together with the datasets, the folder "task2" contains a python file, "task2_scorer.py", that can be run to evaluate the predicted crossword grids. This scorer will be the same used during the systems evaluation. The script expects three arguments: the path of the predictions, the path of the gold crossword grids and the path of the gold jsonl clues file. For example:

python task_2_scorer.py predictions.txt crosswords_datasets/val_grids_gold.txt crosswords_datasets/val_cross_clues.json 

The script will compute the metrics and save a .json file containing the results, for example:

{
    "char_acc": 0.53,
    "word_acc": 0.44,
    "full_match_accuracy": 0.08,
    "num_examples": 50
}

Metrics are:

• —char_acc: the accuracy in inserting the correct characters in the correct slots
• —word_acc: the accuracy in inserting the correct word in the correct slots
• —fullmatchaccuracy: the accuracy in solving the entire grid. A partially filled grid will be evaluated counting empty squares as errors.

The predictions must be structured the same as a the gold crossword grids file, that is a .txt where each line is a predicted crossword grid represented as a matrix, i.e. a list of lists, where each square is either a single uppercase character (i.e., 'B'), a black square (noted as a dot '.') or None for an unsolved slot in case of a partially solved grid.

. For example:

[['E', None, 'A', 'N', 'S'], ['S', '.', 'O', 'N', 'C'], ['C', 'A', 'N', 'E', 'E'], ['H', 'E', 'I', '.', None], ['E', 'R', 'E', 'D', 'E']]
...

In case of no solutions for a specific crossword grid, i.e. there are no candidates for each clue, the prediction for this crossword can be expressed with an emtpy grid or with a None at the corresponding line in the prediction file.

Baseline scores

For the task of solving crossword grids, our baseline is computed by leveraging the aforementioned task 1 BM25 baseline combined with an additional module (Z3, https://github.com/Z3Prover/z3) that optimizes for a solution by maximizing satisfied constraints while respecting the grid’s hard constraints (more info at: https://sites.google.com/view/cruciverbit2026/task?#h.f39dsde0avgq). Importantly, our baseline approach can yield partially filled grids. We simply run the solver with a candidate size of 10 per clue. These are the baseline results:

{
    "char_acc": 0.7309829317135997,
    "word_acc": 0.5840050445483516,
    "full_match_accuracy": 0.08,
    "num_examples": 50
}