sapienzanlp/prelearn

Prerequisite RElation LEARNing (PRELEARN)

Original Paper: https://ceur-ws.org/Vol-2765/paper164.pdf

This dataset contains a collection of binary-labelled concept pairs (A,B) extracted from textbooks on four domains: data mining, geometry, physics and precalculus. Then, domain experts were asked to manually annotate if pairs of concepts showed a prerequisite relation or not, therefore the dataset consists of both positive and negative concept pairs.

We obtained the data from the original repository, making only one modification: undersampling the training data, to have a balanced set. To evaluate generative models in in-context learning, it's essential to have a balanced distribution for sampling examples in a few-shot setting. The undersampling process was carried out randomly, and separately for each domain.

Example

Here you can see the structure of the single sample in the present dataset.

{
  "concept_A": string, # text of the concept A
  "wikipedia_passage_concept_A": string, # paragraph of wikipedia corresponding to concept A
  "concept_B": string, # text of the concept B
  "wikipedia_passage_concept_B": string, # paragraph of wikipedia corresponding to concept B
  "target": int, # 0: B non è preconcetto di A, 1: B è preconcetto di A
}

Statitics

Proposed Prompts

Here we will describe the prompt given to the model over which we will compute the perplexity score, as model's answer we will chose the prompt with lower perplexity. Moreover, for each subtask, we define a description that is prepended to the prompts, needed by the model to understand the task.

Description of the task: "Dati due concetti A e B, indica se il primo concetto è un prerequisito per il secondo.\nIl concetto A è prerequisito per il concetto B, se per comprendere B devi prima aver compreso A.\nI seguenti concetti appartengono al dominio: {{domain}}.\n\n"

Cloze Style:

Label (B non è prerequisito di A): "{{conceptB}} non è un prerequisito per {{conceptA}}"

Label (B è prerequisito di A): "{{conceptB}} è un prerequisito per {{conceptA}}"

MCQA Style:

Domanda: il concetto \"{{concept_B}}\" è un prerequisito per la comprensione del concetto \"{{concept_A}}\"? Rispondi sì o no:

Results

The following results are given by the Cloze-style prompting over some english and italian-adapted LLMs.

Aknwoledge

We would like to thank the authors of this resource for publicly releasing such an intriguing benchmark.

Additionally, we extend our gratitude to the students of the MNLP-2024 course, whose first homework explored various interesting prompting strategies.

The original dataset is freely available for download link.

License

The data come under the license Creative Commons Attribution Non Commercial Share Alike 4.0 International