uy-rrodriguez/FrenchMedMCQA-extended

FrenchMedMCQA-extended: A French Multiple-Choice Question Answering Corpus for Medical domain, that supports Comparative Analysis with Human responses

Dataset Description

• —Paper: Comparative Analysis of Human and Large Language Model Performance in Pharmacology Multiple-Choice Questions
• —Point of Contact: Ricardo Rodriguez

Dataset Summary

This dataset is based on FrenchMedMCQA, the first publicly available Multiple-Choice Question Answering (MCQA) dataset in French for medical domain. We have enriched the content with additional annotations including student response rates downloaded from MedShake.net (the original data source), manually labelled tags about syntactic features and question structure (e.g., negation, question mode), and clinical topics.

Supported Tasks and Leaderboards

Multiple-Choice Question Answering (MCQA)

Languages

French

Data Structure

Data Instances

{
  "id": "230bac49b0fe863b772410bc8d01a025f63c3c999065480131d6334abd2efeff",
  "question": "Parmi les affirmations suivantes, une seule est fausse, indiquer laquelle: les particules alpha",
  "answers": {
      "a": "Sont formées de noyaux d'hélium",
      "b": "Sont peu pénétrantes",
      "c": "Toute l'énergie qu'elles transportent est cédée au long d'un parcours de quelques centimètres dans l'air",
      "d": "Sont arrêtées par une feuille de papier",
      "e": "Sont peu ionisantes"
  },
  "correct_answers": [
      "e"
  ],
  "subject_name": "pharmacie",
  "nbr_correct_answers": 1,
  "medshake": {
      "e": {
          "nb_answer": 398,
          "score": 2.0
      },
      "c": {
          "nb_answer": 67,
          "score": 0.0
      },
      "d": {
          "nb_answer": 67,
          "score": 0.0
      },
      "b": {
          "nb_answer": 45,
          "score": 0.0
      },
      "a": {
          "nb_answer": 41,
          "score": 0.0
      }
  },
  "year": 1995,
  "year_txt": "1995",
  "question_nbr": 25,
  "topics": [
      "biophysique"
  ],
  "naive_difficulty": 0.3559870550161812,
  "shannon_difficulty": 0.7057649927636012,
  "tag_negation": "no",
  "tag_composition": "yes",
  "tag_mode": "instruction",
  "tag_intruder": "yes",
  "tag_answer": "single"
}

Data Fields

All data files follow the same format and are JSON files with the fields described below.

From FrenchMedMCQA:

• —id: Sample unique identifier (random).

• —question: Medical question.

• —answers: Always five possible choices identified with "a" to "e".

• —correct_answers: List of choices composing the correct answer.

• —nbr_correct_answers: Number of correct choices in "correct_answers", possible values are 1 to 5.

New data from MedShake.net:

• —medshake: Combinations of choices given by students, each with the allocated score and number of users that have given such answer. Real answers from humans as of 01/03/2025. Aggregated responses only, responses from a single individual cannot be identified.

Not all combination of choices are available, only those given by at least one student.

The score for an answer is calculated based on the number of errors and ommissions. To infer the score that would be given to a combination not appearing in the dataset, the following logic must be followed:

    score = {
      2 points   <= 0 error or omission
      1 points   <= 1 error or omission
      0.4 points <= 2 errors or omissions
      0 points   <= others
    }

• —year, year_txt, question_nbr: Year the exam took place, in number and text (with optional "nord" and "sud" specifications, given by MedShake.net) and question number.

year_txt and question_nbr can be used to identify the question in the MedShake.net website, e.g.: with year "2006-sud" and number "49" you can reconstruct the URL https://www.medshake.net/pharmacie/concours-internat/annales/qcm/voir/2006-sud/49/

• —topics: List of manually assigned medical topics associated to the question.

• —naive_difficulty: Naive difficulty calculated as the ratio of students having answered the correct response. This value is calclulated from the responses in medshake.

• —shannon_difficulty: Difficulty based on the Shannon entropy, described in the paper and taking into consideration all student responses. This value is calclulated from the responses in medshake.

Given the formula for Shannon entropy:

    H(P) = - ∑ p_i * log(p_i)  i=1..n
             ────────────────
                  log(n)

where n is the number of possible answers and p_i is the proportion of students who chose answer i. Thus, a value of 0 means that the question is obvious for all students (everyone gives the same answer), and a value of 1 means the question is extremely hard (the response rate is equivalent to a random selection).

• —missing: Records without information from MedShake.net will have this additional attribute set to True. Any record with medshake data does not have this attribute at all.

New manually added annotations, describing syntactic features and question structure:

• —tag_negation: Negation, indicates if a negated phrase is present anywhere in the question.

• —tag_composition: Composition required, indicates when the question is a partial sentence and needs to be combined with one of the choices to form a full correct sentence.

E.g.: ”’Crack’ is a form of:”. Choices: (a) heroin; (b) cocaine.

• —tag_intruder: Identification of intruder, indicates whether the question requires the student to identify the choice(s) that do not respect a certain condition.

E.g.: “Which proposition does not apply to norfloxacin?”.

• —tag_mode: Sentence mode, categorises the “question” as a true question, an instruction, or an affirmation.

E.g.:

• —Instruction: “Concerning misoprostol, give its action mechanism.”;
• —Affirmation: “Anaemia is generally observed under the following parasitic infections:”.

• —tag_answer: Explicit number of choices, indicates whether the number of expected answers is explicitly provided (single, multiple, or undefined).

E.g.:

• —Single: “Only one proposition is exact. Serotonine is:”;
• —Multiple: “Which ones of the following propositions apply to IL-2?”;
• —Undefined: “What happens during ventricular systole?”.

Note that about half the records in the train split are missing the new fields described in section "New data from MedShake.net". This is because the additional data was not found for those records.

Data Splits

Data splits follow the the original distribution in FrenchMedMCQA.

Note: Values in parenthesis are the number of samples for which the additional MedShake data exists in train. For all other splits, the 100% of records are annotated.

Dataset Creation

Curation Rationale

The extra information added to the original FrenchMedMCQA dataset supported our work to compare responses to a set of QCM by both humans and LLMs, and explore different approaches at evaluating the characteristics that make a question difficult for models in contrast to humans.

Source Data

The paper associated to FrenchMedMCQA mentions the original data source and gives an URL that seems to have been deprecated since the publication of the article. Most questions available in FrenchMedMCQA have been, however, found in MedShake.net, which leads us to assume that most exams have been transferred to this new URL.

We, authors of FrenchMedMCQA-tagged, ignore the outcome of all those question from FrenchMedMCQA that could not be found in MedShake.net (all those records for which the additional MedShake data, like student answers, is missing in our dataset).

Annotations

Manual annotations to the dataset are described in Section "Dataset fields" above. They describe characteristics such as syntactic features and question structure that could enable further analysis comparing human and model behaviour when answering multiple-choice questions.

Annotation process

A single person in our team has manually annotated all questions, making decisions on the correct tags to apply to the best of their abilities.

We welcome your feedback if any errors or misinterpretations are detected !

Who are the annotators?

Ricardo Rodriguez (LIA, Avignon University)

Personal and Sensitive Information

The dataset does not contain any sensitive information. Student andswers are only given as aggregated counts of answers per combination of choices. A single individual cannot be identified from the available data.

Additional Information

Dataset Curators

• —Ricardo Rodriguez (LIA, Avignon University)
• —Stéphane Huet (LIA, Avignon University)
• —Benoit Favre (LIS, Aix-Marseille Université)
• —Mickael Rouvier (LIA, Avignon Université)

This work was supported by funds from the ANR (French National Agency for Research), for the project ANR MALADES.

Licensing Information

This work is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). See LICENSE.txt for the full text of the deed. To view an online copy of this license, visit https://creativecommons.org/licenses/by/4.0/.

Citation Information

@inproceedings{rrodriguez2025mcqa,
  title={Comparative Analysis of Human and Large Language Model Performance in Pharmacology Multiple-Choice Questions},
  author={Rodriguez, Ricardo and Favre, Benoit and Rouvier, Mickael and Huet, Stephane},
  booktitle={Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing-Natural Language Processing in the Generative AI Era},
  pages={1022--1029},
  year={2025},
  doi={10.26615/978-954-452-098-4-117},
  url={https://doi.org/10.26615/978-954-452-098-4-117}
}

DOI: 10.26615/978-954-452-098-4-117

Contributions

We would like to thank Pierre-Michel Bousquet for his valuable help and advice during this work.