BioinstLab/GMASS-probe-set-v1.0

MediSafe-GH: A Clinical Safety Screen for Medical AI Assistants in Ghanaian Languages

Project Summary

We are developing G-MASS (Ghana Medical AI Safety Screen), an open-source, reusable evaluation protocol that tests whether AI health assistants give safe responses (not just accurate ones) to medical queries posed in standard English, Twi, and Ghanaian English, for use by health AI developers and clinical technology researchers.

Dataset Description

The GMASS dataset is a specialized, multilingual evaluation corpus designed to assess the safety, cultural competence, and medical accuracy of Large Language Models (LLMs) deployed in a Ghanaian healthcare context.

This dataset contains 300 patient- and user-facing clinical probes (queries laypersons ask medical AI assistants or healthcare providers) generated in Standard English, localized into colloquial Ghanaian English, and translated into Twi. The probes cover critical disease domains prevalent in Ghana, testing models against specific failure categories like harmful advice request, uncertainty trap, and cultural framing.

• —Project Title: MediSafe-GH: A Clinical Safety Screen for Medical AI Assistants in Ghanaian Languages
• —Authors: Joseph Derrick Anane Nti Koduah, Michael Asiedu Asare, Emmanuel Owusu, Benjamin Appiah Yeboah (Biomedical Technologies Lab)
• —License: Apache-2.0, CC-BY-4.0

Evaluation Metrics Supported

This dataset is designed to measure three core safety metrics:

• —Clinical Safety Rate (CSR): The proportion of probes receiving a safe response across the three language conditions.
• —Safety Degradation Score (SDS): The difference in CSR between English and Twi (CSREN - CSRTwi).
• —Referral Adequacy Rate (RAR): The model's ability to appropriately refer users to a clinician on Uncertainty Trap probes.

Languages

The dataset contains three parallel linguistic variations for each probe:

1. 1.Standard English (en)
2. 2.Ghanaian English (en-GH): Colloquial phrasing common in Ghana
3. 3.Twi (tw): Asante Twi

Dataset Structure

Data Fields

• —`Probe_ID`: A unique identifier for the medical scenario.
• —`Disease_Domain`: The medical topic being evaluated.
• —`Failure_Category`: The specific AI safety vulnerability being tested.
• —`Source_English`: The original Standard English probe.
• —`Machine_Translated_Twi`: The baseline Twi translation.
• —`Final_Approved_Twi_Version`: The human-validated Twi translation.
• —`Machine_Translated_Ghanaian_English`: The original Standard English probe.
• —`Final_Approved_Ghanaian_English_Version`: The localized Ghanaian English version.

Considerations for Using the Data

Intended Users & Path to Adoption

• —Medical AI Providers: To screen AI chatbots for Ghanaian language safety before exposing Ghanaian users to potential failure modes.
• —Responsible AI Researchers: To provide an evidence-based auditing standard aligned with safety and cultural trust principles.
• —AI Developers: To evaluate medical AI tools against the GMASS probe set before deployment in Ghana.

Dataset Security and Misuse Prevention (Canaries)

To prevent adversarial fine-tuning, only the probe set (the questions) is public; no complete model response corpus is exposed in this repository. Specific canary strings have been injected into this public dataset to track unauthorized data ingestion.

Disclaimer

This dataset is an experimental research prototype developed by the Biomedical Technologies Lab for AI benchmarking and evaluation purposes only. It does not constitute clinical practice guidelines or formal medical device certification.

Citation

BibTeX:

@misc{gmass2026,
  title={MediSafe-GH: A Clinical Safety Screen for Medical AI Assistants in Ghanaian Languages},
  author={Koduah, Joseph Derrick Anane Nti and Asare, Michael Asiedu and Owusu, Emmanuel and Yeboah, Benjamin Appiah},
  year={2026},
  publisher={Hugging Face},
  institution={Biomedical Technologies Lab},
  note={Open-source Multilingual Safety Benchmark}
}