Modelpublic

UMCU/cardioner_medroberta.nl_multilabel

sourceHugging Facegpl-3.0updated 2y agoView on Hugging Face

0likes8.2kdownloads

Model Card

Model Card for Cardioner Medroberta.Nl Multilabel

This a medroberta.nl base model finetuned for span classification. This specific model is the average of the best checkpoints per fold over a ten-fold cross-validation. For this model we used the IOB-tagged. Using the IOB-tagging schema facilitates the aggregation of predictions over sequences.

For the chunking we used paragraph based chunking, and we assumed the maximum context length of the base model, i.e. 512 tokens.

Expected input and output

The input should be a string with Dutch cardio clinical text.

CardioNERMedRoBERTa.nlmultilabel is a muticlass span classification model. The classes that can be predicted are disease, medication, procedure and symptom.

Extracting span classification from CardioNERMedRoBERTa.nlmultilabel

The following script converts a string of <512 tokens to a list of span predictions.

python

from transformers import pipeline

le_pipe = pipeline('ner', 
                    model=model, 
                    tokenizer=model, aggregation_strategy="simple", 
                    device=-1)

named_ents = le_pipe(SOME_TEXT)

To process a string of arbitrary length you can split the string into sentences or paragraphs using e.g. pysbd or spacy(sentencizer) and iteratively parse the list of with the span-classification pipe.

Data description

CardioCCC; Manually annotated parallel-language corpus for the clinical cardiology domain

On a 10-fold cross-validations the multilabel metrics are:

Metric	Mean	Median	Stdev
---	---	---	---
evalf1B-DISEASE	0.782	0.777	0.024
evalf1B-MEDICATION	0.898	0.905	0.036
evalf1B-PROCEDURE	0.788	0.79	0.027
evalf1B-SYMPTOM	0.73	0.73	0.018
evalf1I-DISEASE	0.776	0.781	0.022
evalf1I-MEDICATION	0.8	0.803	0.086
evalf1I-PROCEDURE	0.759	0.757	0.018
evalf1I-SYMPTOM	0.725	0.723	0.017
evalf1O	0.935	0.936	0.005
evalf1macro	0.799	0.799	0.018
evalf1micro	0.884	0.886	0.008
eval_loss	0.095	0.092	0.01
evalprecisionB-DISEASE	0.784	0.774	0.029
evalprecisionB-MEDICATION	0.907	0.917	0.035
evalprecisionB-PROCEDURE	0.791	0.795	0.031
evalprecisionB-SYMPTOM	0.721	0.72	0.017
evalprecisionI-DISEASE	0.79	0.79	0.025
evalprecisionI-MEDICATION	0.835	0.863	0.075
evalprecisionI-PROCEDURE	0.784	0.779	0.023
evalprecisionI-SYMPTOM	0.727	0.72	0.021
evalprecisionO	0.935	0.938	0.009
evalprecisionmacro	0.808	0.81	0.015
evalprecisionmicro	0.888	0.889	0.008
evalraucmacro	0.883	0.885	0.012
evalraucmicro	0.933	0.934	0.005
evalrecallB-DISEASE	0.781	0.785	0.025
evalrecallB-MEDICATION	0.889	0.893	0.039
evalrecallB-PROCEDURE	0.785	0.783	0.025
evalrecallB-SYMPTOM	0.739	0.74	0.023
evalrecallI-DISEASE	0.763	0.774	0.028
evalrecallI-MEDICATION	0.77	0.767	0.103
evalrecallI-PROCEDURE	0.735	0.744	0.025
evalrecallI-SYMPTOM	0.724	0.724	0.032
evalrecallO	0.934	0.934	0.004
evalrecallmacro	0.791	0.795	0.022
evalrecallmicro	0.88	0.883	0.009
evalrocauc_B-DISEASE	0.888	0.889	0.013
evalrocauc_B-MEDICATION	0.944	0.946	0.019
evalrocauc_B-PROCEDURE	0.89	0.889	0.013
evalrocauc_B-SYMPTOM	0.866	0.867	0.011
evalrocauc_I-DISEASE	0.873	0.879	0.014
evalrocauc_I-MEDICATION	0.884	0.883	0.052
evalrocauc_I-PROCEDURE	0.862	0.866	0.013
evalrocauc_I-SYMPTOM	0.85	0.851	0.016
evalrocauc_O	0.887	0.889	0.008

Acknowledgement

This is part of the DT4H project.

Doi and reference

For more details about training/eval and other scripts, see CardioNER github repo. and for more information on the background, see Datatools4Heart Huggingface/Website