elbonaparte/asclepius_v1
Asclepius
Version: Initial (v1)
Overview
Asclepius is a state-of-the-art Language Model developed with a special focus on healthcare queries. It is as LLama-2-70b based model fine-tuned on a broad range of publicly available healthcare data. It can render timely and consistent answers to health-related queries with good accuracy.
Setup Guide & Usage
The model is available on this huggingface repository. It supports up to 4-bit quantization with sustained performance. To use the model you can use the usual HuggingFace Transformers API, like this:
from transformers import AutoTokenizer
import transformers
import torch
model = "elbonaparte/asclepius_v1"
tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
"text-generation",
model=model,
torch_dtype=torch.float16,
device_map="auto",
)
sequences = pipeline(
'Question: What is the most common cause of chest pain in men < 50? Answer:\n',
do_sample=True,
top_k=10,
num_return_sequences=1,
eos_token_id=tokenizer.eos_token_id,
max_length=200,
)
for seq in sequences:
print(f"Result: {seq['generated_text']}")The output would be:
The most common causes of chest pain in men < 50 are:
1. Gastroesophageal reflux disease (GERD): This occurs when stomach acid flows back into the esophagus, causing a burning sensation in the chest (heartburn).
2. Musculoskeletal issues: Strained muscles or inflammation in the chest wall can cause chest pain, especially during physical activity or after an injury.
3. Anxiety and stress-related chest pain: Anxiety or panic attacks can cause chest tightness and discomfort, mimicking heart-related pain.
4. Costochondritis: Inflammation of the cartilage that connects the ribs to the breastbone can cause sharp chest pain, often aggravated by physical activity or deep breathing.Although this is plain, non-structured text, several prompt strategies can be used to obtain structured text as lists or JSON format from the LLM.
In the asclepius_api_helpers.py we provide several prompt-optimized functions for returning structured responses for the most common use cases: medications list, most/least common findings, diagnostic hypothesis, etc.
These functions can be used to create a server tasked with querying the model (API gateway) or integrated within the application (example provided on https://asclepiusv1.streamlit.app/).
Inference server
Alternatively, a RunPod image template is available pre-configured for text-generation-inference allowing you to get the model up and running in a few clicks following these steps:
- Choose an instance to deploy. This model needs a GPU with at least 48gb of VRAM.
- Choose the provided "Asclepius Template" image template from the dropdown list. It should be the first. Runpod will provision the VM and setup is done automatically. It takes around 15 to 20 minutes to download the model.
- The model server will be available 15 to 20 minutes after that. The URL will look like this:
[POD ID]-80.proxy.runpod.net. The POD ID is the string of text under the Pod Name in the top left.
- A easy-to-use UI for interacting with Asclepius is available over https://asclepiusv1.streamlit.app. Change the Pod ID to the runnning POD ID and it will return the answers from that model.
The UI code is also accessible in this repository. It can be utilized with the installation of streamlit.io and by running this command in the terminal inside the repo folder: streamlit run app.py.
The example app and use cases interact optimally with the model using input prompts engineered within the functions found in the asclepius_api_helpers.py file.
Training Data
An extensive exploration of publicly available healthcare information datasets was conducted and these were considered as potential inputs:
Evaluation
The model evaluation is based on its consistency and accuracy in responding to healthcare inquiries.
- Custom QA dataset (20 questions)
(1-shot) | Model Name | Correct Answers (%) | Total Questions | | :-------------------- | :------------------- | :-------------- | | GPT-4 | 18 (90%) | 20 | | Asclepius | 16 (80%) | 20 | | Llama-2 | 15 (70%) | 20 | | Falcon-40b | 11 (55%) | 20 | | Others (GPT-J, T5, Graph-based) | < 8 | 20 |
- MMLU (Professional Medicine + Clinical Knowledge + College Medicine)
(5-shot) | Model Name | Score | | :-------------------- | :------------------- | | GPT-4 | 88% | | MedPalm-2 | 88% | | Asclepius | 67.2% | | GPT-3.5 | 67.2% | | Llama-2 | 66.8% |
- MedQA-USMLE (1200 USMLE-style questions)
(5-shot) | Model Name | Score | | :-------------------- | :------------------- | | GPT-4 | 86.1% | | MedPalm-2 | 79.7% | | Asclepius | 60.1% | | Llama-2 | 58.9% | | GPT-3.5 | 53.5% |
Limitations
Despite its impressive capabilities in answering medical questions, Asclepius occasionally errs particularly in the section of clinical cases. Consequently, output from the model should not be used without the supervision of a professional healthcare practitioner.
As a tool for physicians, it has immense potential in suggesting diagnoses and prescriptions, in addition to reviewing notes. Nevertheless, potential harm to patients due to erroneous output needs consideration and studies on outcomes should be undertaken before utilization.
This is in agreement with the FDA's recommendation on the usage of AI, ML and Software as a Medical Device (https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device)
License
Asclepius has been developed by Leonardo Canela Almeida, an independent contractor at Phire Health LLC. All rights are reserved by Phire Health LLC.