GAD-Research-Lab/MedicalAI-Light-Weight

1---2license: apache-2.03library_name: transformers4pipeline_tag: image-text-to-text5tags:6  - medical7  - chest-xray8  - radiology9  - onnx10  - clip11  - blip12  - multimodal13  - cpu14base_model:15  - openai/clip-vit-base-patch3216  - emilyalsentzer/Bio_ClinicalBERT17  - Salesforce/blip-image-captioning-base18language:19  - en20---21 22# MedicalAI — Light Weight23 24Chest X-ray analysis that runs on consumer hardware — a laptop CPU, no GPU, no cloud.25 26> **⚠️ Not a medical device.** This is a research and educational project. It is **not** FDA/CE cleared, has not been clinically validated, and must not be used to diagnose, treat, or make any decision about a real patient. Outputs are frequently wrong. See [Limitations](#limitations) — they are substantial and you should read them before using anything here.27 28## What's in this repo29 30> **The X-ray and the symptoms go into one model, not two.** The fusion model is a single network that consumes the radiograph *and* the symptom text together and emits one set of logits — `fusion_full.onnx` is one graph with three inputs (`pixel_values`, `input_ids`, `attention_mask`). There is no separate image classifier and text classifier whose outputs get merged afterwards; the two modalities are fused inside the model, before the classifier head. The BLIP captioner below is a **separate, optional** model that only writes a text description of the image — it takes no symptom input and plays no part in the diagnosis path.31 32| Component | Path | Size | What it does |33|---|---|---|---|34| **Fusion model** (ONNX, end-to-end) — *the main model* | `checkpoints/onnx_full/fusion_full.onnx` | 787 MB | X-ray **and** symptom text → diagnosis logits, in one graph. Runs with `onnxruntime` alone — no PyTorch. |35| Fusion classifier head (PyTorch) | `checkpoints/fusion_model.pth` | 5.4 MB | Trained classifier head only; needs CLIP + Bio_ClinicalBERT at runtime. |36| Fusion classifier head (ONNX) | `checkpoints/onnx/fusion_classifier.onnx` | 4.5 MB | Head-only ONNX; encoders still run in PyTorch. |37| **BLIP X-ray captioner** | `blip-xray-finetuned/` | 896 MB | `Salesforce/blip-image-captioning-base` fine-tuned on IU-Xray reports → radiology-style caption. |38| Default/demo classifier | `models/default/fusion_classifier.onnx` | 1.3 MB | 15 NIH classes, **random weights**. Ships so the app runs before training. Not predictive. |39| Application code | `*.py`, `launch.*`, `config.json` | — | CLI, Gradio web UI, batch predictor, training and ONNX export scripts. |40 41The training dataset is **not** included — see [Data](#data).42 43## Architecture44 45**Fusion (Symptom Check)** — one multimodal classifier over both inputs. Both encoders feed a single shared head, so the prediction is a joint function of the image and the symptoms; neither modality is scored on its own:46 47```48image ──► CLIP ViT-B/32 vision tower ──► visual_projection ──► L2-normalize ──┐49                                                                              ├─► concat ──► MLP classifier ──► logits50symptom text ──► Bio_ClinicalBERT ──► mean-pool last_hidden_state ────────────┘51```52 53Encoders are **frozen**; only the MLP head is trained. `fusion_full.onnx` bakes the whole graph — encoders included — into one file, which is why it is 787 MB.54 55ONNX signature (opset 14, dynamic batch and sequence length):56 57| | Name | Shape | dtype |58|---|---|---|---|59| in | `pixel_values` | `[batch, 3, 224, 224]` | float32 |60| in | `input_ids` | `[batch, seq_len]` | int64 |61| in | `attention_mask` | `[batch, seq_len]` | int64 |62| out | `logits` | `[batch, 3018]` | float32 |63 64Preprocess with `CLIPProcessor` (`openai/clip-vit-base-patch32`) for the image and `AutoTokenizer` (`emilyalsentzer/Bio_ClinicalBERT`) for the text. Class names are in `checkpoints/onnx_full/labels.json`, index-aligned to the logits.65 66**Vision (captioning)** — a separate BLIP model, image-only, loadable with `BlipForConditionalGeneration.from_pretrained`. It does not see the symptoms and does not feed the fusion model; it exists to write a human-readable description alongside the diagnosis.67 68## Usage69 70### ONNX, no PyTorch71 72```python73import json74import numpy as np75import onnxruntime as ort76from PIL import Image77from transformers import CLIPProcessor, AutoTokenizer78from huggingface_hub import hf_hub_download79 80repo = "GAD-Research-Lab/MedicalAI-Light-Weight"81onnx_path = hf_hub_download(repo, "checkpoints/onnx_full/fusion_full.onnx")82labels = json.load(open(hf_hub_download(repo, "checkpoints/onnx_full/labels.json")))83 84clip = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")85tok = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")86 87image = Image.open("xray.jpg").convert("RGB")88pixel_values = clip(images=image, return_tensors="np")["pixel_values"]89text = tok("cough and fever", return_tensors="np", padding="max_length",90           truncation=True, max_length=64)91 92sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])93logits = sess.run(["logits"], {94    "pixel_values": pixel_values.astype(np.float32),95    "input_ids": text["input_ids"].astype(np.int64),96    "attention_mask": text["attention_mask"].astype(np.int64),97})[0]98 99probs = np.exp(logits - logits.max()) / np.exp(logits - logits.max()).sum()100top = probs[0].argmax()101print(labels[top], float(probs[0][top]))102```103 104### BLIP captioning105 106```python107from transformers import BlipProcessor, BlipForConditionalGeneration108from PIL import Image109 110repo = "GAD-Research-Lab/MedicalAI-Light-Weight"111processor = BlipProcessor.from_pretrained(repo, subfolder="blip-xray-finetuned")112model = BlipForConditionalGeneration.from_pretrained(repo, subfolder="blip-xray-finetuned")113 114inputs = processor(Image.open("xray.jpg").convert("RGB"), return_tensors="pt")115print(processor.decode(model.generate(**inputs, max_new_tokens=64)[0],116                       skip_special_tokens=True))117```118 119### Full application120 121```bash122git clone https://huggingface.co/GAD-Research-Lab/MedicalAI-Light-Weight123cd MedicalAI-Light-Weight124pip install -r requirements.txt gradio125python web_ui.py          # http://127.0.0.1:7860126```127 128Or `launch.ps1` (Windows) / `launch.sh` (Linux/macOS) to set up a venv and start the UI in one step. `python run.py` for the interactive CLI, `python batch_predict.py <dir> -o out.csv` for batch.129 130## Training131 132- **Fusion head** — cross-entropy over the label set, 85/15 random train/val split, AdamW, frozen encoders. `python training.py --mode train --epochs 10 --batch_size 8`.133- **BLIP** — fine-tuned on IU-Xray image/report pairs. `python xray_training.py --mode train --epochs 3 --batch_size 4 --max_samples 500`.134 135Sources: IU-Xray (~6,687 rows), NIH Chest X-ray (~3,000 rows), plus 160 synthetic rare-finding rows from `expand_dataset.py` — ~9,847 total.136 137## Limitations138 139Read these. They are not boilerplate.140 141- **The label space is degenerate.** `labels.json` has **3,018 classes**, and most are not diagnoses — they are raw, deduplicated report strings scraped from IU-Xray, e.g. `"findings: . impression: 1. all lines and tubes in stable , xxxx position..."`. Only a handful (`atelectasis`, `cardiomegaly`, `consolidation`, `edema`, `effusion`, `emphysema`, `fibrosis`, …) are clean condition names. With ~9.8k training rows across 3,018 classes there are roughly **3 examples per class**, and the reported "confidence" is a softmax over that space — it is not calibrated and should not be read as a probability of disease. Treat the classifier as a demonstration of the architecture, not as a working diagnostic.142- **No held-out evaluation is published.** Training reports validation *loss* only. There is no accuracy, AUROC, sensitivity/specificity, or per-class breakdown in this repo, and no evaluation on an external site or scanner. Nothing here supports a claim about real-world performance.143- **Encoders are frozen and general-purpose.** CLIP ViT-B/32 was pretrained on web images, not radiographs. Only a small MLP adapts it to this domain.144- **Narrow data.** Two US datasets, adult chest radiographs, frontal views, English free-text reports. Expect degradation on pediatric films, lateral views, other modalities, other populations, and other equipment. Known demographic and label-noise problems in IU-Xray and NIH ChestX-ray14 are inherited wholesale — NIH labels were themselves NLP-mined from reports and are noisy.145- **BLIP captions are fluent, not faithful.** The captioner will produce confident, plausible, well-formed radiology prose for an image it has no ability to read correctly, including for non-X-ray inputs. Fluency here carries no signal about correctness.146- **`models/default/` is random weights** by construction, so the app can start before training. Its predictions are noise.147- **Automation bias is the main risk.** The most likely harm from this repo is a person believing a confident-looking output. Do not put it in front of patients or in any workflow where a wrong answer reaches one.148 149## Data150 151The training data is **not redistributed here** — download it yourself with `python training.py --mode prepare-data`.152 153- **IU-Xray** (Indiana University / Open-i, NLM) — de-identified, public.154- **NIH ChestX-ray14** (NIH Clinical Center) — de-identified, public.155 156Both are de-identified at source; no PHI is contained in this repo. The BLIP model was fine-tuned on IU-Xray report text and can emit dataset artifacts such as `xxxx` anonymization tokens. Check each dataset's own terms before redistributing derivatives.157 158## License159 160Apache-2.0 for the code and the trained weights in this repo. Upstream components carry their own licenses — CLIP (MIT), BLIP (BSD-3-Clause), Bio_ClinicalBERT (MIT) — and the source datasets carry their own terms. The license permits use; it does not make the model safe or fit for clinical use.161 162## Citation163 164```bibtex165@software{medicalai_light_weight,166  title  = {MedicalAI — Light Weight: CPU-friendly chest X-ray analysis},167  author = {GAD Research Lab},168  year   = {2026},169  url    = {https://huggingface.co/GAD-Research-Lab/MedicalAI-Light-Weight}170}171```172