josuediazflores/insignis-eb1a-lora-comparison

Insignis -- EB-1A Petition Intelligence

LoRA fine-tuned models and a terminal CLI for analyzing EB-1A extraordinary ability visa petitions. Three adapters trained on 1,467 curated AAO appellate decisions, plus a full-featured terminal interface for case management, evidence analysis, and AI-powered criteria evaluation grounded in USCIS policy.

  

Pre-publication research artifact. Part of the alphaXiv x marimo paper "How Much Does Base Model Choice Matter?" Not independently evaluated for production legal use. Don't rely on outputs as legal advice.

Author

Josue Diaz Flores

• —GitHub: github.com/josuediazflores
• —LinkedIn: linkedin.com/in/josuediazfl

Models

This repository hosts three LoRA adapters trained on the same EB-1A dataset with identical hyperparameters, so that the only experimental variable is the base model.

josuediazflores/insignis-eb1a-lora-comparison/
├── gemma-4-E4B/         Gemma 4 E4B Instruct + LoRA r16  (best performer)
├── qwen-2.5-7b/         Qwen 2.5 7B Instruct + LoRA r16
├── llama-3.1-8b/        Llama 3.1 8B Instruct + LoRA r16
└── cli/                 Insignis Terminal UI source code

Gemma 4 E4B + LoRA (recommended)

Best overall performer. 64.2% outcome prediction accuracy on 260 held-out AAO cases -- a +55 point lift over the 9.2% base model accuracy. This is the default model used by the Insignis CLI.

Qwen 2.5 7B + LoRA

Strong base model performance (52.3% accuracy without fine-tuning) but LoRA training degraded outcome prediction to 36.5%. Criteria analysis remains competitive.

Llama 3.1 8B + LoRA

Modest +6.1 point lift (15.4% to 21.5%). Lowest validation loss of the three (0.8176), but benchmark accuracy trails Gemma significantly.

Benchmark Results

Evaluated on 260 held-out AAO decisions never seen during training, scoring, or deduplication. Extraction performed independently with Claude Opus.

Outcome Prediction Accuracy

Criteria Identification (Weighted F1)

Training Metrics

Note: Gemma's higher absolute loss is due to differences in loss computation (HF Trainer includes padding tokens in denominator). Qualitative output quality is strong across all three.

Insignis Terminal UI

A full terminal interface for EB-1A petition analysis, built with Ink (React for CLI) and TypeScript.

<!-- Screenshot placeholder: run npm run dev and capture the splash screen -->

Features

• —Case management -- create, browse, and track EB-1A petition cases
• —Evidence upload -- attach and organize supporting documents per criterion
• —AI-powered criteria analysis -- evaluate petitioner evidence against all 10 EB-1A criteria at 8 CFR 204.5(h)(3)
• —Conversational follow-up -- ask questions about analysis results, request deeper dives into specific criteria
• —RAG-grounded policy citations -- responses cite the USCIS Policy Manual directly, not just model knowledge
• —Streaming inference -- token-by-token output from local Ollama or remote endpoints

Quick Start

1. Install Ollama

# macOS
brew install ollama
ollama serve

2. Create the Insignis model with the LoRA adapter

Create a Modelfile in your working directory:

FROM gemma-4-E4B-it
ADAPTER ./gemma-4-E4B

Then build it:

ollama create insignis-eb1a -f Modelfile

3. Run the CLI

cd cli
npm install
npm run dev

The CLI defaults to local Ollama at localhost:11434 with model insignis-eb1a.

Environment Variables

Examples:

# Use the Qwen adapter instead
INSIGNIS_MODEL=insignis-qwen npm run dev

# Point at a remote GPU server
INSIGNIS_API_URL=http://your-gpu-server:8000 npm run dev

Slash Commands

Stack

• —Ink 7 + React 19 for the terminal UI
• —TypeScript strict mode, ESM
• —Ollama as the inference backend
• —RAG over USCIS Policy Manual for grounded citations

Dataset

All three adapters are trained on the same curated dataset.

Pipeline

4,643 raw AAO decisions
  -> quality scoring + filtering
1,467 curated cases
  -> 4 task types x 10 criteria
13,083 ChatML training examples

Details

• —Source: Publicly available AAO EB-1A appellate decisions
• —Curation: Quality scoring rubric applied to each decision; duplicates removed at 0.95 cosine similarity threshold
• —Task types: criterion identification, outcome prediction, evidence summarization, reasoning extraction
• —Criteria covered: All 10 evidentiary criteria at 8 CFR 204.5(h)(3) (awards, membership, published material, judging, original contributions, scholarly articles, exhibitions, leading/critical role, high salary, commercial success)
• —Split: 10,466 train / 1,308 validation / 260 held-out test (independently extracted)
• —Dataset on the Hub: `josuediazflores/aao-eb1a-chatml`

Training

LoRA Configuration

Trained with HuggingFace PEFT. Qwen and Llama used a custom PyTorch + torchxla 2.5 loop on TPU v4-32. Gemma used the HuggingFace Trainer on AMD MI300X due to torchxla incompatibilities with its multimodal architecture.

Loading an Adapter

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

BASE_MODEL = "google/gemma-4-E4B-it"
ADAPTER_SUBFOLDER = "gemma-4-E4B"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
base = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(
    base,
    "josuediazflores/insignis-eb1a-lora-comparison",
    subfolder=ADAPTER_SUBFOLDER,
)
model.eval()

Limitations

• —Domain narrowness. Trained only on EB-1A AAO decisions. Performance on adjacent categories (O-1A, EB-2 NIW) is unknown.
• —Appellate bias. AAO decisions over-represent contested or denied petitions.
• —Not legal advice. Outputs are pattern matches over historical decisions, not attorney judgment.
• —Single seed, single recipe. No rank ablations or recipe sweeps.
• —Base model biases inherited. LoRA at r=16 modifies a small subspace of base model behavior.

License

MIT

Citation

@misc{diazflores2026insignis,
  title  = {How Much Does Base Model Choice Matter? A Three-Way Comparison of LoRA Fine-Tuning on Domain-Specific Legal Reasoning},
  author = {Diaz Flores, Josue},
  year   = {2026},
  note   = {alphaXiv x marimo competition entry},
  url    = {https://github.com/josuediazflores/ProjectGreenlight}
}

Acknowledgements

• —Google TPU Research Cloud grant for v4-32 compute (Qwen + Llama)
• —DigitalOcean AMD MI300X GPU credits (Gemma)
• —Alibaba Cloud / Qwen team, Meta / Llama team, and Google / Gemma team for releasing their base models
• —HuggingFace for hosting the dataset and adapters