Samika16/ferendis-risk-classifier

EU AI Act Risk Classifier

Fine-tuned DistilBERT model that classifies AI system descriptions into EU AI Act risk tiers.

Intended Use

Classifies a natural-language description of an AI system into one of five categories:

• —minimal_risk
• —limited_risk
• —high_risk
• —prohibited_risk
• —review (flagged for human review — description is ambiguous or spans multiple tiers)

Out of scope: This model does not provide legal advice and should not be used as the sole basis for a compliance decision.

Training Data

• —Dataset size: 1,538 labeled examples
• —Labels sourced via a dual-labeling pipeline with human review on flagged conflicts
• —Ambiguous/boundary-testing "review" cases which require human review.

Training Procedure

• —Base model: distilbert-base-uncased
• —Learning rate: 3e-5
• —Epochs: 10 - 20
• —Batch size: 8
• —Max sequence length: 256

Evaluation Results

Per-class F1:

Limitations

• —The "review" class has the lowest F1, the model is least confident distinguishing genuinely ambiguous cases, which is expected since these cases are excluded from training by design.
• —High Risk and Limited Risk are the most frequently confused pair among the four primary tiers, particularly at the boundary between Annex III decision-influencing systems and Article 50 transparency-only systems.

Ethical Considerations / Risks

• —The training labels came from an LLM-assisted first pass, then a decision tree based second pass, then a human reviewer.
• —The results of the model are not intended to provide legal advice

Version History/Release Notes

v1.0 — 2026-08-14

• —Initial release
• —Trained on 1,538 labeled examples across 5 tiers: minimalrisk, limitedrisk, highrisk, prohibitedrisk, review
• —Labels produced via a dual-pipeline: AI-assisted first pass → decision-tree second pass → human review, with conflicts between the two automated passes flagged for review
• —Accuracy: 92.08% · Macro F1: 0.9209
• —Per-class F1: minimalrisk 0.97, limitedrisk 0.93, highrisk 0.91, prohibitedrisk 0.93, review 0.84
• —Known limitation: review has the lowest F1 of the five classes — expected, since ambiguous cases are excluded from confident training signal by design. High Risk / Limited Risk is the most frequently confused pair among the four primary tiers, particularly at the Annex III (decision-influencing) vs. Article 50 (transparency-only) boundary.