Modelpublic

Aakash1005/sdg-classifier

sourceHugging Facemitupdated 6d agoView on Hugging Face

0likes46downloads

Model Card

sdg-classifier

A 16-class text classifier that assigns a passage to one UN Sustainable Development Goal. Fine-tuned from distilbert-base-uncased on the OSDG Community Dataset.

Built primarily as an investigation into the benchmark itself. Read the limitations section before using the per-class scores to compare against other models — one class in this dataset is separable by writing style rather than topic, which inflates aggregate metrics.

Intended use

—First-pass SDG tagging of policy text, dataset descriptions, and report excerpts, for human review
—A reproducible baseline for OSDG-CD experiments

Out of scope

—Any use where the output is acted on without review
—SDG 17 (Partnerships), which is absent from the training data
—Detecting that a text is not SDG-related — there is no negative class, so the model will assign a goal to any input, including unrelated text
—Short inputs. Training excerpts average ~100 words; performance on titles or single phrases is untested

Results

Test set: 8,605 held-out examples. Training set filtered to agreement ≥ 0.5.

Configuration	Accuracy	Macro F1
3 epochs	0.793	0.762
6 epochs (this model)	0.799	0.772

Per-class (6 epochs):

Class	Precision	Recall	F1	Support
SDG 1 — No Poverty	0.729	0.619	0.670	494
SDG 2 — Zero Hunger	0.777	0.800	0.789	506
SDG 3 — Good Health	0.904	0.861	0.882	526
SDG 4 — Quality Education	0.887	0.829	0.857	736
SDG 5 — Gender Equality	0.814	0.848	0.831	875
SDG 6 — Clean Water	0.764	0.807	0.785	549
SDG 7 — Affordable Energy	0.785	0.838	0.810	610
SDG 8 — Decent Work	0.399	0.532	0.456	284
SDG 9 — Industry & Infrastructure	0.782	0.753	0.767	543
SDG 10 — Reduced Inequalities	0.613	0.648	0.630	457
SDG 11 — Sustainable Cities	0.743	0.793	0.767	497
SDG 12 — Responsible Consumption	0.755	0.687	0.719	233
SDG 13 — Climate Action	0.779	0.726	0.751	452
SDG 14 — Life Below Water	0.871	0.792	0.830	231
SDG 15 — Life on Land	0.851	0.794	0.822	554
SDG 16 — Peace & Justice	0.984	0.985	0.984	1058

Limitations

SDG 16's score is partly an artefact. At 0.984 F1 it is far above every other class, on arguably the framework's most abstract goal. A TF-IDF + logistic regression model trained on only the first eight words of each excerpt reaches 0.739 F1 on SDG 16 against a macro average of 0.327 — eight words carry no topical content, so the class is identifiable from stylistic register alone. Sampled SDG 16 excerpts are overwhelmingly academic law and political-science abstracts ("This article provides a theoretically informed analysis…"). The signal is distributed through the text, not confined to the opening: removing the lead sentence cost SDG 16 only 0.016 F1 while costing other classes 0.05–0.10.

SDG 16 is 12.3% of the test set, so this inflates the aggregate numbers above. Per-class scores on this benchmark are not comparable across goals.

SDG 8 is unreliable. Precision 0.399 — when this model predicts SDG 8, it is wrong roughly 6 times in 10. SDG 1, 8, and 10 form a confusion cluster (1→10 at 0.16, 8→10 at 0.12, 10→8 at 0.11) reflecting genuine conceptual overlap between poverty, decent work, and inequality. Treat SDG 8 predictions as low confidence.

Single-label on a multi-label problem. Real passages often address several goals at once. This model returns one.

No confidence intervals. Single seed, single run.

Training


Base model	`distilbert-base-uncased`
Dataset	`albertmartinez/OSDG`, config `2024-04-01`
Filter	agreement ≥ 0.5
Splits	30,978 train / 3,442 val / 8,605 test
Max length	256 tokens
Epochs	6
Batch size	32
Learning rate	3e-5, 10% warmup
Hardware	1 Kaggle session, T4
Model selection	best val macro-F1

Usage

python

from transformers import pipeline

clf = pipeline("text-classification", model="USERNAME/sdg-classifier")
clf("Rural households gained access to piped drinking water and sanitation.")
# [{'label': 'SDG 6', 'score': 0.59}]

Citation

Dataset: OSDG, UNDP IICPSD SDG AI Lab, & PPMI. OSDG Community Dataset (OSDG-CD). Zenodo. https://doi.org/10.5281/zenodo.5550238 (CC BY 4.0)