Aleksandruz/Binary-Relevance-RoBERTa

Model Card for JD Relevance Classifier (RoBERTa)

A binary text classifier that labels individual sentences from job descriptions as relevant (1) or not relevant (0) for downstream skill/knowledge extraction. Built as Stage 1 of a multi-stage skill-extraction pipeline.

Model Details

Model Description

This model fine-tunes roberta-base with LoRA adapters for binary sentence classification on job-description text. After training, the LoRA adapter was merged back into the base weights, so the published artifact is a standalone RobertaForSequenceClassification and can be loaded without PEFT.

A decision threshold tuned on a held-out validation split is stored in config.json under decision_threshold. Downstream code can read this value to convert positive-class probabilities into 0/1 labels, or fall back to argmax (0.5) if absent.

• —Developed by: Aleksander Bielinski
• —Funded by: ES/P000681/1; Project Ref: 2839036
• —Model type: Encoder-only transformer for binary sequence classification (RoBERTa)
• —Language(s) (NLP): English
• —License: cc-by-4.0
• —Finetuned from model: roberta-base

Model Sources

• —Repository: Github

Uses

Direct Use

Sentence-level relevance gating for job-description text. Given a single sentence, the model returns either:

• —a [P(not_relevant), P(relevant)] probability distribution, or
• —a 0/1 label using the stored decision_threshold.

Intended to filter JD sentences before downstream skill/knowledge span extraction.

Downstream Use

Designed as Stage 1 of a skill-extraction pipeline: sentences flagged relevant are forwarded to a span-tagging model/direct skill retrieval; sentences flagged not relevant are dropped.

Out-of-Scope Use

• —Non-English text.
• —Text outside the job-description domain (resumes, generic web text, etc.) — the training data was JD-specific and behaviour on other domains is unknown.
• —Hiring decisions about individual candidates. This model judges whether a sentence contains skill/knowledge-relevant content, not whether a person is qualified.
• —Multi-sentence inputs. The model is trained on individual sentences and expects pre-segmented input.

Bias, Risks, and Limitations

• —Domain bias: training data is drawn from job descriptions in specific industries/regions; generalisation to other JD sources may be uneven.
• —Label-definition shift: part of the training corpus consists of span-annotated datasets converted to binary by the rule "contains a skill/knowledge span → relevant." This is a proxy for relevance and may diverge slightly from human relevance judgements on edge cases.
• —Segmentation dependency: the model was trained on sentence-segmented inputs. Performance is sensitive to how sentences are split, especially around bullet points, headers, and run-on lines common in JDs. Train- and inference-time segmentation should match.
• —Class imbalance: training data is approximately 1.73:1 (relevant:not_relevant). Macro-F1 and a tuned decision threshold mitigate this at inference, but the model has seen more positive than negative examples.

Recommendations

• —Use the stored decision_threshold rather than plain argmax for binary decisions.
• —Apply consistent sentence segmentation between training and inference; validate on a small sample of real JDs before deploying.
• —For new JD sources or languages, re-evaluate on a held-out sample before trusting predictions.

How to Get Started with the Model

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch
import torch.nn.functional as F

repo_id = "Aleksandruz/Binary-Relevance-RoBERTa"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSequenceClassification.from_pretrained(repo_id).eval()

sentences = ["Manage a team of engineers.", "Free snacks provided."]
enc = tokenizer(sentences, padding=True, truncation=True,
                max_length=128, return_tensors="pt")

with torch.no_grad():
    probs = F.softmax(model(**enc).logits.float(), dim=-1).numpy()

threshold = getattr(model.config, "decision_threshold", 0.5)
labels = (probs[:, 1] >= threshold).astype(int)

for s, p, y in zip(sentences, probs, labels):
    print(f"[{y}] P(relevant)={p[1]:.3f}  {s}")

Training Details

Training Data

Combined sentence-level dataset built from:

• —A primary annotated corpus of JD sentences with direct binary relevance labels (~17k sentences).
• —Three span-annotated JD datasets (see below) (skills/knowledge spans), converted to binary by the rule: any annotated span present → relevant (1), otherwise not relevant (0). Their train and validation splits were merged into training; test splits were kept separate.
• —SAYFULLINA: https://huggingface.co/datasets/jjzha/sayfullina
• —GREEN: [https://huggingface.co/datasets/jjzha/green
• —SKILLSPAN: [https://huggingface.co/datasets/jjzha/skillspan

Final training set: approximately 30k sentences (train + val combined for the deployed model), with an internal 20% stratified holdout used for early stopping and threshold calibration. Class distribution: ~31% not_relevant, ~69% relevant (ratio 1.73 : 1).

Evaluation used four separate held-out test sets:

• —test_expert — expert-annotated (1.64:1 ratio) (obtained from proprietary project data: data evaluated by labour market expert)
• —test_crossannot — cross-annotated (2.49:1 ratio) (obtained from proprietary project data: data by two student annotators)

Training Procedure

Preprocessing

• —Sentence segmentation: (For segmentation refer to: https://github.com/segment-any-text/wtpsplit toolkit)
• —Rows with empty or whitespace-only sentences dropped.
• —Labels cast to int (0/1).
• —Tokenization: RoBERTa tokenizer, max_length=128, padding to max length, truncation.

Training Hyperparameters

• —Training regime: fp16 mixed precision on CUDA.
• —Architecture: roberta-base + LoRA (target modules: query, value; alpha = 2 * r).
• —Selected via Optuna (TPE sampler, median pruner):
• —learning rate: 0.0004093813608598784
• —LoRA r: 4
• —LoRA dropout: 0.20526990795364705
• —weight decay: 0.04401524937396013
• —warmup ratio: 0.012203823484477884
• —gradient accumulation: 4
• —Fixed:
• —per-device batch size: 8
• —max epochs: 8 (early stopping patience = 2, threshold = 0.001)
• —max sequence length: 128
• —seed: 42
• —Selection metric: macro-F1 on validation (argmax for best-epoch selection; threshold tuned afterwards over [0.05, 0.95]).
• —Decision threshold: 0.54
• —After training, the LoRA adapter was merged back into the base model with peft.PeftModel.merge_and_unload() before saving.

Speeds, Sizes, Times

• —Hardware: RTX 4070 TI SUPER
• —Training time: 0.5h
• —Final artifact size: standalone roberta-base checkpoint (~500 MB).

Evaluation

Testing Data, Factors & Metrics

Testing Data

Four held-out test sets, never seen during training or threshold tuning:

• —test_expert: expert-annotated JD sentences (n ≈ 1,435).
• —test_crossannot: cross-annotated JD sentences (n ≈ 2,472).
• —'GREEN (test)': span-labeled to binary (n = 336)
• —'SKILLSPAN (test)': span-labeled to binary (n = 3570)
• —SAYFULLINA (test)': span-labeled to binary (n = 1852)

Factors

• —Per-class F1 reported separately to surface minority-class behaviour.
• —Results reported at both argmax (0.5) and the tuned decision threshold.

Metrics

• —Accuracy — overall sanity check.
• —Macro-F1 — primary metric; weights both classes equally despite imbalance.
• —Per-class F1 — f1_class0 (notrelevant), `f1class1` (relevant).

Results

NOTE that for SAYFULLINA there are only 2 sentences that does not contain the skill, hence skewed performance.

Summary

The tuned (threshold) model perofrms on par with the standard threshold at 0.5.

Technical Specifications

Model Architecture and Objective

RoBERTa-base encoder with a sequence classification head (2 labels). Trained with cross-entropy loss; LoRA adapters on attention query and value projections during fine-tuning, merged back into base weights before publishing.

Glossary [optional]

• —Relevant sentence: a JD sentence that contains skill or knowledge content meaningful for downstream extraction.
• —Decision threshold: the cutoff on P(relevant) above which a sentence is labelled 1; tuned on validation to maximise macro-F1.

Model Card Authors

Aleksander Bielinski

Model Card Contact

olekbielinski@gmail.com