daniel-jurado/mRoBERTa-fine-tuned-for-job-classification-v1-lite

Job classifier sequence classification model for SIRTAR Framework

This is a fine-tuned model of a BSC-LT/mRoBERTa base model for sequence classification. The expected input can be a job résumé and the output is the predicted job position.

In difference with the previous models, this uses a much more improved version of the same dataset used for training. The previous dataset was wrongly clustered, had naming issues and got more redundant class names. All this is being amended, and its far better version can be found here. Compared to v0, it's able to improve its precision and recall over 4% or 5%, training over only 50,828 samples or 18,777 more than the one used in the previous model. It raises the number of classes from 225 to 260 too.

It's a LITE version because it was trained with less output classes. However, it saved redundant classnames such as a "web developer" and "frontend developer", not existing here (just "frontend developer" given the previous example). This means more improving yet to be done for the dataset. This will get it done at some time.

[The full version](https://huggingface.co/daniel-jurado/mRoBERTa-fine-tuned-for-job-classification-v1) included 65.374 rows of data grouping 469 different job titles. It was a improvement over v0 because precision and recall upgraded over 2% as accuracy fell under 3% using double the data and almost double the output classes. I disadvice you to use the full version and use this lite version until i get to correct the dataset once again. The lite version improves testing set recall upto 6% and precision 4% compared to the full version.

Extended description (in-progress)

This is one piece element for the Master Dissertation's Framework for a AI instructed job advisor for seekers called SIRTAR (Provisional name). Its purpose is to use the information generalized by this model to detect the job a candidate posts by his job résumé.

One key element is the information can be given by the own recruiters posting job, describing you brief and essential pieces of information about each position. This is the key element to use jobposts datasets, in fact these three are used that can be found in Kaggle: Linkedin-jobpost-dataset, Indeed-job-dataset & Jobstreet-job-dataset.

These datasets are not really clean. It features silly job titles with NER stuff that ruins the data distribution. I had to correct this by just removing as most noise as possible on job titles and that improved results training this model. Still have not preprocessed at all as there is still noisy words in some job titles. Aside, i removed bad characters and stranger formats on descriptions but all needless Named Entities such URL addresses. Additional info is on the dataset referred to this model.

However is still is unbalanced and output dimensionality is too high. This model was trained restricting classes over 100 examples (still not enough) and the most frequent classes got capped to 300 maximum samples, so lows the impact of sample starving, making it a less overfit model (by sample starving).

I used mRoBERTa because it promised great results having half of the parameters than the XlM-Roberta, which meaningfully saves VRAM during training. Also, has multilanguage support for various languages in the world, including Spanish together the most used regional languages (Galician, Euskara and Catalonian), adding more possibilities to perform this fine-tuning.

Testing setup

• —GPU: GIGABYTE GeForce RTX 5070 TI WINDFORCE 16GB GDDR7.
• —OS: Windows 11 Build 25H2 Build 26200.8524
• —Python ver.: 3.13.12
• —GPU Software: GeForce 610.47 Driver, CUDA Toolkit 12.9, cuDNN 9.21, cuPTI.
• —Neural Computing Unit framework: Pytorch 2.11.0 (CUDA 12.8 edition)

Usage

The simplest example is using the pipeline() from transformers this way.

from transformers import AutoTokenizer, RobertaForSequenceClassification, pipeline

model_name = "daniel-jurado/mRoBERTa-fine-tuned-for-job-classification-v1-lite"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = RobertaForSequenceClassification.from_pretrained(model_name)

job_guesser = pipeline("text-classification", model=model, tokenizer=tokenizer)

data = [
    "I am skilled in javascript, react, nodejs, and have experience with web development. I have improved usability and performance of web applications, and have a strong understanding of front-end and back-end. Now i've specialized as an ai engineer",
    "I care children. I have experience in early childhood education and have worked as a preschool teacher for several years. I am passionate about creating a nurturing and stimulating environment for young learners.",
    "I am expert at eye surgery. I have performed over 1000 successful surgeries and have a deep understanding of ocular anatomy. I am passionate about helping patients regain their vision and improve their quality of life.",
    "I have participated on the design of several logos and have a strong understanding of color theory and typography. I am proficient in Adobe Creative Suite and have experience with branding and visual identity. I am passionate about creating visually appealing and effective designs.",
    "I dedicate to prevent digital threat intrusion. I have experience in cybersecurity and have worked as a security analyst for several years. I am passionate about protecting organizations from cyber threats and ensuring the safety of their digital assets.",
    "I am a skilled software engineer with experience in developing and maintaining software applications. I have a strong understanding of programming languages such as Python, Java, and C++. I am passionate about creating efficient and scalable software solutions."
]

for text in data:
    result = job_guesser(text)
    print(f"Input: {text}\nPredicted Job: {result[0]['label']}\n")

Hyperparameters

• —Context Length: 512
• —Eval Strategy: Epoch
• —Learning rate: 4 · 10^(-5)
• —Loss function curve: Cosine
• —Batch strategy: Minibatch
• —Minibatch size: 32
• —Training optimizer: AdamW
• —Floating-number format: bfloat16
• —Max. epochs: 30
• —Stop strategy: Early Stopping
• —Early Stopping Threshold: 1 · 10^(-3)
• —Early Stopping Patience: 2
• —Weight-based λ-regularizer penalty: 0.01 (Standard)
• —Warmup step percentage per epoch: 10%
• —Return best model strategy: Validation Loss (val_loss)

Metrics

This used a hold-out of 90%/10%. Adding early stopping turns into 80% for training, 10% for validation & 10% for testing.

Cites

Attribution

This is a scholarly work. I find no reason to restrict this model from its usage. You can freely use and fine-tune this on your way. If you find this project enough interesting, you can either cite my work or my repository. There might be a research paper in the future.

Scholarly BibteX cite (SPANISH)

Not available because Master dissertation's not approved nor defended yet.

Scholarly APA Cite (SPANISH)

Not available because Master dissertation's not approved nor defended yet.