WilliamRabuel/roberta-student-fine-tuned

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

roberta-student-fined-tunned

This model is a fine-tuned version of roberta-base on a dataset provided by Kim Taeuk (김태욱), NLP teacher at Hanyang University.

The model was trained for multi-intent detection using the BlendX dataset, focusing on complex utterances containing multiple intents.

It achieves the following results on the evaluation set:

• —Loss: 0.0053
• —Exact Match Accuracy: 0.9075

Model description

The model is based on roberta-base, a robust transformer model pretrained on a large corpus of English text.

Fine-tuning was conducted on a specialized dataset focusing on multi-intent detection in utterances with complex intent structures.

Model Architecture

• —Base Model: roberta-base
• —Task: Multi-Intent Detection
• —Languages: English

Strengths

High accuracy on evaluation data.

Capable of detecting multiple intents within a single utterance.

Limitations

Fine-tuned on a specific dataset; performance may vary on other tasks.

Limited to English text.

Intended uses & limitations

Use Cases

Multi-intent detection tasks such as customer service queries, virtual assistants, and dialogue systems.

Academic research and educational projects.

Limitations

May require additional fine-tuning for domain-specific applications.

Not designed for multilingual tasks.

Training and evaluation data

The model was trained on the BlendX dataset, a multi-intent detection benchmark focusing on realistic combinations of user intents in task-oriented dialogues.

Data Details:

The dataset used for training this model is based on the BlendX dataset, focusing on multi-intent detection in task-oriented dialogues. While the actual BlendX dataset covers instances that can have varying number of intents between 1 to 3, the dataset for this assignment only includes instances where there are 2 intents for simplicity.

Dataset License and Source

The dataset used for training this model is licensed under the [GNU General Public License v2](https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html).

Important Notes:

• —Any use, distribution, or modification of this dataset must comply with the terms of the GPL v2 license.
• —The dataset source and its original license can be found in its official GitHub repository.
• —Dataset File: Download Here

Dataset Format:

• —File Type: JSON
• —Size: 28,815 training samples, 1,513 validation samples
• —Data Fields:
• —split (string): Indicates if the sample belongs to the training or validation set.
• —utterance (string): The text input containing multiple intents.
• —intent (list of strings): The associated intents.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 1e-05
• —trainbatchsize: 32
• —evalbatchsize: 32
• —seed: 42
• —optimizer: Use OptimizerNames.ADAMWTORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizerargs=No additional optimizer arguments
• —lrschedulertype: cosinewithrestarts
• —warmup_steps: 200
• —num_epochs: 20
• —savetotallimit: 3
• —weight_decay: 0.01
• —eval_strategy: epoch
• —save_strategy: epoch
• —metricforbestmodel: evalexactmatchaccuracy
• —loadbestmodelatend: True
• —dataloaderpinmemory: True
• —fp16: False
• —greaterisbetter: True

Training results

Framework versions

• —Transformers 4.47.0
• —Pytorch 2.5.1+cu124
• —Datasets 3.2.0
• —Tokenizers 0.21.0

Improvement Perspectives

To achieve better results, several improvement strategies could be explored:

• —Model Capacity Expansion: Test larger models like roberta-large or other bigger models.
• —Batch Size Increase: Use larger batches for more stable updates.
• —Gradient accumulation steps parameter: Play with the number of updates steps to accumulate the gradients for, before performing a backward/update pass.
• —Learning Rate Management:
• —Experiment with strategies like polynomial or others, with dynamic adjustment.
• —Further reduce the learning rate
• —Enhanced Preprocessing:
• —Test data augmentation techniques such as random masking or synonym replacement.
• —Further reduce the gap between the different categories.
• —Change the weights according to the representativeness of the category.
• —Use another dataset.
• —Longer Training Duration: Increase the number of epochs and refine stopping criteria for more precise convergence.
• —Model Ensembling: Use multiple models to improve prediction robustness.
• —Advanced Attention Mechanisms: Test models using hierarchical attention or enhanced multi-head architectures.
• —Metric: Choosing the best metric based on our problem.

These strategies require significant computational resources and extended training time but offer substantial potential for performance improvement.