Modelpublic

leminhhung0101/R-ViHSDModel

sourceHugging Faceupdated 1mo agoView on Hugging Face

0likes

Model Card

R-ViHSD — ViSoBERT + TF-IDF/LinearSVM Stacking

A Vietnamese hate-speech and text-noise classification pipeline combining:

—ViSoBERT for semantic representation.
—TF-IDF character + word n-grams for robust lexical features.
—LinearSVC models for hate-speech and noise classification.
—Logistic Regression stacking for the final hate-speech prediction.

The pipeline is trained using Out-of-Fold (OOF) stacking to reduce data leakage. After OOF training, the final base models are retrained on the full labeled dataset for inference on new data.

1. Task Definition

For each Vietnamese text sample, the system predicts two outputs.

Hate-Speech Label

text

CLEAN
OFFENSIVE
HATE

Noise Type

text

ORIGINAL
NO_DIACRITICS
TEENCODE
CHAR_REPEAT
PUNCT_NOISE
OBFUSCATION
MIXED

The minimum inference input format is:

csv

id,text
0001,"sample text"
0002,"another sample"

The id column is preserved in the output but is not used as a model feature.

2. Overall Architecture

text

                            ┌─────────────────────────┐
                            │          TEXT           │
                            └────────────┬────────────┘
                                         │
                  ┌──────────────────────┼──────────────────────┐
                  │                      │                      │
                  ▼                      ▼                      ▼
         ┌────────────────┐    ┌──────────────────┐   ┌─────────────────┐
         │    ViSoBERT    │    │ TF-IDF char+word│   │ Surface Features│
         └───────┬────────┘    └────────┬─────────┘   └────────┬────────┘
                 │                      │                      │
       3 class probabilities      ┌─────┴─────┐                │
                                  │           │                │
                                  ▼           ▼                │
                           Hate LinearSVC  Noise LinearSVC     │
                                  │           │                │
                           3 decision     7 decision            │
                              scores         scores             │
                                  │           │                │
                                  │        softmax             │
                                  │           │                │
                                  └─────┬─────┘                │
                                        │                      │
                 ┌──────────────────────┴──────────────────────┘
                 │
                 ▼
       ┌──────────────────────┐
       │ 18 Stacking Features │
       └──────────┬───────────┘
                  │
                  ▼
       ┌──────────────────────┐
       │ Logistic Regression  │
       │     Meta-Model       │
       └──────────┬───────────┘
                  │
                  ▼
       CLEAN / OFFENSIVE / HATE

The final noise_type prediction does not pass through the meta-model. It is obtained directly from:

text

TF-IDF → Noise LinearSVC → argmax

3. Saved Model Structure

After training, the main working directory is expected to contain:

text

rvihsd_stacking_work/
│
├── visobert_full/
│   ├── config.json
│   ├── model.safetensors / pytorch_model.bin
│   ├── tokenizer_config.json
│   ├── tokenizer.json
│   └── ...
│
├── full_vectorizer.joblib
├── full_hate_svm.joblib
├── full_noise_svm.joblib
├── meta_model.joblib
│
├── visobert_oof.npy
├── svm_oof.npy
├── noise_oof_scores.npy
├── visobert_fold*_pred.npy
│
└── submissions/

Required Files for Inference

Only the following files are required for inference:

text

visobert_full/
full_vectorizer.joblib
full_hate_svm.joblib
full_noise_svm.joblib
meta_model.joblib

OOF files are used during training and evaluation but are not required for deployment.

4. ViSoBERT Model

Backbone:

text

uitnlp/visobert

ViSoBERT is fine-tuned for three-class hate-speech classification:

text

0 → CLEAN
1 → OFFENSIVE
2 → HATE

Main training configuration:

Parameter	Value
Backbone	`uitnlp/visobert`
Max sequence length	`128`
Epochs	`3`
Learning rate	`2e-5`
Weight decay	`0.01`
Warmup ratio	`0.08`
Label smoothing	`0.05`
Train batch size	`128`
Evaluation batch size	`64`
Gradient accumulation	`2`
Seed	`42`

During training, ViSoBERT uses weighted cross-entropy with sqrt-balanced class weights to reduce the effect of class imbalance.

The model produces three probabilities:

text

P(CLEAN)
P(OFFENSIVE)
P(HATE)

These three values are passed to the stacking meta-model.

5. TF-IDF Features

The pipeline combines two TF-IDF vectorizers.

Character TF-IDF

python

analyzer="char"
ngram_range=(3, 5)
max_features=150_000
min_df=2
sublinear_tf=True

Word TF-IDF

python

analyzer="word"
ngram_range=(1, 2)
max_features=80_000
min_df=2
sublinear_tf=True

The two sparse matrices are concatenated:

text

Character TF-IDF + Word TF-IDF
              │
              ▼
       Sparse Feature Matrix

Character n-grams are particularly useful for noisy social-media text, including:

—teencode,
—misspellings,
—missing Vietnamese diacritics,
—repeated characters,
—obfuscation,
—punctuation noise.

6. LinearSVC Models

Two independent LinearSVC models are used.

Hate-Speech SVM

Classes:

text

CLEAN
OFFENSIVE
HATE

Configuration:

python

LinearSVC(
    C=2.0,
    class_weight="balanced",
    random_state=42
)

The stacking model uses the three decision_function scores, not calibrated probabilities.

Noise SVM

Classes:

text

ORIGINAL
NO_DIACRITICS
TEENCODE
CHAR_REPEAT
PUNCT_NOISE
OBFUSCATION
MIXED

Configuration:

python

LinearSVC(
    C=2.0,
    class_weight="balanced",
    random_state=42
)

For stacking, the seven decision scores are transformed using:

python

softmax(noise_score, axis=1)

These values are only used as stacking features and should not be interpreted as calibrated probabilities.

7. Surface Features

The pipeline also extracts five handcrafted features:

text

1. log(1 + text length)
2. punctuation ratio
3. Vietnamese-diacritic ratio
4. repeated-character score
5. obfuscation score

The final meta-model input consists of:

text

ViSoBERT probabilities        3
Hate SVM decision scores      3
Noise SVM softmax scores      7
Surface features              5
-------------------------------
Total                        18

The feature order must remain exactly the same during inference.

8. Meta-Model

The final hate-speech meta-model is:

text

StandardScaler
      ↓
LogisticRegression

Configuration:

python

Pipeline([
    ("scale", StandardScaler()),
    ("lr", LogisticRegression(
        C=1.0,
        max_iter=3000,
        class_weight="balanced",
        random_state=42
    ))
])

Input:

text

18 features

Output:

text

CLEAN / OFFENSIVE / HATE

The model is saved as:

text

meta_model.joblib

9. Out-of-Fold Stacking

The meta-model should not be trained on predictions produced by base models that have already seen the same samples.

The training pipeline therefore uses:

python

StratifiedGroupKFold(
    n_splits=5,
    shuffle=True,
    random_state=42
)

Workflow:

text

Fold 1 → train base models on other folds → predict Fold 1
Fold 2 → train base models on other folds → predict Fold 2
...
Fold 5 → train base models on other folds → predict Fold 5

These predictions form the Out-of-Fold feature matrix.

The meta-model is then trained on:

text

ViSoBERT OOF predictions
+
Hate SVM OOF scores
+
Noise SVM OOF scores
+
Surface features
        ↓
Logistic Regression Meta-Model

Grouping based on normalized text can also be used to reduce duplicate or augmentation leakage across folds.

10. Final Training

After OOF features are generated and the meta-model is trained:

1.The TF-IDF vectorizer is fitted again on all labeled data.
2.The hate-speech LinearSVC is trained on all labeled data.
3.The noise LinearSVC is trained on all labeled data.
4.ViSoBERT is fine-tuned on all labeled data.
5.All final models are saved for inference.

If:

python

USE_VALIDATION_FOR_FINAL_TRAIN = True

the final models use:

text

training_set + validation_set

11. Environment Requirements

Recommended installation:

bash

pip install -U \
    "transformers>=4.46" \
    "accelerate>=1.0" \
    "scikit-learn>=1.4" \
    sentencepiece \
    joblib \
    scipy \
    pandas \
    numpy \
    torch

Main dependencies:

text

Python
PyTorch
Transformers
scikit-learn
SciPy
NumPy
Pandas
Joblib
SentencePiece

CUDA GPU support is recommended for ViSoBERT inference but is not required.

12. Loading the Models

Example:

python

from pathlib import Path
import joblib

from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
)

WORK_DIR = Path(
    "/content/drive/MyDrive/rvihsd_stacking_work"
)

tokenizer = AutoTokenizer.from_pretrained(
    WORK_DIR / "visobert_full"
)

visobert = AutoModelForSequenceClassification.from_pretrained(
    WORK_DIR / "visobert_full"
)

vectorizer = joblib.load(
    WORK_DIR / "full_vectorizer.joblib"
)

hate_svm = joblib.load(
    WORK_DIR / "full_hate_svm.joblib"
)

noise_svm = joblib.load(
    WORK_DIR / "full_noise_svm.joblib"
)

meta_model = joblib.load(
    WORK_DIR / "meta_model.joblib"
)

full_vectorizer.joblib contains a custom DualTfidf class. The inference environment must define a compatible DualTfidf class before loading the file with joblib.

13. Inference Flow

For each new text sample:

text

text
 │
 ├── ViSoBERT
 │      └── 3 class probabilities
 │
 ├── TF-IDF
 │      ├── Hate LinearSVC
 │      │      └── 3 decision scores
 │      │
 │      └── Noise LinearSVC
 │             ├── 7 decision scores
 │             └── softmax → 7 stacking features
 │
 └── Surface features
        └── 5 features

The final stacking input is:

python

meta_X = np.hstack([
    visobert_prob,      # 3
    hate_svm_score,     # 3
    noise_soft,         # 7
    surface_features,   # 5
])

The feature dimension must satisfy:

python

assert meta_X.shape[1] == 18

Final hate-speech prediction:

python

hate_pred = meta_model.predict(meta_X)

Final noise prediction:

python

noise_pred = noise_score.argmax(axis=1)

14. Input Format

A new CSV file should contain at least:

csv

id,text
1,"first sentence"
2,"second sentence"
3,"third sentence"

Additional columns may exist, but inference should only depend on:

text

id
text

This prevents accidental use of labels or unrelated metadata.

15. Output Format

Recommended output:

csv

id,pred_label,pred_noise_type
1,CLEAN,ORIGINAL
2,OFFENSIVE,TEENCODE
3,HATE,NO_DIACRITICS

Columns:

Column	Description
`id`	Original sample ID
`pred_label`	`CLEAN`, `OFFENSIVE`, or `HATE`
`pred_noise_type`	One of the seven supported noise classes

16. Label Mapping

Hate-Speech Labels

python

LABELS = [
    "CLEAN",
    "OFFENSIVE",
    "HATE",
]

Mapping:

text

0 → CLEAN
1 → OFFENSIVE
2 → HATE

Noise Labels

python

NOISE_LABELS = [
    "ORIGINAL",
    "NO_DIACRITICS",
    "TEENCODE",
    "CHAR_REPEAT",
    "PUNCT_NOISE",
    "OBFUSCATION",
    "MIXED",
]

Mapping:

text

0 → ORIGINAL
1 → NO_DIACRITICS
2 → TEENCODE
3 → CHAR_REPEAT
4 → PUNCT_NOISE
5 → OBFUSCATION
6 → MIXED

Do not change the class order when using the already-trained models.

17. Recommended Inference Checks

Useful safety checks:

python

assert len(output) == len(test_df)
assert output["id"].is_unique

assert set(
    output["pred_label"]
).issubset(LABELS)

assert set(
    output["pred_noise_type"]
).issubset(NOISE_LABELS)

assert meta_X.shape[1] == 18

If the stacking matrix does not contain exactly 18 features, the inference feature construction no longer matches training.

18. Components That Must Stay Consistent

When using the existing trained models, keep the following unchanged:

—hate-speech label order,
—noise label order,
—MAX_LENGTH = 128,
—tokenizer saved in visobert_full,
—meta_surface_features implementation,
—DualTfidf implementation,
—18-feature stacking order,
—noise-score softmax transformation before the meta-model.

The stacking order must remain:

text

[ViSoBERT: 3]
+
[Hate SVM: 3]
+
[Noise SVM: 7]
+
[Surface Features: 5]

19. Do New Test Sets Require Retraining?

No.

If the following trained artifacts are available:

text

visobert_full/
full_vectorizer.joblib
full_hate_svm.joblib
full_noise_svm.joblib
meta_model.joblib

a new dataset only requires:

text

LOAD MODELS
     ↓
LOAD NEW CSV
     ↓
TF-IDF + SVM INFERENCE
     ↓
ViSoBERT INFERENCE
     ↓
SURFACE FEATURE EXTRACTION
     ↓
STACKING
     ↓
SAVE PREDICTIONS

There is no need to rerun:

text

5-fold OOF training
TF-IDF fitting
SVM training
ViSoBERT fine-tuning
Meta-model training

20. Training Cache

The notebook may use:

python

REUSE_CACHE = True

Typical cache files include:

text

svm_oof.npy
noise_oof_scores.npy
visobert_oof.npy
visobert_fold*_pred.npy

These files are useful for resuming training or reusing OOF predictions.

They are not required for deployment.

21. Main Training Hyperparameters

python

N_FOLDS = 5
SEED = 42

MODEL_NAME = "uitnlp/visobert"
MAX_LENGTH = 128
EPOCHS = 3

TRAIN_BATCH_SIZE = 128
EVAL_BATCH_SIZE = 64
GRAD_ACCUM_STEPS = 2

LEARNING_RATE = 2e-5
WEIGHT_DECAY = 0.01
WARMUP_RATIO = 0.08
LABEL_SMOOTHING = 0.05

CHAR_NGRAM = (3, 5)
WORD_NGRAM = (1, 2)

CHAR_MAX_FEATURES = 150_000
WORD_MAX_FEATURES = 80_000

MIN_DF = 2

SVM_C_HATE = 2.0
SVM_C_NOISE = 2.0

META_C = 1.0

22. Evaluation Metric

Both tasks are evaluated using Macro-F1.

Hate-speech classification:

python

f1_score(
    y_hate,
    hate_pred,
    average="macro"
)

Noise classification:

python

f1_score(
    y_noise,
    noise_pred,
    average="macro"
)

The notebook may also compute a combined score:

text

0.85 × Hate Macro-F1
+
0.15 × Noise Macro-F1

This README intentionally does not report a fixed F1 score because the actual metric depends on the specific training run and cached predictions.

23. Minimal Deployment Package

For deployment on another machine, the project can be organized as:

text

model/
├── visobert_full/
├── full_vectorizer.joblib
├── full_hate_svm.joblib
├── full_noise_svm.joblib
├── meta_model.joblib
├── inference.py
└── README.md

A command-line inference interface may look like:

bash

python inference.py \
    --input new_test.csv \
    --output predictions.csv \
    --model-dir model

24. Notes

—The meta-model predicts only the hate-speech label.
—The noise label is produced directly by the noise LinearSVC.
—LinearSVC.decision_function() values are not probabilities.
—The softmax applied to noise scores is used as a stacking transformation rather than probability calibration.
—Model compatibility depends on preserving the preprocessing and feature-ordering logic used during training.
—When transferring joblib artifacts between environments, compatible versions of Python and scikit-learn are recommended.

25. Summary

The final inference system is:

text

ViSoBERT
   +
TF-IDF Character/Word Features
   +
Hate LinearSVC
   +
Noise LinearSVC
   +
Surface Features
   ↓
Logistic Regression Stacking
   ↓
Final Hate-Speech Prediction

Noise LinearSVC
   ↓
Final Noise-Type Prediction

This architecture combines transformer-based semantic information with sparse lexical features that are robust to noisy Vietnamese social-media text.