CoolFace
Modelpublic

rpudathu/distilbert-dbpedia-14-97acc

sourceHugging Faceupdated 5mo agoView on Hugging Face
0likes20downloads
Model Card

rpudathu/distilbert-dbpedia-14-97acc

Model Description

This model is a fine-tuned version of DistilBERT for multi-class text classification on the DBpedia 14 dataset. It classifies short text passages into one of 14 ontology-based categories such as Company, Artist, Athlete, etc. The model achieves high accuracy (~97%) and is optimized for fast inference with reduced computational cost compared to standard BERT.

  • —Developed by: PUDATHU RAMA KRISHNA
  • —Funded by: Ardentix
  • —Model type: DistilBERT (Transformer-based text classifier)
  • —Language(s): English
  • —License: Apache 2.0
  • —Finetuned from model: distilbert-base-uncased

Training Procedure

Training Data

The model was trained on the DBpedia 14 dataset, which contains Wikipedia-derived structured content categorized into 14 classes.

Preprocessing

  • —Removed noise (HTML tags, special characters)
  • —Lowercased all text
  • —Tokenization using DistilBERT tokenizer
  • —Synthetic data augmentation applied to improve generalization

Training Hyperparameters

  • —Epochs: 3
  • —Batch size: 16
  • —Learning rate: 2e-5
  • —Optimizer: AdamW
  • —Max sequence length: 256
  • —Training regime: fp16 mixed precision
  • —Loss function: CrossEntropyLoss

Training Platform

  • —Environment: Google Colab
  • —Framework: Hugging Face Transformers + PyTorch

Evaluation

Testing Data

  • —Standard DBpedia 14 test split (~70k samples)

Metrics

  • —Accuracy
  • —Precision
  • —Recall
  • —F1 Score

Results

  • —Accuracy: 97.0%
  • —F1 Score (macro): 0.97
  • —Precision (macro): 0.97
  • —Recall (macro): 0.97

Summary

The model performs strongly across all classes with balanced precision and recall, making it suitable for general-purpose topic classification tasks.

Model Examination

  • —Captures semantic meaning effectively using transformer attention
  • —Works well on structured and semi-structured text
  • —Limitations:
  • —Struggles with very short or ambiguous inputs
  • —May not generalize well to out-of-domain data

Environmental Impact

  • —Hardware Type: NVIDIA Tesla T4 GPU
  • —Hours used: ~2 hours
  • —Cloud Provider: Google Colab
  • —Compute Region: Likely US-based
  • —Carbon Emitted: ~0.02 kg CO2eq (estimated)

Technical Specifications

Architecture

  • —DistilBERT (6-layer transformer)
  • —Hidden size: 768
  • —Task: Sequence classification (14 classes)

Software

  • —Python 3.10
  • —PyTorch
  • —Hugging Face Transformers
  • —Datasets library

Usage

python
from transformers import pipeline

classifier = pipeline("text-classification", model="rpudathu/distilbert-dbpedia-14-97acc")

result = classifier("Apple is a technology company based in California.")
print(result)