CoolFace
Modelpublic

opendatalab/meta-rater-professionalism-rating

sourceHugging Facemitupdated 1y agoView on Hugging Face
0likes194downloads
Model Card

Professionalism Rating Model

This repository contains the model described in the paper Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models.

Code: https://github.com/opendatalab/Meta-rater

Model Description

This model is a fine-tuned version of ModernBERT-base designed to evaluate the Professionalism dimension of text quality on a 5-point scale (0-5). Professionalism measures the degree of expertise and prerequisite knowledge required to comprehend a text, focusing on the depth, accuracy, and accessibility of content.

Model Details

  • —Base Model: ModernBERT-base
  • —Parameters: 149M
  • —Context Window: 4,096 tokens
  • —Task: Text quality rating (regression)
  • —Score Range: 0-5 (continuous)
  • —Performance: 91.57% F1 score, 93.78% accuracy

Rating Scale

The model uses an additive 5-point rating system:

  • —0: Content requiring no technical knowledge
  • —1: Simple content requiring minimal technical knowledge (nursery rhymes, children's books)
  • —2: Somewhat complex content requiring basic specialized knowledge (popular books, popular science)
  • —3: Moderate complexity requiring some expertise (advanced books, detailed articles)
  • —4: Complex content requiring significant expertise (academic papers, technical reports)
  • —5: Extremely high professionalism requiring advanced subject matter expertise (advanced academic papers, patents)

Usage

python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load the model and tokenizer
model_name = "opendatalab/meta-rater-professionalism-rating"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Example text
text = "The quantum entanglement phenomenon demonstrates non-local correlations between particles..."

# Tokenize and predict
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=4096)
with torch.no_grad():
    outputs = model(**inputs)
    score = outputs.logits.squeeze().argmax(dim=0)

print(f"Professionalism Score: {score:.2f}")

Training Details

  • —Training Data: 747,422 examples from SlimPajama dataset
  • —Annotation Model: Llama-3.3-70B-Instruct
  • —Training Epochs: 10
  • —Evaluation Split: 93,428 test examples
  • —Data Split: 8:1:1 (train:dev:test)

Applications

This model is particularly useful for:

  • —Data curation for language model pre-training
  • —Content filtering based on technical complexity
  • —Educational content difficulty assessment
  • —Research paper and technical document evaluation
  • —Curriculum development and content sequencing

Limitations

  • —The model focuses on content professionalism, not writing style or grammar
  • —Performance may vary across different domains not well-represented in training data
  • —Designed primarily for English text
  • —Should not be used as the sole criterion for content quality assessment

Citation

If you use this model in your research, please cite:

bibtex
@article{zhuang2025meta,
  title={Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models},
  author={Zhuang, Xinlin and Peng, Jiahui and Ma, Ren and Wang, Yinfan and Bai, Tianyi and Wei, Xingjian and Qiu, Jiantao and Zhang, Chi and Qian, Ying and He, Conghui},
  journal={arXiv preprint arXiv:2504.14194},
  year={2025}
}

License

This model is released under the same license as the base ModernBERT model.

Code

The code is available at: https://github.com/opendatalab/Meta-rater

Contact

For questions or issues, please contact the authors or open an issue in the repository.