CoolFace
Modelpublic

opendatalab/meta-rater-cleanliness-rating

sourceHugging Facemitupdated 1y agoView on Hugging Face
0likes79downloads
Model Card

Cleanliness Rating Model

This repository contains the model described in the paper Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models.

Code: https://github.com/opendatalab/Meta-rater

Model Description

This model is a fine-tuned version of ModernBERT-base designed to evaluate the Cleanliness dimension of text quality on a 5-point scale (0-5). Cleanliness measures how well-formatted, complete, and noise-free a text is, focusing on structural integrity rather than semantic content.

Model Details

  • —Base Model: ModernBERT-base
  • —Parameters: 149M
  • —Context Window: 4,096 tokens
  • —Task: Text quality rating (regression)
  • —Score Range: 0-5 (continuous)
  • —Performance: 87.88% F1 score, 92.25% accuracy

Rating Scale

The model uses a 5-point rating system based on four key criteria:

  • —0: Absolute noisy content.
  • —1: Serious formatting/structural issues that significantly affect fluency
  • —2: Obvious problems that noticeably affect reading fluency
  • —3: Some problems present but don't seriously impact reading fluency
  • —4: Minor issues that don't affect overall readability
  • —5: Perfect formatting and structure across all criteria

Evaluation Criteria

The model assesses text across four main dimensions:

1. Correct Formatting

  • —Text appears human-edited rather than machine-extracted
  • —No inappropriate or corrupted characters
  • —Proper text structure and layout

2. Appropriate Content

  • —No irrelevant links, advertisements, or spam
  • —Sufficient content length to extract clear structure and theme
  • —Content focused on the main topic

3. Completeness Content

  • —Complete sentences written naturally by humans
  • —Coherent opinions, facts, or stories rather than fragments
  • —Proper article structure and flow

Note: Text ending with $TRUNCATED$ is considered a manual ending flag and doesn't affect completeness scoring.

Usage

python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load the model and tokenizer
model_name = "opendatalab/meta-rater-cleanliness-rating"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Example text
text = "This is a well-formatted article about renewable energy. It contains complete sentences and proper structure."

# Tokenize and predict
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=4096)
with torch.no_grad():
    outputs = model(**inputs)
    score = outputs.logits.squeeze().argmax(dim=0)

print(f"Cleanliness Score: {score:.2f}")

Training Details

  • —Training Data: 747,422 examples from SlimPajama dataset
  • —Annotation Model: Llama-3.3-70B-Instruct
  • —Training Epochs: 10
  • —Evaluation Split: 93,428 test examples
  • —Data Split: 8:1:1 (train:dev:test)

Applications

This model is particularly useful for:

  • —Web scraping quality control and content filtering
  • —Data preprocessing for machine learning datasets
  • —Content management systems for automated quality checks
  • —Data curation for language model pre-training
  • —Document digitization quality assessment
  • —Automated content moderation and filtering

Common Issues Detected

The model can identify various types of text quality problems:

  • —Formatting artifacts from web scraping or OCR
  • —Incomplete sentences or fragmented text
  • —Excessive links or promotional content
  • —Corrupted characters or encoding issues
  • —Poor structure with inadequate content organization
  • —Advertisement contamination and irrelevant insertions

What the Model Does NOT Consider

  • —The specific language the text is written in
  • —The length of the text
  • —Usage of placeholders for data privacy or safety
  • —Content topic, professionalism, or semantic meaning
  • —Writing style or grammatical sophistication

Use Cases by Score Range

  • —4.0-5.0: High-quality content suitable for training data
  • —3.0-3.9: Acceptable content with minor cleaning needed
  • —2.0-2.9: Moderate issues requiring preprocessing
  • —1.0-1.9: Significant problems, may need extensive cleaning
  • —0.0-0.9: Poor quality, likely unsuitable for most applications

Limitations

  • —Designed primarily for English text
  • —May not detect all domain-specific formatting conventions
  • —Performance may vary for highly technical formats (code, mathematical notation)
  • —Should be used in conjunction with other quality metrics for comprehensive assessment

Citation

If you use this model in your research, please cite:

bibtex
@article{zhuang2025meta,
  title={Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models},
  author={Zhuang, Xinlin and Peng, Jiahui and Ma, Ren and Wang, Yinfan and Bai, Tianyi and Wei, Xingjian and Qiu, Jiantao and Zhang, Chi and Qian, Ying and He, Conghui},
  journal={arXiv preprint arXiv:2504.14194},
  year={2025}
}

You can find more details about Meta-rater at https://github.com/opendatalab/Meta-rater.

License

This model is released under the same license as the base ModernBERT model.

Contact

For questions or issues, please contact the authors or open an issue in the repository.