CIS5190ml/bert5

1---2language:3- en4metrics:5- accuracy6base_model:7- google-bert/bert-base-uncased8pipeline_tag: text-classification9library_name: transformers10---11 12# PLEASE CHECK ED FOR TOKEN13 14 15 16 17 18 19# Model Evaluation Guide20 21This document provides the necessary instructions to evaluate a pre-trained sequence classification model using a test dataset.22 23## Prerequisites24 25Before running the evaluation pipeline, ensure you have the following installed:26 27- Python 3.7+28- Required Python libraries  29  Install them by running:30 31```bash32pip install transformers datasets evaluate torch33```34 35## Dataset Information36 37The test dataset is hosted on the Hugging Face Hub under the namespace `CIS5190ml/Dataset`. The dataset should have the following structure:38- Column: `title`39- Column: `label`40 41Example entries:42- "Jack Carr's take on the late Tom Clancy..." (label: 0)43- "Feeding America CEO asks community to help..." (label: 0)44- "Trump's campaign rival decides between..." (label: 0)45 46## Model Information47 48The model being evaluated is hosted under the Hugging Face Hub namespace `CIS5190ml/bert4`.49 50## Evaluation Pipeline51 52The complete evaluation pipeline is provided in the file:53**Evaluation_Pipeline.ipynb**54 55This Jupyter Notebook walks you through the following steps:561. Loading the pre-trained model and tokenizer572. Loading and preprocessing the test dataset583. Running predictions on the test data594. Computing the evaluation metric (e.g., accuracy)60 61## Quick Start62 63Clone this repository and navigate to the directory:64 65```bash66git clone <repository-url>67cd <repository-directory>68```69 70Open the Jupyter Notebook:71 72```bash73jupyter notebook Evaluation_Pipeline.ipynb74```75 76Follow the step-by-step instructions in the notebook to evaluate the model.77 78## Code Example79 80Here is an overview of the evaluation pipeline used in the notebook:81 82```python83from transformers import AutoTokenizer, AutoModelForSequenceClassification84from datasets import load_dataset85import evaluate86import torch87from torch.utils.data import DataLoader88 89# Load model and tokenizer90tokenizer = AutoTokenizer.from_pretrained("CIS5190ml/bert5")91model = AutoModelForSequenceClassification.from_pretrained("CIS5190ml/bert5")92 93# Load dataset94ds = load_dataset("CIS5190ml/test_20_rows", split="train")95 96# Preprocessing97def preprocess_function(examples):98    return tokenizer(examples["title"], truncation=True, padding="max_length")99 100encoded_ds = ds.map(preprocess_function, batched=True)101encoded_ds = encoded_ds.remove_columns([col for col in encoded_ds.column_names if col not in ["input_ids", "attention_mask", "label"]])102encoded_ds.set_format("torch")103 104# Create DataLoader105test_loader = DataLoader(encoded_ds, batch_size=8)106 107# Evaluate108accuracy = evaluate.load("accuracy")109model.eval()110 111for batch in test_loader:112    with torch.no_grad():113        outputs = model(input_ids=batch["input_ids"], attention_mask=batch["attention_mask"])114        preds = torch.argmax(outputs.logits, dim=-1)115        accuracy.add_batch(predictions=preds, references=batch["label"])116 117final_accuracy = accuracy.compute()118print("Accuracy:", final_accuracy["accuracy"])119```120 121## Output122 123After running the pipeline, the evaluation metric (e.g., accuracy) will be displayed in the notebook output. Example:124 125```126Accuracy: 0.82127```128 129## Notes130 131* If your dataset or column names differ, update the relevant sections in the notebook.132* To use a different evaluation metric, modify the `evaluate.load()` function in the notebook.133* For any issues or questions, please feel free to reach out.