nikolina-p/intention-request-to-talk-to

Prompt Authenticator (Intent to Talk to Named Person)

Overview

This model is a fine-tuned version of distilbert-base-uncased on the nikolina-p/intention-request-to-talk-to dataset.

It is a binary classifier that detects whether a user prompt expresses intent to engage in a live conversation with a specific named person.

It is designed as a guardrail component in a multi-stage authentication pipeline.

Task

• —1 (Positive): User intends to talk/speak/connect or get live help from a named individual
• —0 (Negative): No such intent (e.g., message relay, references to teams, offices, or unnamed individuals, indirect or future/async communicationinfo requests, non-person targets)

Intended Use

This model is not a final decision-maker, but a first-stage filter in an authentication pipeline. 👉 The classifier is intentionally permissive to avoid false negatives 👉 Final authorization is handled downstream

Limitations

• —Synthetic data may not reflect real-world distribution
• —Struggles with implicit intent and long prompts
• —Mild overconfidence on some edge cases
• —English only

Training

• —Base: distilbert-base-uncased
• —Task: Binary classification
• —Training: Hugging Face Trainer
• —Early stopping enabled (best model ~epoch 4–5)

Training and evaluation data

The dataset is synthetically generated and iteratively refined via error analysis nikolina-p/intention-request-to-talk-to

Covers:

• —direct and indirect requests
• —imperative phrasing
• —assist/help vs message/relay
• —multi-sentence and adversarial cases

Evaluation results

The best model corresponds to the checkpoint from epoch 4 and achieves the following results on the evaluation set: | Loss | Accuracy | Precision (0) | Precision (1) | Recall (0) | Recall (1) | F1 (0) | F1 (1) | Model Select Score | |-------|----------|----------------|----------------|------------|------------|--------|--------|--------------------| | 0.1105 | 0.9826 | 1.0000 | 0.9623 | 0.9688 | 1.0000 | 0.9841 | 0.9808 | 0.9797 |

Training procedure

The classifier is intentionally somewhat permissive to reduce false negatives on genuine request-to-speak prompts. This was achieved mainly through labeling policy and iterative dataset refinement: borderline valid requests were labeled as positive, and common false-negative patterns were added back into training data. Early stopping used F1 on the positive class to keep a reasonable balance between recall and precision.

The best model is selected from checkpoints using:

• —highest F1_1 (primary criterion)
• —lowest evaluation loss (secondary criterion in case of ties or plateau)

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 2e-05
• —trainbatchsize: 16
• —evalbatchsize: 16
• —seed: 42
• —optimizer: Use OptimizerNames.ADAMWTORCHFUSED with betas=(0.9,0.999) and epsilon=1e-08
• —lrschedulertype: cosine
• —num_epochs: 8

Training results

Framework versions

• —Transformers 5.2.0
• —Pytorch 2.10.0+cu128
• —Datasets 4.5.0
• —Tokenizers 0.22.2