Fernandosr85/emscad-employment-scam-detection-adapter

Employment Scam Detection — EMSCAD Adapter

LoRA adapter for detecting employment scams in job advertisements, fine-tuned on Mixtral-8x7B-Instruct-v0.1 via Adaption's AutoScientist platform.

Given a job advertisement, the adapter answers with one word: LEGITIMATE or FRAUDULENT.

### Holdout performance has not been measured The training run converged (final train loss 0.0030, eval loss 0.0243) and the LoRA weights carry a real update, but recall and false-positive rate on the held-out set are not yet known. Do not deploy this model. A false positive removes a real applicant from a real job.

Task

Binary classification at a 4.84% base rate. That base rate is the whole difficulty: always answering LEGITIMATE scores 95.2% accuracy and F1 0.000 on the class that matters. Accuracy is not reported anywhere in this card — precision and recall on the fraudulent class are.

Labels are external and human: EMSCAD's fraudulent ads were annotated by Workable staff through quality-assurance procedures, based on suspicious client activity, false contact information and applicant complaints. The label predates this project and comes from no heuristic in it.

Text-only by design. EMSCAD ships metadata columns that leak the label — the single rule no company logo → fraud scores F1 0.258 on its own, rivalling the published Random Forest baseline that used them (67.3% of fraudulent ads lack a logo against 18.1% of legitimate ones). has_company_logo, has_questions, telecommuting and salary_range are excluded from every prompt and every training example. A model that needs those flags is doing tabular classification on three binary features, not reading an advertisement.

Evaluation results

Training win rate (Adaption internal)

Win rate is the wrong instrument for this task, and the number shows it. The metric asks a judge to prefer one response over another. Both responses here are a single word drawn from a set of two: when both are right, or both wrong, there is nothing to prefer and the judge is scoring noise. 48 against 52 is a coin landing four points off even.

Training convergence

The final loss rules out the obvious failure mode. On a 22%-fraud mixture, a model that always answers LEGITIMATE bottoms out near 0.589; one that predicts the base rate without discriminating bottoms out at 0.527. The observed 0.0030 is 176× below that floor, and eval loss tracked train loss without diverging. The adapter learned to discriminate on its training distribution.

Held-out baselines (this is the comparable table)

Measured on a 210-row holdout, group-split on normalized description, text-only. Precision is derived analytically at the true 4.84% base rate, never measured on the sample — the eval set is fraud-boosted so recall is estimable at all (a natural 150-row sample contains about 7 fraudulent ads), and measuring precision at the boosted 31% rate overstates it by roughly 3.5×.

The headroom survived three attempts to explain it away

The zero-shot F1 of 0.426 is one point on an ROC curve. Two further experiments tested whether that point was an artifact of how the model was asked:

Thresholding does not help and removing the prior does not help. AUROC moved +0.032 between the anchored and unanchored conditions — 0.86 standard errors at this sample size, indistinguishable from zero. The ceiling is not the prompt's framing.

The finding that matters most: a text-only ceiling

The base model reaches recall 0.308 with FPR 0.007 — extremely conservative, near-perfect on what it flags, silent on the rest. Two independent observations suggest this is not simple failure.

1. EMSCAD's fraud label encodes provenance, not text. Several fraudulent ads are scam operators reposting real job advertisements verbatim. Checked against EMSCAD's own metadata:

AECOM and Aker Solutions are real companies and the descriptions are detailed, plausible job specs. The text is legitimate; what is fraudulent is who posted it. A text-only model cannot see that, and neither can a human reading only the ad.

The converse also occurs. One advertisement labelled LEGITIMATE carries a scam operator's handle inline in the description (ayobamijegsonmoney: mid-sentence, amid misspelled role titles). The platform's own regenerated answer called it fraudulent. On that row the gold label is the questionable one.

2. A linear model finds signal the frontier model does not. TF-IDF + SGD reaches recall 0.846 against Claude's 0.308, both text-only. The signal is in the text — but it appears to be largely lexical and distributional (templated scam vocabulary repeated across the 2012–2014 fraudulent class) rather than semantic. A model reading for meaning misses it; a model counting tokens catches it.

The risk this creates for the adapter: it may be learning 2012–2014 lexical artifacts that do not transfer to contemporary scam advertisements. Measuring holdout recall establishes in-distribution performance only. Nothing in this evaluation demonstrates generalization to present-day fraud.

Model details

Read directly from the distributed adapter_config.json and trainer_state.json.

On the module selection. This adapter touches attention projections only. Mixtral is a mixture-of-experts model whose FFN experts (w1/w2/w3) hold much of its parametric knowledge and are left untouched here — a narrower configuration than an all-linear run.

Archive note. The distributed archive is zstd-compressed despite the `.tgz` extension. Extract with tar --use-compress-program=unzstd -xf, not tar -xzf.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "mistralai/Mixtral-8x7B-Instruct-v0.1"
tokenizer = AutoTokenizer.from_pretrained(base)
base_model = AutoModelForCausalLM.from_pretrained(
    base, torch_dtype=torch.float16, device_map="auto",
)
model = PeftModel.from_pretrained(
    base_model, "Fernandosr85/emscad-employment-scam-detection-adapter",
)

# The exact prompt used during training. The stated base rate is part of it:
# removing that sentence moves AUROC 0.767 -> 0.799 but best F1 0.395 -> 0.329.
PROMPT = """You are reviewing a job advertisement submitted to an applicant tracking system, to decide whether it is a legitimate posting or an employment scam.

Employment scams typically aim to harvest personal data or money from applicants. Roughly 5% of postings in this system are fraudulent, so most ads you see are legitimate.

Answer with exactly one word: LEGITIMATE or FRAUDULENT. Nothing else.

JOB ADVERTISEMENT:
{text}"""

ad = (
    "Title: Data Entry Clerk\n\n"
    "Description: Work from home, no experience needed. Earn $500/week "
    "processing simple forms. Immediate start, flexible hours."
)

# CRITICAL: data_format was 'chat'. Wrap as a user turn via the chat template.
# Feeding raw text with tokenizer(prompt) will NOT match the training format.
messages = [{"role": "user", "content": PROMPT.format(text=ad)}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt",
).to(model.device)
outputs = model.generate(inputs, max_new_tokens=8, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

Inference requires GPU VRAM for Mixtral's full 46.7B footprint — sparse activation reduces compute, not memory, so all experts must be resident. 4-bit quantization fits on ~24 GB.

Training dataset

[Fernandosr85/adaption-job-scam-detection](https://huggingface.co/datasets/Fernandosr85/adaption-job-scam-detection)

3,218 prompt/completion pairs derived from EMSCAD:

On the 22% target. Training at the natural 4.10% teaches the model to answer LEGITIMATE; training at 50% teaches it to over-flag and destroys the false-positive rate, which is the property worth protecting. 22% is a hypothesis about that tradeoff, unverified until the holdout is scored.

On hard-negative selection. The first attempt selected them by the vocabulary of the scams the base model missed, and failed instructively: the legitimate baseline was drawn from the 145-ad eval set, a word appearing in 1% of ads has a 23% chance of appearing zero times in 145, and a 1e-3 floor in the lift denominator turned that zero into lift=133x. leveraging and aptitude were promoted to scam signals and 49% of all fraud matched — a hard-case selector selecting half the corpus. Out-of-fold probability replaced it: a fraudulent ad that a linear model trained without it still scores as legitimate is hard by definition.

On the platform's regenerated completions. Adaptive Data produces its own answer for each row rather than transforming the supplied label. On a 47-row sample it reached recall 28.6% against the base model's measured 30.8% — the same operating point, which is what it is designed to do. Training used the original completions, so this does not affect the adapter. It is noted because several of the disagreements are cases where the platform read the text correctly and EMSCAD's label encodes information the text does not carry (see the text-only ceiling above).

Known limitations

• —Holdout recall and FPR are unmeasured. This is the headline limitation, not a footnote.
• —Text-only ceiling. Part of EMSCAD's fraud signal is provenance rather than text (scam operators reposting real advertisements). No text-only model can recover those.
• —Possible lexical overfitting. TF-IDF reaching recall 0.846 suggests much of the recoverable signal is 2012–2014 vocabulary. Generalization to contemporary scams is untested.
• —Vintage. EMSCAD ads predate remote-work normalization; scam patterns have moved on.
• —Screening aid only. Never an automated decision.

Ethics

Automated fraud screening acts on people. A false positive removes a real applicant from a real job, which is why the false-positive rate is reported alongside recall throughout and why the training mixture is calibrated at 22% rather than balanced: a balanced set produces a model that over-flags, and over-flagging has a victim.

The base model's failure mode is worth naming, because an adapter inherits the context it was trained in. Asked for P(scam) as a number, 42.4% of all ads and 27.7% of actual scams receive exactly 0.05 — with no prior stated in the prompt. The model does not express uncertainty as an intermediate value; it returns a default that reads as a confident low probability. At threshold 0.70 it is perfectly precise on what it flags and silent on three quarters of the scams. That is a knowledge-boundary failure rather than a calibration one.

Repositories

Credits

• —Fine-tuning platform: Adaption — AutoScientist & Adaptive Data
• —Challenge: AutoScientist Challenge 2026 — HR category
• —Source dataset: EMSCAD / Real or Fake Job Posting Prediction
• —Source paper: Vidros, Kolias, Kambourakis & Akoglu, Automatic Detection of Online Recruitment Frauds: Characteristics, Methods, and a Public Dataset, Future Internet 9(1), 2017 — MDPI
• —Author: Fernando Rodrigues · Kaggle: fernandosr85 · HuggingFace: Fernandosr85

Disclaimer

Experimental research artifact submitted to the AutoScientist Challenge 2026 (HR category). Holdout performance is unmeasured. Classification outputs are automated, are a screening aid for human review, and must not be used as an automated hiring or rejection decision.