Rogendo/JengaNLP-multi-task-distilbert-base-uncased

JengaNLP Multi-Task NLP (3-Task Attention Fusion)

A multi-task NLP model built with the JengaNLP framework that performs fraud detection, sentiment analysis, and call quality scoring simultaneously through a shared encoder with attention-based task fusion. Designed for Kenyan national security and telecommunications applications.

Model Capabilities

This model handles 3 tasks with 8 prediction heads producing 22 total output dimensions in a single forward pass:

Call Quality Sub-Metrics (17 Binary Outputs)

The call quality task evaluates customer service transcripts across 6 quality dimensions:

Architecture

Input Text
    |
    v
[DistilBERT Encoder] ---- 6 layers, 768 hidden, 12 attention heads
    |
    v
[Attention Fusion] ------- task-conditioned attention with residual connections
    |
    +-- [Task 0: Fraud Head] ----------- Linear(768, 2) --> softmax
    +-- [Task 1: Sentiment Head] ------- Linear(768, 3) --> softmax
    +-- [Task 2: QA Scoring 6 Heads] --- 6x Linear(768, 1..5) --> sigmoid

Key design choices:

• —Shared encoder: All 3 tasks share a single DistilBERT encoder, enabling knowledge transfer between fraud patterns, sentiment signals, and call quality indicators
• —Attention fusion: A learned attention mechanism modulates the shared representation per task, allowing each task to attend to different parts of the encoder output while still benefiting from shared features
• —Residual connections: Fusion output is added to the original representation (gateinitvalue=0.5), ensuring stable training and allowing each task to fall back on the base representation
• —Multi-head QA: Call quality uses 6 independent classification heads with different output sizes (1-5 binary outputs each), weighted by importance during training (resolution: 2.0x, listening: 1.5x, hold: 0.5x)

Usage

With JengaNLP Framework (Recommended)

pip install torch transformers pydantic pyyaml huggingface_hub

from huggingface_hub import snapshot_download
from jenga_ai.inference import InferencePipeline

# Download model
model_path = snapshot_download(
    "Rogendo/JengaNLP-multi-task-nlp",
    ignore_patterns=["checkpoints/*", "logs/*"],
)

# Load pipeline
pipeline = InferencePipeline.from_checkpoint(
    model_dir=model_path,
    config_path=f"{model_path}/experiment_config.yaml",
    device="auto",
)

# Run all 3 tasks at once
result = pipeline.predict("Suspicious M-Pesa transaction from unknown account")
print(result.to_json())

# Or run a single task
fraud_result = pipeline.predict(
    "WARNING: Your Safaricom account has been compromised. Send 5000 KES to unlock.",
    task_name="fraud_detection",
)
fraud = fraud_result.task_results["fraud_detection"].heads["fraud"]
print(f"Fraud: {fraud.prediction} (confidence: {fraud.confidence:.1%})")
# Fraud: 1 (confidence: 96.9%)

Batch Inference

texts = [
    "Suspicious M-Pesa notification asking me to send money.",
    "Normal airtime top-up of 100 KES via M-Pesa.",
    "WARNING: Your account has been compromised.",
]

results = pipeline.predict_batch(texts, task_name="fraud_detection", batch_size=32)

for text, result in zip(texts, results):
    fraud = result.task_results["fraud_detection"].heads["fraud"]
    label = "FRAUD" if fraud.prediction == 1 else "LEGIT"
    print(f"[{label} {fraud.confidence:.1%}] {text}")

CLI

# Single text
python -m jenga_ai predict \
    --config experiment_config.yaml \
    --model-dir ./model \
    --text "Suspicious M-Pesa transaction from unknown account" \
    --format report

# Batch from file
python -m jenga_ai predict \
    --config experiment_config.yaml \
    --model-dir ./model \
    --input-file transcripts.jsonl \
    --output predictions.json \
    --batch-size 16

Call Quality Scoring Example

result = pipeline.predict(
    "Hello, welcome to Safaricom customer care. I understand you're having "
    "a billing issue. Let me look into that for you right away. I've found "
    "the discrepancy and corrected your balance. Is there anything else?",
    task_name="call_quality",
)

for head_name, head in result.task_results["call_quality"].heads.items():
    print(f"{head_name:16s} {head.prediction}  (conf: {head.confidence:.2f})")

Output:

opening          {'greeting': True}  (conf: 0.82)
listening        {'acknowledgment': True, 'empathy': True, ...}  (conf: 0.75)
proactiveness    {'initiative': True, 'follow_up': True, 'suggestions': False}  (conf: 0.58)
resolution       {'identified_issue': True, 'provided_solution': True, ...}  (conf: 0.69)
hold             {'asked_permission': False, 'explained_reason': False}  (conf: 0.02)
closing          {'proper_farewell': True}  (conf: 0.52)

Low-Level Usage (Without JengaNLP Framework)

If you only need the raw model weights and want to integrate into your own pipeline:

import torch
import json
from transformers import AutoTokenizer, AutoModel, AutoConfig

# Load components
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
encoder_config = AutoConfig.from_pretrained("./model/encoder_config")

with open("./model/metadata.json") as f:
    metadata = json.load(f)

# Load full state dict
state_dict = torch.load("./model/model.pt", map_location="cpu", weights_only=True)

# Extract encoder weights (keys starting with "encoder.")
encoder_state = {k.replace("encoder.", ""): v for k, v in state_dict.items() if k.startswith("encoder.")}
encoder = AutoModel.from_config(encoder_config)
encoder.load_state_dict(encoder_state)
encoder.eval()

# Run encoder
inputs = tokenizer("Suspicious transaction", return_tensors="pt", padding="max_length",
                    truncation=True, max_length=256)
with torch.no_grad():
    outputs = encoder(**inputs)
    cls_embedding = outputs.last_hidden_state[:, 0]  # [1, 768]

# Extract fraud head weights (task 0, head "fraud")
fraud_weight = state_dict["tasks.0.heads.fraud.1.weight"]  # [2, 768]
fraud_bias = state_dict["tasks.0.heads.fraud.1.bias"]       # [2]

logits = cls_embedding @ fraud_weight.T + fraud_bias
probs = torch.softmax(logits, dim=-1)
print(f"Fraud probability: {probs[0, 1].item():.4f}")

Intended Use

Primary Use Cases

• —M-Pesa Fraud Detection: Classify M-Pesa transaction descriptions as fraudulent or legitimate. Designed for Safaricom and Kenyan mobile money contexts.
• —Customer Sentiment Monitoring: Analyze customer feedback and communications for sentiment polarity (negative / neutral / positive).
• —Call Center Quality Assurance: Score customer service call transcripts across 17 quality sub-metrics in 6 categories, replacing manual QA audits.
• —Multi-Signal Analysis: Run all 3 tasks simultaneously on the same text to get a comprehensive analysis (is this a fraud attempt? what's the sentiment? how good was the agent's response?).

Intended Users

• —Kenyan telecommunications companies (Safaricom, Airtel Kenya)
• —Financial institutions monitoring mobile money transactions
• —Call center operations teams performing quality audits
• —Security analysts processing incident reports
• —NLP researchers working on African language and context models

Downstream Use

The model can be integrated into:

• —Real-time fraud alerting systems
• —Call center dashboards with automated QA scoring
• —Customer feedback analysis pipelines
• —Security operations center (SOC) threat triage workflows
• —Mobile money transaction monitoring platforms

Out-of-Scope Use

• —Not for automated decision-making without human oversight. This model should support human analysts, not replace them. High-stakes fraud decisions require human review.
• —Not for non-Kenyan contexts without retraining. Entity names, transaction patterns, and call center norms are Kenyan-specific.
• —Not for languages other than English. While some Swahili words appear in the training data (M-Pesa, Safaricom, KRA), the model is primarily English.
• —Not for legal evidence. Model outputs are analytical signals, not forensic evidence.
• —Not for surveillance of individuals. The model analyzes text content, not identity.

Bias, Risks, and Limitations

Known Biases

• —Training data imbalance: Fraud detection was trained on only 20 samples (16 train / 4 eval). The model achieves 1.0 F1 on eval but this is likely due to the tiny eval set and potential overfitting. Real-world fraud patterns are far more diverse.
• —Sentiment data: Only 15 samples, with accuracy stuck at 33.3% (random baseline for 3 classes). The sentiment head needs significantly more training data to be production-useful.
• —Call quality data: 4,996 synthetic transcripts. While metrics are strong (0.65-0.97 F1), the synthetic nature means real-world transcripts with noise, code-switching (Swahili-English), and non-standard grammar may perform differently.
• —Geographic bias: All training data reflects Kenyan contexts. The model may not generalize to other East African countries without adaptation.

Risks

• —False positives in fraud detection: Legitimate transactions flagged as fraud can block real users. Always use this model with human review for enforcement actions.
• —False negatives in fraud detection: Sophisticated fraud patterns not in the training data will be missed. This model is one signal among many, not a standalone detector.
• —Over-reliance on QA scores: Call quality scores should augment, not replace, human QA reviewers. Edge cases (cultural nuances, sarcasm, escalation scenarios) may be scored incorrectly.

Recommendations

• —Use fraud detection as a triage signal (flag for review), not an automatic block
• —Retrain with production-scale data before deploying to production
• —Monitor prediction confidence — route low-confidence predictions to human review using the built-in HITL routing (enable_hitl=True)
• —Enable PII redaction (enable_pii=True) when processing real customer data
• —Enable audit logging (enable_audit=True) for compliance and accountability

Training Details

Training Data

Train/eval split: 80/20 random split (seed=42)

All datasets are synthetic, generated to reflect linguistic patterns in Kenyan telecommunications and financial services contexts. They contain English text with occasional Swahili terms and Kenyan-specific entities (M-Pesa, Safaricom, KRA, Kenyan phone numbers).

Training Procedure

Preprocessing

• —Tokenizer: distilbert-base-uncased WordPiece tokenizer
• —Max sequence length: 256 tokens
• —Padding: max_length (padded to 256)
• —Truncation: enabled

Architecture

• —Encoder: DistilBERT (6 layers, 768 hidden, 12 heads) — 66.4M parameters
• —Fusion: Attention fusion with residual connections — 1.2M parameters
• —Task heads: 8 linear heads across 3 tasks — 17K parameters
• —Total: 67.6M parameters (258MB on disk)

Training Hyperparameters

Task Loss Weights

Training Loss Progression

The best checkpoint was selected at epoch 3 based on eval_loss. Training continued to epoch 12 but eval loss increased after epoch 3, indicating overfitting — expected given the small fraud and sentiment datasets.

Speeds, Sizes, Times

Evaluation

Metrics

All metrics are computed on the 20% held-out eval split.

Fraud Detection (binary classification):

Sentiment Analysis (3-class classification):

Call Quality (multi-label binary per head):

Results Summary

• —Fraud detection achieves perfect metrics on the eval set, but this is a very small eval set (4 samples). Production deployment requires evaluation on a larger, more diverse dataset.
• —Sentiment analysis performs at random baseline (33.3% accuracy for 3 classes), indicating the 15-sample dataset is insufficient. This head needs retraining with production data.
• —Call quality shows strong performance across most heads (0.80-0.97 F1), with the "hold" category being the weakest (0.647 F1) due to fewer hold-related examples in the training data.

Model Examination

Attention Fusion

The attention fusion mechanism learns task-specific attention patterns over the shared encoder output. This allows:

• —The fraud head to attend to transaction-related tokens (amounts, account references)
• —The sentiment head to attend to opinion-bearing words
• —The QA heads to attend to conversational flow patterns

The fusion uses a gated residual connection (initialized at 0.5), meaning each task's representation is a learned blend of the task-specific attended output and the original encoder output.

Security Features

When used with the JengaNLP inference framework, the model supports:

• —PII Redaction: Masks Kenyan-specific PII (phone numbers, national IDs, KRA PINs, M-Pesa transaction IDs) before inference
• —Explainability: Token-level importance scores via attention analysis or gradient methods
• —Human-in-the-Loop: Automatic routing of low-confidence predictions to human reviewers based on entropy-based uncertainty estimation
• —Audit Trail: Tamper-evident logging of every inference call with SHA-256 hash chains

Technical Specifications

Model Architecture and Objective

• —Architecture: DistilBERT encoder + attention fusion + multi-task heads
• —Encoder: 6 transformer layers, 768 hidden size, 12 attention heads, 30,522 vocab
• —Fusion: Single-head attention with residual gating
• —Objectives: CrossEntropy (fraud, sentiment) + BCEWithLogits (call quality)

Compute Infrastructure

Hardware

• —Training: CPU (Intel/AMD, standard workstation)
• —Inference: CPU or CUDA GPU

Software

• —PyTorch 2.x
• —Transformers 5.x
• —JengaNLP Framework V2
• —Python 3.11+

Environmental Impact

• —Hardware Type: CPU (standard workstation)
• —Training Time: ~5 minutes
• —Carbon Emitted: Negligible (short training run on CPU)

Citation

@software{jengaNLP2026,
  title = {JengaNLP: Low-Code Multi-Task NLP for African Security Applications},
  author = {Rogendo},
  year = {2026},
  url = {https://huggingface.co/Rogendo/JengaNLP-multi-task-nlp},
}

Model Card Authors

Rogendo

Model Card Contact

For questions, issues, or contributions: GitHub Issues