itsjhuang/watsonx-docs-type-classifier

Watsonx Docs Document Type Classifier

Binary classifier for IBM Watsonx technical documentation pages. Given a documentation page, the model predicts whether it is:

• —conceptual (0): primarily used to understand or look up information
• —how-to (1): primarily used to complete a procedure or fix a problem

Model Details

Evaluation Results

Three conditions were trained and evaluated. The best model (B) was selected by test macro F1.

Confusion matrices for each condition are available in the repository files.

Usage

import joblib
import numpy as np
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
clf = joblib.load("best_model.joblib")

def softmax(x):
    e = np.exp(x - np.max(x))
    return e / e.sum()

def predict(text):
    embedding = embedder.encode([text], convert_to_numpy=True)
    scores = clf.decision_function(embedding)[0]
    if np.ndim(scores) == 0:
        scores = np.array([-scores, scores])
    probs = softmax(scores)
    labels = ["conceptual", "how-to"]
    return dict(zip(labels, probs))

Limitations

• —Trained on IBM Watsonx documentation only; may not generalize to other technical documentation domains.
• —Label boundary between weak procedural pages and conceptual capability descriptions remains a residual source of error.

Source Dataset

Derived from `ibm-research/watsonxDocsQA`, licensed under Apache 2.0.