skt/A.X-Encoder-base

A.X Encoder

<div align="center"> <img src="./assets/A.Xfromscratchlogoko_4x3.png" alt="A.X Logo" width="300"/> </div>

A.X Encoder Highlights

A.X Encoder (pronounced "A dot X") is SKT's document understanding model optimized for Korean-language understanding and enterprise deployment. This lightweight encoder was developed entirely in-house by SKT, encompassing model architecture, data curation, and training, all carried out on SKT’s proprietary supercomputing infrastructure, TITAN. This model utilizes the ModernBERT architecture, which supports flash attention and long-context processing.

• —Longer Context: A.X Encoder supports long-context processing of up to 16,384 tokens.
• —Faster Inference: A.X Encoder achieves up to 3x faster inference speed than earlier models.
• —Superior Korean Language Understanding: A.X Encoder achieves superior performance on diverse Korean NLU tasks.

Core Technologies

A.X Encoder represents an efficient long document understanding model for processing a large-scale corpus, developed end-to-end by SKT.

This model plays a key role in data curation for A.X LLM by serving as a versatile document classifier, identifying features such as educational value, domain category, and difficulty level.

Benchmark Results

Model Inference Speed (Run on an A100 GPU)

<div align="center"> <img src="./assets/speed.png" alt="inference" width="500"/> </div>

Model Performance

<div align="center"> <img src="./assets/performance.png" alt="performance" width="500"/> </div>

🚀 Quickstart

with HuggingFace Transformers

• —transformers>=4.51.0 or the latest version is required to use skt/A.X-Encoder-base

pip install transformers>=4.51.0

⚠️ If your GPU supports it, we recommend using A.X Encoder with Flash Attention 2 to reach the highest efficiency. To do so, install Flash Attention as follows, then use the model as normal:

pip install flash-attn --no-build-isolation

Example Usage

Using AutoModelForMaskedLM:

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "skt/A.X-Encoder-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id, attn_implementation="flash_attention_2", torch_dtype=torch.bfloat16)

text = "한국의 수도는 <mask>."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

# To get predictions for the mask:
masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)
predicted_token = tokenizer.decode(predicted_token_id)
print("Predicted token:", predicted_token)
# Predicted token: 서울

Using a pipeline:

import torch
from transformers import pipeline
from pprint import pprint

pipe = pipeline(
    "fill-mask",
    model="skt/A.X-Encoder-base",
    torch_dtype=torch.bfloat16,
)

input_text = "한국의 수도는 <mask>."
results = pipe(input_text)
pprint(results)
# [{'score': 0.07568359375,
#  'sequence': '한국의 수도는 서울.',
#  'token': 31430,
#  'token_str': '서울'}, ...

License

The A.X Encoder model is licensed under Apache License 2.0.

Citation

@article{SKTAdotXEncoder-base,
  title={A.X Encoder-base},
  author={SKT AI Model Lab},
  year={2025},
  url={https://huggingface.co/skt/A.X-Encoder-base}
}

Contact

• —Business & Partnership Contact: a.x@sk.com