NetherlandsForensicInstitute/Multilingual-ASMBERT

Model Description

The ASMTransformers model is a machine learning model that calculates embeddings of assembly functions to enable comparison and similarity operations. Instruction set architectures (ISAs) ARM, AMD64, RISC-V and i386 assembly are supported. The aim is to aid in the process of reverse engineering.

Status

Not actively maintained after publishing

Relevant links

• —ASMtransformers on GitHub

Version

2026-08-06

Usage

To use this model, use inference.py in our repository.

Intended use

The model is trained and tested to be used for similarity search of assembly code. It is not trained/tested on any other languages than ARM, AMD64, RISC-V or i386, nor is it tested on other downstream tasks.

Architecture description

The model is a Bert-base architecture with 12 layers, 12 attention heads and a hidden size of 768. It uses a vocabulary of size 6161 derived from a custom assembler architecture-aware tokenizer.

The model is trained in two phases - a masked language modeling (MLM) pretraining phase and a fine-tuning phase for function similarity.

The MLM pre-training phase uses a loss function that includes jump target prediction (see Wang et al.). This pretraining is perfomed using a masking probability of 0.4 that was determined through ablation experiments. This is in line with the findings of Wettig et al., 2023.

We perform the finetuning phase using Batched Semi Hard Triplet Loss on batches of size 1024. The training set for this phase includes triplets that are formed using versions of the same function compiled for different instruction set architectures.

Estimated training time:

pretraining: 38 hours on 4 NVIDIA H200s finetuning: 4 hours on 1 NVIDIA H200

Full architecture:

  [
    {
      "idx": 0,
      "name": "0",
      "path": "0_ASMTransformerModule",
      "type": "asmtransformers.models.asmsentencebert.ASMTransformerModule"
    },
    {
      "idx": 1,
      "name": "1",
      "path": "1_Pooling",
      "type": "sentence_transformers.sentence_transformer.modules.pooling.Pooling"
    }
  ]

  {
    "architectures": [
      "BertForMaskedLM"
    ],
    "attention_probs_dropout_prob": 0.1,
    "classifier_dropout": null,
    "hidden_act": "gelu",
    "hidden_dropout_prob": 0.1,
    "hidden_size": 768,
    "initializer_range": 0.02,
    "intermediate_size": 3072,
    "layer_norm_eps": 1e-12,
    "max_position_embeddings": 512,
    "model_type": "bert",
    "num_attention_heads": 12,
    "num_hidden_layers": 12,
    "pad_token_id": 0,
    "position_embedding_type": "absolute",
    "torch_dtype": "float32",
    "transformers_version": "4.12.5",
    "type_vocab_size": 2,
    "use_cache": true,
    "vocab_size": 6161
  }

Pre-training parameters used:

  {
    "epochs": 19, 
    "eval_steps": 10000, 
    "batch_size": 512, 
    "gradient_accumulation_steps": 1, 
    "mlm_prob": 0.4, 
    "bf16": True, 
    "tf32": True
  }

Output

The model outputs embeddings of size 768.

Data

The dataset is derived from the official Debian Repository. We used the distribution's build and packaging system apt to cross-compile source packages for multiple ISAs and optimisation levels. We used the standard optimisation levels supported by the distribution's compiler, O0, O1, O2, O3 and Os. We also added an optimization level that includes vectorization and cryptographic instructions that we refer to as Optimized-custom, or Oc. The binary files from each source package are disassembled using Ghidra and the resulting control-flow graphs (CFGs) are exported.

This results in a maximum of 24 (6 optimisation levels and 4 ISAs) potential CFGs which are semantically identical, i.e. they represent the same functionality. However, because not all source packages can be built for all ISAs and because some optimisation levels result in identical code, the actual number is often lower than that.

The dataset is split into a train, test and an evaluation set. This in done on source package basis, so all binaries and functions belonging to one source package are part of either the training set or the test set, not both.

Total amount of functions per architecture

Amount of functions per architecture per optimization

Preprocessing

Several preprocessing steps were taken. The CFGs are processed into a flat token list using ISA aware tokenizers. Immediate operands and addresses have been normalized to the nearest power-of-two to reduce token explosion caused by raw numeric values. Finally, target addresses of jumps, conditional branches or call instructions are normalised to function-relative JUMP_ADDR_* tokens.

Performance

We report two performance metrics: Mean Reciprocal Rank (MRR) and Accuracy@1.

These metrics are suitable for evaluating tasks where a single example is retrieved from a large collection. Accuracy@1 shows how many times the positive example ranked first. It is a useful metric, but it's also very strict, which is why we used it combined with MRR.