nvidia/llama-nemotron-rerank-vl-1b-v2-fp8

Model Overview

Description:

The NVIDIA llama-nemotron-rerank-vl-1b-v2-fp8 is the quantized version of the NVIDIA llama-nemotron-rerank-vl-1b-v2 which is developed for multimodal question-answering retrieval. For more information, please check here. The NVIDIA llama-nemotron-rerank-vl-1b-v2-fp8 model is quantized with TensorRT Model Optimizer.

For this quantization checkpoint we quantize only the language portion of the model, llama-3-1B. All other parts including vision encoder (SigLIP 400M) and the binary classification head are not quantized and are left in default bf-16 or fp-32 precision.

This model is ready for commercial use. <br>

License/Terms of Use:

The use of this model is governed by the NVIDIA Open Model License Agreement and the use of the post-processing scripts are licensed under Apache 2.0. Additional Information: Llama 3.2 Community Model License Agreement. Built with Llama. |

Deployment Geography:

Global <br>

Use Case: <br>

The llama-nemotron-rerank-vl-1b-v2-fp8 is suitable for users who want to build a multimodal question-and-answer application over a large corpus, leveraging the latest dense retrieval technology.

The input of the model is a query combined with a text or image document and the output is a float representing the relevance score (logit) reflecting how strongly it predicts that the passage or document contains the information relevant to the query. Multiple documents could be passed with a single query for batched processing, in this case the output contains a logit per document.

The rerank model is a cross-encoder that supports context in textual format (e.g. the query or the OCR text of a page or a section of a document) or the image of a document page.

Typically, the embedding model is used first to embed (vectorize) the whole corpus (document images or text chunks), and embeddings are stored in a vector database associated to its raw content (image or text). Then at inference time, the embedding model is used to embed the query and pick closest documents based on the cosine similarity of document and query embeddings. Finally, the rerank model is used on the query and top selections from the embedding model to improve the selection document relevance scores with better context information.

Release Date

Hugging Face 07/28/2026 via https://huggingface.co/nvidia/llama-nemotron-rerank-vl-1b-v2-fp8

Model Architecture:

Architecture Type: Transformer<br> Network Architecture: : Eagle VLM architecture with SigLIP 2 400M vision encoder and llama-nemotron-rerank-1b-v2 model as language model. <br>

The llama-nemotron-rerank-vl-1b-v2 is a cross-encoder model with approximately 1.7B parameters. It is a fine-tuned version of an NVIDIA Eagle-family model, which consists of the SigLIP 2 400M vision encoder and the Llama 3.2 1B language model. The final embedding output by the decoder is aggregated using a mean pooling strategy, and a binary classification head is fine-tuned for the ranking task. The CrossEntropy loss is used to maximize the likelihood of (visual) documents containing information to answer the question and minimize the likelihood for (negative) documents that do not contain information to answer the question.

The vision-language model reranker incorporates key innovations from NVIDIA, including Eagle 2 work which uses a tiling-based VLM architecture, and nemoretriever-parse. The Eagle 2 architecture, available on Hugging Face, significantly enhances multimodal understanding through its dynamic tiling and mixture of vision encoders design. It particularly improves performance on tasks that involve high-resolution images and complex visual content.

Number of model parameters:

• —Llama 3.2 1B language model: 1.23 B (Transformer parameters: 973 M, Token embedding parameters: 262 M)
• —SigLip 2 image encoder: 428.77 M

Input(s): <br>

Input Type(s): Image, Text<br>

Input Format(s): <br>

• —Image: Red, Green, Blue (RGB) <br>
• —Text: String <br>

Input Parameters:

• —Image: Two-Dimensional (2D) <br>
• —Text: One-Dimensional (1D) <br>

Other Properties Related to Input: <br>

The model was fine-tuned exclusively on image data, using maxinputtiles = 4 and the maximum context length of 2048 tokens. For evaluation, it was tested on image-only, image+text, and text-only inputs, with maxinputtiles = 6 and the maximum context length of 10240 tokens. Inputs exceeding the maximum length are truncated. <br>

Output(s)

Output Type(s): Floats <br>

Output Format(s): List of Floats <br>

Output Parameters: One-Dimensional (1D) <br>

Other Properties Related to Output: Each value corresponds to a raw logit. Users can choose to apply a Sigmoid activation function to the logits to convert them into probabilities during model usage.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. <br>

vLLM Usage

The model can be served with vLLM for high-throughput reranking. A chat template must be provided via --chat-template to correctly apply the question: / passage: prefixes based on the message role — without it, the prompt template is not applied and results will be incorrect.

See the vLLM documentation for full details.

Online Serving

Create the score template file and start the server:

cat > nemotron-rerank-vl.jinja << 'JINJA'
{%- set query_msg = (messages | selectattr("role", "equalto", "query") | list | first) -%} 
{%- set doc_msg   = (messages | selectattr("role", "equalto", "document") | list | first) -%} 
{%- set q = query_msg["content"] -%} 
{%- set d = doc_msg["content"] -%} 
{# If the doc contains <image> anywhere, hoist a single <image> to the front #} 
{%- set has_image = ("<image>" in d) -%} 
{%- set d_clean = d | replace("<image>", "") -%} 
{%- set q_clean = q | replace("<image>", "") -%} 
{%- if has_image -%}<image>{{ " " }}{%- endif -%} 
question:{{ q_clean }}{{ " " }} 
{{ " " }} 
{{ " " }}passage:{{ d_clean }} 
JINJA

vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2-fp8 \
  --runner pooling \
  --trust-remote-code \
  --max-model-len 10240 \
  --chat-template nemotron-rerank-vl.jinja

Note: Use --max-model-len 10240 to support all modalities including image+text. A smaller value like 2048 can be used if only processing image-only or text-only inputs.

The chat template uses the message role to combine the correct prefixes: set role to "query" for queries (prepends question:) and "document" for passages (prepends passage:).

Send a rerank request:

import requests

url = "http://localhost:8000/v1/rerank"

# Text-only reranking
response = requests.post(url, json={
    "model": "nvidia/llama-nemotron-rerank-vl-1b-v2-fp8",
    "query": "How is AI improving the intelligence and capabilities of robots?",
    "documents": [
        "AI enables robots to perceive, plan, and act autonomously.",
        "A biological foundation model designed to analyze DNA, RNA, and protein sequences.",
    ],
})
print(response.json())

# Image document reranking
response = requests.post(url, json={
    "model": "nvidia/llama-nemotron-rerank-vl-1b-v2-fp8",
    "query": "How is AI improving the intelligence and capabilities of robots?",
    "documents": [
        {"content": [{"type": "image_url", "image_url": {"url": "https://example.com/page.png"}}]},
        {"content": [
            {"type": "text", "text": "AI enables robots to perceive, plan, and act autonomously."},
            {"type": "image_url", "image_url": {"url": "https://example.com/page2.png"}},
        ]},
    ],
})
print(response.json())

Offline / In-Process

from vllm import LLM

SCORE_TEMPLATE = """\
{%- set query_msg = (messages | selectattr('role', 'equalto', 'query') | list | first) -%}
{%- set doc_msg   = (messages | selectattr('role', 'equalto', 'document') | list | first) -%}
{%- set q = query_msg['content'] -%}
{%- set d = doc_msg['content'] -%}
{%- set has_image = ("<image>" in d) -%}
{%- set d_clean = d | replace("<image>", "") -%}
{%- set q_clean = q | replace("<image>", "") -%}
{%- if has_image -%}<image>{{ " " }}{%- endif -%}
question:{{ q_clean }}{{ " " }}
{{ " " }}
{{ " " }}passage:{{ d_clean }}"""

llm = LLM(
    model="nvidia/llama-nemotron-rerank-vl-1b-v2-fp8",
    runner="pooling",
    max_model_len=10240,
    trust_remote_code=True,
)

query = "How is AI improving the intelligence and capabilities of robots?"
documents = [
    "AI enables robots to perceive, plan, and act autonomously.",
    "A biological foundation model designed to analyze DNA, RNA, and protein sequences.",
]

# Text-only scoring
outputs = llm.score(query, documents, chat_template=SCORE_TEMPLATE)
for doc, output in zip(documents, outputs):
    print(f"Score: {output.outputs.score:.4f} | {doc}")

# Image document scoring
outputs = llm.score(
    query,
    [
        {"content": [{"type": "image_url", "image_url": {"url": "https://example.com/page.png"}}]},
        {"content": [
            {"type": "text", "text": "AI enables robots to perceive, plan, and act autonomously."},
            {"type": "image_url", "image_url": {"url": "https://example.com/page2.png"}},
        ]},
    ],
    chat_template=SCORE_TEMPLATE,
)
for output in outputs:
    print(f"Score: {output.outputs.score:.4f}")

Software Integration:

Runtime Engine(s): vLLM<br> Supported Hardware Microarchitecture Compatibility: NVIDIA Blackwell, NVIDIA Hopper, NVIDIA Lovelace <br> Preferred/Supported Operating System(s): Linux<br>

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment. <br>

Model Version(s):

llama-nemotron-rerank-vl-1b-v2-fp8<br> The model is quantized with nvidia-modelopt v0.43.0

Post Training Quantization:

This model was obtained by quantizing the weights and activations of nvidia/llama-nemotron-rerank-vl-1b-v2 to FP8 data type, ready for inference with vLLM

Training, Testing, and Evaluation Datasets:

Dataset Overview

This checkpoint is an FP8 post-training-quantized derivative of nvidia/llama-nemotron-rerank-vl-1b-v2. No additional supervised training or fine-tuning data was used to create this FP8 checkpoint.

For this FP8 release, data was used for two purposes: post-training quantization calibration and evaluation. The cnn_dailymail dataset was used for FP8 quantization calibration.

The FP8 checkpoint was produced by applying post-training quantization to the BF16 parent model using NVIDIA TensorRT Model Optimizer. Calibration samples were converted into model inputs and used to estimate quantization parameters for weights and activations. This calibration process did not involve supervised training, fine-tuning, or modification of the model objective.

Calibration dataset

• —Link: cnn_dailymail Data Collection Method by dataset <br>
• —Automated <br>

Labeling Method by dataset <br>

• —Automated <br>

Properties: <br>

• —The cnn_dailymail dataset is an English-language dataset containing just over 300k unique news articles as written by journalists at CNN and the Daily Mail. 512 samples from the train split were used for calibration. <br>

Training Dataset

Data Modality:

• —Image <br>

Training Data Size:

Image Training Data Size <br>

• —Undisclosed <br>

Data Collection Method by dataset <br>

• —Not Applicable <br>

Labeling Method by dataset <br>

• —Not Applicable <br>

Properties: <br>

• —Not Applicable <br>

Evaluation Dataset

In this section, we compare the performance of quantized model llama-nemotron-rerank-vl-1b-v2-fp8 with baseline implementation llama-nemotron-rerank-vl-1b-v2. We applied the ranking model to the candidates retrieved from the llama-nemotron-embed-vl-1b-v2 model.

ViDoRe V3, KoViDoRe, and ZhViDoRe (an internal Chinese visual document) retrieval benchmark were used to evaluate ranking quality relative to the BF16 baseline model. Evaluation inputs were formatted as text-only, image-only, and image-plus-text examples to compare retrieval accuracy between the FP8 checkpoint and the BF16 baseline.

We evaluate the embedding + reranking pipeline on a set of evaluation benchmarks. Both the baseline BF16 and the quantized FP8 checkpoints were provided with the same candidates retrieved from llama-nemotron-embed-vl-1b-v2 BF16 model. Top 20 candidates were selected based on cosine similarity of embeddings, then rerank model was used to score relevancy of those top-20 documents and compute recall of top 5 scoring documents.

In each configuration, the embedding modality matched the reranking modality (e.g., text-only reranking used text-only embeddings, image-only reranking used image-only embeddings, and image-plus-text reranking used image-plus-text embeddings).

<table> <caption><b><em>FP8 model accuracy relative to baseline BF16 model, on Visual Document Retrieval benchmarks. Chinese: ZhViDoRe (internal); Korean: KoViDoRe; English/French: ViDoRe V3.</em></b></caption> <thead> <tr> <th></th> <th colspan="3">Dataset</th> </tr></thead> <tbody> <tr> <td>Modality</td> <td>All</td> <td>Chinese/Korean</td> <td>English/French</td> </tr> <tr> <td>image+text</td> <td>99.53%</td> <td>100.28%</td> <td>99.34%</td> </tr> <tr> <td>image</td> <td>99.68%</td> <td> - </td> <td>99.68%</td> </tr> <tr> <td>text</td> <td>99.03%</td> <td>98.88%</td> <td>99.07%</td> </tr> </tbody> </table>

Note that the model does not have Chinese/Korean language support for image-only inputs.

Data Collection Method by dataset <br>

• —Hybrid: Automated, Human, Synthetic <br>

Labeling Method by dataset: <br>

• —Hybrid: Automated, Human, Synthetic <br>

Properties <br>

• —ZhViDoRe comprises 922 queries, KoViDoRe comprises 706 queries, and ViDoRe V3 comprises 14,514 queries. More details on ViDoRe benchmarks can be found on their Hugging Face page.

Inference

Acceleration Engine: vLLM <br> Test Hardware: <br>

• —NVIDIA Hopper (H100 SXM) <br>

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

Please make sure you have proper rights and permissions for all input image content; if image includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included. <br>

For more detailed information on ethical considerations for this model, please see the Explainability, Bias, Safety & Security, and Privacy sections.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

Citation

@inproceedings{moreira2025_nvretriever,
author = {Moreira, Gabriel de Souza P. and Osmulski, Radek and Xu, Mengyao and Ak, Ronay and Schifferer, Benedikt and Oldridge, Even},
title = {Improving Text Embedding Models with Positive-aware Hard-negative Mining},
year = {2025},
isbn = {9798400720406},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3746252.3761254},
doi = {10.1145/3746252.3761254},
pages = {2169–2178},
numpages = {10},
keywords = {contrastive learning, distillation, embedding models, hard-negative mining, rag, text retrieval, transformers},
location = {Seoul, Republic of Korea},
series = {CIKM '25}
}

Bias

Explainability

Privacy

Safety & Security