dkhursen/InternVL2-2b-LoRA-25k-drivelm

InternVL2-2B — LoRA fine-tuned on DriveLM (25k custom i.i.d. split)

Part of the Master's thesis:

Visual Question Answering for Autonomous Driving Dmytro Khursenko, Czech Technical University in Prague, Faculty of Electrical Engineering, 2026. Supervised by Ing. David Hurych, Ph.D. (Valeo) and doc. Georgios Tolias, Ph.D. (CTU FEE).

[GitHub](https://github.com/dmitrykhursen/VQA-AD-CTU) · [Demo](https://dmitrykhursen.github.io/VQA-AD-CTU/) · [Pseudo-label dataset](https://huggingface.co/datasets/dkhursen/drivelm-pseudo-labels)

📄 Thesis PDF expected by end of June 2026, following successful defense (CTU FEE).

This is the distributional alignment baseline: InternVL2-2B fine-tuned with LoRA on the custom 25k i.i.d. DriveLM-nuScenes split — without any pseudo-label augmentation. It isolates the effect of distributional alignment alone.

Key finding: distributional alignment matters more than data volume. This model uses 12× fewer training pairs than LoRA-300k yet outperforms it by 6.7 points on Final Score (0.560 vs. 0.493).

This checkpoint contains fully merged weights — the LoRA adapter is already merged into the base model. Load and use it directly with AutoModel.

All released models

Pseudo-label dataset: dkhursen/drivelm-pseudo-labels

Results on custom test split

Evaluated on the custom i.i.d. DriveLM-nuScenes test split (3,340 QA pairs).

Score definitions (all table values normalised to [0, 1]) - Accuracy — exact-match on MCQ (A/B/C/D) and Yes/No questions; strict letter-only format required - Language — mean of BLEU-1–4, ROUGE-L, and CIDEr/10 - ChatGPT — GPT-3.5-turbo semantic similarity judge (0–100 scale, ÷100 in Final) - Match — (F1_coord × 100 + GPT_match) / 2 on prediction answers (0–100 scale, ÷100 in Final); blends spatial coordinate F1 with a GPT judge - Coord — pure coordinate token F1 at L1 < 16 px; diagnostic only, not included in Final - Final — 0.4 × (GPT/100) + 0.2 × Language + 0.2 × (Match/100) + 0.2 × Accuracy Full metric breakdown: evaluation/README.md

Note on evaluation scope. All scores are measured on the custom i.i.d. local test split (3,340 QA pairs) using the local DriveLM evaluation script. The official DriveLM evaluation server was not used as the primary reporting source: it returns only aggregate scores without per-metric breakdown, the infrastructure was intermittently unreliable (model output parsing errors could not be diagnosed directly — only via GitHub issues), and ChatGPT/Match metrics depend on OpenAI API calls that fail when the API quota for the billing period is exhausted.

Full comparison (all models, online inference):

† Mini-DA = OpenGVLab/Mini-InternVL2-2B-DA-DriveLM, fine-tuned by OpenGVLab on DriveLM. DL-PL 30%, 50%, 100% variants not publicly released (lower performance without quality filtering).

Training data

Fine-tuned on the custom DriveLM-nuScenes split (25,825 QA pairs) — no pseudo-label augmentation.

The official DriveLM-nuScenes training split is non-i.i.d. due to severe QA-template distributional mismatch: ~65% of training QA templates are absent from the test set, ~10% of test templates never appear in training, and per-template frequency differences reach up to 20 percentage points. This causes models trained on the official split to perform poorly at test time despite large data volume.

The custom split is a scene-level 80/10/10 train/val/test repartition constructed to be distributionally aligned with the test question-type distribution, producing 25,825 training QA pairs. See github.com/dmitrykhursen/VQA-AD-CTU for the full split construction methodology.

Training configuration

Full configuration: `configs/finetune/internvl2_2b_lora.yaml`

How to use

import torch
from PIL import Image
from transformers import AutoTokenizer, AutoModel
import torchvision.transforms as T
from torchvision.transforms.functional import InterpolationMode

MODEL_ID   = "dkhursen/InternVL2-2b-LoRA-25k-drivelm"
IMAGE_PATH = "path/to/stitched_6camera.jpg"
QUESTION   = "<image>\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision."

IMAGENET_MEAN = (0.485, 0.456, 0.406)
IMAGENET_STD  = (0.229, 0.224, 0.225)


def build_transform(input_size):
    return T.Compose([
        T.Lambda(lambda img: img.convert("RGB") if img.mode != "RGB" else img),
        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
        T.ToTensor(),
        T.Normalize(mean=IMAGENET_MEAN, std=IMAGENET_STD),
    ])


def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
    best_ratio_diff = float("inf")
    best_ratio = (1, 1)
    area = width * height
    for ratio in target_ratios:
        ratio_diff = abs(aspect_ratio - ratio[0] / ratio[1])
        if ratio_diff < best_ratio_diff:
            best_ratio_diff = ratio_diff
            best_ratio = ratio
        elif ratio_diff == best_ratio_diff:
            if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
                best_ratio = ratio
    return best_ratio


def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
    orig_width, orig_height = image.size
    aspect_ratio = orig_width / orig_height
    target_ratios = sorted(
        {(i, j)
         for n in range(min_num, max_num + 1)
         for i in range(1, n + 1)
         for j in range(1, n + 1)
         if min_num <= i * j <= max_num},
        key=lambda x: x[0] * x[1],
    )
    best_ratio = find_closest_aspect_ratio(
        aspect_ratio, target_ratios, orig_width, orig_height, image_size
    )
    target_w = image_size * best_ratio[0]
    target_h = image_size * best_ratio[1]
    cols = best_ratio[0]
    resized = image.resize((target_w, target_h))
    tiles = []
    for i in range(best_ratio[0] * best_ratio[1]):
        col = i % cols
        row = i // cols
        box = (col * image_size, row * image_size,
               (col + 1) * image_size, (row + 1) * image_size)
        tiles.append(resized.crop(box))
    if use_thumbnail and len(tiles) != 1:
        tiles.append(image.resize((image_size, image_size)))
    return tiles


def load_image(image_path, input_size=448, max_num=12):
    image = Image.open(image_path).convert("RGB")
    transform = build_transform(input_size)
    tiles = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
    return torch.stack([transform(tile) for tile in tiles])


tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True, use_fast=False)
model = AutoModel.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True,
).eval().cuda()

pixel_values = load_image(IMAGE_PATH, max_num=12).to(torch.bfloat16).cuda()
generation_config = {"max_new_tokens": 512, "do_sample": False}

response = model.chat(tokenizer, pixel_values, QUESTION, generation_config)
print("Answer:", response)

Batch inference and evaluation (requires cloning the repo):

git clone https://github.com/dmitrykhursen/VQA-AD-CTU
cd VQA-AD-CTU
bash scripts/setup_env.sh && source vqa-ad-ctu-env/bin/activate

# Run inference (edit MODEL= inside the script)
sbatch scripts/05_inference.sh

# Evaluate
bash scripts/06_evaluate.sh inference/outputs/<MODEL>/local_test.json

Example predictions

Input: stitched 6-camera panoramic view from a single nuScenes keyframe.

Browse 25 sampled scenes: [dmitrykhursen.github.io/VQA-AD-CTU](https://dmitrykhursen.github.io/VQA-AD-CTU/)

Intended use and limitations

Intended use:

• —Ablation baseline for isolating the effect of distributional alignment without pseudo-label augmentation
• —Reproducing the distributional alignment vs. data volume experiment from the thesis
• —Starting point for further fine-tuning or pseudo-label augmentation experiments

Limitations:

• —No pseudo-label augmentation — for best performance use dkhursen/InternVL2-2b-LoRA-25k_plus_DL-PL-10pct (Final 0.589)
• —Visual object localization is the dominant bottleneck: providing ground-truth annotations at test time raises Final Score from 0.560 to 0.775
• —Not intended for deployment in real autonomous driving systems
• —Trained and evaluated on nuScenes scenes from Boston and Singapore only

Resources

Citation

@mastersthesis{khursenko2026vqa,
  author     = {Khursenko, Dmytro},
  title      = {Visual Question Answering for Autonomous Driving},
  school     = {Czech Technical University in Prague, Faculty of Electrical Engineering},
  year       = {2026},
  supervisor = {Hurych, David and Tolias, Georgios}
}

@inproceedings{sima2024drivelm,
  title     = {DriveLM: Driving with Graph Visual Question Answering},
  author    = {Sima, Chonghao and Renz, Katrin and Chitta, Kashyap and others},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2024}
}