ni-eminen/object-centric-lmdrive-8qt

object-centric-lmdrive-8qt

Object-centric LMDrive with 8 Q-Former query tokens (ablation variant of the 4qt model).

Overview

This is one of four model artefacts released alongside a thesis on replacing the multimodal sensor-fusion encoder in LMDrive with an object-centric encoder based on Stochastic Slot Attention from Video (SAVi), evaluated on the CARLA LangAuto benchmark. The four artefacts are:

• —ni-eminen/savi-pretrained-30slots -- Stage 1a encoder
• —ni-eminen/savi-finetuned-30slots -- Stage 1b encoder
• —ni-eminen/object-centric-lmdrive-4qt -- Q-Former + LLM (4 query tokens)
• —ni-eminen/object-centric-lmdrive-8qt -- Q-Former + LLM (8 query tokens)

Architecture

• —encoder: StoSAVi (30 slots, 256-dim)
• —adapter: BLIP-2 Q-Former (BERT-base-uncased backbone, 8 query tokens)
• —llm: LLaVA-1.5-7B (Vicuna-7B backbone, LoRA fine-tuned)
• —waypoint head: 2-layer MLP (4096 -> 4096 -> 10)
• —end head: 2-layer MLP (4096 -> 4096 -> 2)
• —input frames per instruction: 40
• —sample interval: 2
• —max txt len: 64

Inputs

Same as object-centric-lmdrive-4qt.

Training

Same hyperparameters as the 4qt variant, differing only in the number of Q-Former query tokens.

Evaluation

Evaluated on the LangAuto benchmark (32 long routes from CARLA), averaged across 3 repetitions.

Citation

@mastersthesis{nieminen2026objcentric,
  title  = {Object-centric world models for language-instructed autonomous driving},
  author = {Nieminen, Matias},
  year   = {2026},
  school = {Aalto University}
}

Related repositories

• —Encoder training code: https://github.com/ni-eminen/SAVi-for-LMDrive-BEV
• —Instruction-fine-tuning + evaluation code: https://github.com/ni-eminen/lmdrive-fix