RichardScottOZ/comic-panel-vlm-v1

Comic Panel VLM Embedder v1

Comic Panel VLM Embedder v1 is a multimodal panel feature extraction model designed to produce rich 512-dimensional embeddings for individual comic book panels.

It represents Stage 3 of the Comic Analysis Framework v2.0, and is trained downstream of CoSMo v4 (the page classifier that filters raw archives to narrative content). Its embeddings are intended as input to Stage 4 sequence modeling and similarity search over comic page collections.

This v1 iteration uses VLM-enriched panel descriptions generated by Gemini 2.5 Flash Lite — replacing the sparse OCR text used in prior versions — enabling the model to ground panel embeddings in narrative content (character descriptions, dialogue, mood, scene context) rather than raw detected text alone.

Model Architecture

The model is based on the PanelFeatureExtractor class. It fuses three independent modalities per panel into a single 512-dim embedding using an adaptive gated fusion mechanism.

1. Visual Encoder — Dual Backbone (~111M params, frozen)

Both visual backbones are frozen during training. Their 512-dim features are combined using a learned attention fusion: a 2-layer MLP computes a softmax weight over the two streams, producing a single 512-dim visual feature.

2. Text Encoder (~22M params, frozen)

Panel text is constructed from the VLM analysis JSON: description + joined text_content[].text dialogue. The backbone is frozen; a single linear projection maps 384 → 512.

3. Compositional Encoder (~0.2M params, trainable)

A 3-layer MLP encodes 7 spatial/layout features per panel:

Panel bounding boxes are sourced from the VLM JSON box_2d field ([y1, x1, y2, x2] in 0–1000 normalised coordinates, converted from Gemini 2.5 Flash Lite output).

4. Adaptive Fusion (~0.8M params, trainable)

An AdaptiveFusion module independently normalises each modality with LayerNorm(512), then computes a 3-way softmax gate over the concatenation of all three features plus optional modality presence indicators. The final embedding is a weighted sum of the three normalised modalities plus a small learned residual.

Total: ~115M params | ~2.5M trainable (frozen backbones)

Training

Training Objectives

Loss = 1.0 × L_contrastive + 0.5 × L_reconstruction + 0.3 × L_modality_alignment

• —Contrastive: Panels from the same page should be mutually similar (temperature=0.07)
• —Reconstruction: Predict one masked panel embedding from the remaining context
• —Modality alignment: Cross-entropy alignment between vision and text embeddings for the same panel

Loss Curve

Input Format

The model operates per-panel. For a given page it expects:

• —Panel image crop: (3, 224, 224) float32 tensor, normalised with ImageNet mean/std
• —Panel text tokens: input_ids + attention_mask from all-MiniLM-L6-v2 tokenizer (max 128 tokens), constructed from description + dialogue
• —Compositional features: (7,) float32 tensor of spatial/layout values
• —Modality mask: (3,) binary indicator — [has_vision, has_text, has_comp]

At the page level, up to 16 panels are batched together with zero-padding. A boolean panel_mask (N,) indicates valid vs padded slots.

Output Format

panel_embeddings: (N_panels, 512)  # float32, one embedding per panel

When run over a full dataset via generate_stage3_embeddings_vlm.py, output is stored in a Zarr store:

stage3_embeddings_vlm.zarr/
├── panel_embeddings   shape: (N_pages, 16, 512)  float32
└── panel_masks        shape: (N_pages, 16)        bool

Usage

Because the model requires VLM-annotated panel JSONs, inference uses the pipeline scripts from the Comic Analysis Repository.

Quick Start — Load Model

import torch
from stage3_panel_features_framework import PanelFeatureExtractor

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = PanelFeatureExtractor(
    visual_backbone='both',
    visual_fusion='attention',
    feature_dim=512,
    freeze_backbones=True
).to(device)

checkpoint = torch.load('best_model_vlm.pt', map_location=device)
state_dict = checkpoint.get('model_state_dict', checkpoint)
model.load_state_dict(state_dict)
model.eval()

Generate Embeddings for a Dataset

python src/version2/generate_stage3_embeddings_vlm.py \
  --manifest manifests/master_manifest_20251229.csv \
  --vlm_cache_dir /data/vlm_cache \
  --pss_labels pss_labels_v1.json \
  --checkpoint checkpoints/stage3_vlm/best_model_vlm.pt \
  --output_zarr stage3_embeddings_vlm.zarr \
  --output_metadata stage3_metadata_vlm.json \
  --batch_size 64 \
  --num_workers 8

Intended Use

This model is designed as an intermediate representation layer in a comic analysis pipeline:

1. 1.CoSMo v4 classifies pages → filters to narrative pages
2. 2.This model embeds panels → 512-dim per-panel features
3. 3.Stage 4 (PanelSequenceTransformer) contextualises panel sequences → strip embeddings
4. 4.Stage 5 search performs similarity search over the final embeddings

Panel embeddings from this model are suitable for:

• —Similarity search over individual panels (find visually/narratively similar panels)
• —Input to sequence models that require panel-level features
• —Downstream clustering or classification of panels

They are not recommended for cover/advertisement pages — the model was trained exclusively on narrative story pages and its embedding space reflects that distribution.

Citation

If you use this model or the Comic Analysis Framework, please reference the repository:

@misc{comic-analysis-framework,
  author = {RichardScottOZ},
  title  = {Comic Analysis Framework v2.0},
  year   = {2026},
  url    = {https://github.com/RichardScottOZ/Comic-Analysis}
}