fztkm/lc_video_description_videomae_gpt2

Length Controllable Video Description Model

This was introduced in the paper Fine-grained length controllable video captioning with ordinal embeddings, IEEE Access, Vol.12, pp. 189667-189688, 2024 by Nitta et al. It enables fine-grained control over the length of generated video captions, a novel feature achieved using ordinal embeddings. An online demo of the model is available at Hugging Face Spaces.

Model Details

Model Description

This model uses VideoMAE as the encoder and GPT2 as the decoder, with ordinal embeddings designed to manage caption lengths effectively. The training data is sourced from the ActivityNet Captions dataset. The model facilitates both token-length control and temporal duration control for captions, making it a powerful tool for applications such as video narration and storytelling.

• —Model type: Encoder-Decoder with autoregressive decoding
• —Languages: English
• —Fine-tuned from: VideoMAE (ViT-B) and GPT2

Training Details

• —Datasets: ActivityNet Captions
• —Encoder: VideoMAE (pretrained on Kinetics-400)
• —Decoder: GPT2 (small version, pretrained on WebText)
• —Length embeddings: Ordinal embeddings defined in the paper
• —Maximum length: 256 tokens
• —Optimizations: AdamW optimizer with learning rate 1e-4 and weight decay 5e-4
• —Frame size: 16 frames
• —Clip duration: 1.5 seconds

How to Use

You can load and use the model as follows:

from transformers import AutoModel, AutoTokenizer

def transform(video_pixel):
    transform_list = [
        transforms.Lambda(lambda x: x / 255.),
        Normalize(
            [0.485, 0.456, 0.406],
            [0.229, 0.224, 0.225]),
        ShortSideScale(256),
        CenterCrop(224),
    ]
    transform = Compose(transform_list)
    return transform(video_pixel).transpose(0, 1)

def captioning(video_tensor):

  video = transform(video)

  model = AutoModel.from_pretrained("fztkm/lc_video_description_videomae_gpt2", trust_remote_code=True).requires_grad_(False)
  tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
  
  output_tokens = model.generate(pixel_values=video_tensor, desired_length=desired_length, max_new_tokens=100)
  caption = tokenizer.batch_decode(output_tokens, skip_special_tokens=True)

  print(caption)

Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

@ARTICLE{Nitta2024_length_control_captioning,
  author={Nitta, Tomoya and Fukuzawa, Takumi and Tamaki, Toru},
  journal={IEEE Access}, 
  title={Fine-grained length controllable video captioning with ordinal embeddings}, 
  year={2024},
  doi={10.1109/ACCESS.2024.3506751}
}