Neleac/SpaceTimeGPT

1---2datasets:3- HuggingFaceM4/vatex4language:5- en6metrics:7- bleu8- meteor9- rouge10pipeline_tag: video-text-to-text11inference: true12tags:13- video-captioning14model-index:15- name: Caelen16  results:17  - task:18      type: video-captioning19    dataset:20      type: video-captioning21      name: VATEX22    metrics:23    - name: CIDEr24      type: image-captioning25      value: 67.326      verified: false27base_model:28- facebook/timesformer-base-finetuned-k60029- openai-community/gpt230---31<h1 align='center'> SpaceTimeGPT - Video Captioning Model </h1>32 33<div align="center">34  <a href="https://github.com/Neleac/SpaceTimeGPT">35    <img src="https://img.shields.io/badge/GitHub-Neleac/SpaceTimeGPT-purple.svg">36  </a>37  <img src="https://raw.githubusercontent.com/Neleac/SpaceTimeGPT/main/model.JPG" width="75%" height="75%">38  <p> (partial diagrams from <a href="https://arxiv.org/abs/2103.15691">1</a>, <a href="https://arxiv.org/abs/2102.05095">2</a>, <a href="https://arxiv.org/abs/1706.03762">3</a>) </p>39</div>40 41SpaceTimeGPT is a video description generation model capable of spatial and temporal reasoning. Given a video, eight frames are sampled and analyzed by the model. The output is a sentence description of the events that occured in the video, generated using autoregression.42 43## Architecture and Training44Vision Encoder: [timesformer-base-finetuned-k600](https://huggingface.co/facebook/timesformer-base-finetuned-k600) \45Text Decoder: [gpt2](https://huggingface.co/gpt2)46 47The encoder and decoder are initialized using pretrained weights for video classification and sentence completion, respectively. Encoder-decoder cross attention is used to unify the visual and linguistic domains. The model is fine-tuned end-to-end on the video captioning task. See [GitHub repository](https://github.com/Neleac/SpaceTimeGPT) for details.48 49#### Example Inference Code:50```python51import av52import numpy as np53import torch54from transformers import AutoImageProcessor, AutoTokenizer, VisionEncoderDecoderModel55 56device = "cuda" if torch.cuda.is_available() else "cpu"57 58# load pretrained processor, tokenizer, and model59image_processor = AutoImageProcessor.from_pretrained("MCG-NJU/videomae-base")60tokenizer = AutoTokenizer.from_pretrained("gpt2")61model = VisionEncoderDecoderModel.from_pretrained("Neleac/timesformer-gpt2-video-captioning").to(device)62 63# load video64video_path = "never_gonna_give_you_up.mp4"65container = av.open(video_path)66 67# extract evenly spaced frames from video68seg_len = container.streams.video[0].frames69clip_len = model.config.encoder.num_frames70indices = set(np.linspace(0, seg_len, num=clip_len, endpoint=False).astype(np.int64))71frames = []72container.seek(0)73for i, frame in enumerate(container.decode(video=0)):74    if i in indices:75        frames.append(frame.to_ndarray(format="rgb24"))76 77# generate caption78gen_kwargs = {79    "min_length": 10, 80    "max_length": 20, 81    "num_beams": 8,82}83pixel_values = image_processor(frames, return_tensors="pt").pixel_values.to(device)84tokens = model.generate(pixel_values, **gen_kwargs)85caption = tokenizer.batch_decode(tokens, skip_special_tokens=True)[0]86print(caption) # A man and a woman are dancing on a stage in front of a mirror.87```88 89#### Author Information:90👾 [Discord](https://discordapp.com/users/297770280863137802) \91🐙 [GitHub](https://github.com/Neleac) \92🤝 [LinkedIn](https://www.linkedin.com/in/caelenw/)