ControlNet/marlin_vit_small_ytf

1---2library_name: transformers3tags:4- video5- feature6- face7license: cc8base_model:9- ControlNet/MARLIN10pipeline_tag: feature-extraction11---12 13 14# MARLIN: Masked Autoencoder for facial video Representation LearnINg15 16This repo is the official PyTorch implementation for the paper 17[MARLIN: Masked Autoencoder for facial video Representation LearnINg](https://openaccess.thecvf.com/content/CVPR2023/html/Cai_MARLIN_Masked_Autoencoder_for_Facial_Video_Representation_LearnINg_CVPR_2023_paper) (CVPR 2023) ([arXiv](https://arxiv.org/abs/2211.06627)).18 19 20## Use `transformers` (HuggingFace) for Feature Extraction21 22Requirements:23- Python24- PyTorch25- transformers26- einops27 28Currently the huggingface model is only for direct feature extraction without any video pre-processing (e.g. face detection, cropping, strided window, etc).29 30 31```python32import torch33from transformers import AutoModel34 35model = AutoModel.from_pretrained(36    "ControlNet/marlin_vit_small_ytf",  # or other variants37    trust_remote_code=True38)39tensor = torch.rand([1, 3, 16, 224, 224])  # (B, C, T, H, W)40output = model(tensor)  # torch.Size([1, 1568, 384])41```42 43## License44 45This project is under the CC BY-NC 4.0 license. See [LICENSE](LICENSE) for details.46 47## References48If you find this work useful for your research, please consider citing it.49```bibtex50@inproceedings{cai2022marlin,51  title = {MARLIN: Masked Autoencoder for facial video Representation LearnINg},52  author = {Cai, Zhixi and Ghosh, Shreya and Stefanov, Kalin and Dhall, Abhinav and Cai, Jianfei and Rezatofighi, Hamid and Haffari, Reza and Hayat, Munawar},53  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},54  year = {2023},55  month = {June},56  pages = {1493-1504},57  doi = {10.1109/CVPR52729.2023.00150},58  publisher = {IEEE},59}60```61