FoundationVision/unitok_tokenizer

1---2license: apache-2.03pipeline_tag: feature-extraction4---5 6# UniTok: A Unified Tokenizer for Visual Generation and Understanding7 8This repository contains UniTok, a unified visual tokenizer for both image generation and understanding tasks, as presented in [UniTok: A Unified Tokenizer for Visual Generation and Understanding](https://hf.co/papers/2502.20321).9 10Project Page: https://foundationvision.github.io/UniTok/ <br>11Code: https://github.com/FoundationVision/UniTok12 13<p align="center">14<img src="https://github.com/FoundationVision/UniTok/blob/main/assets/teaser.png?raw=true" width=93%>15<p>16 17UniTok encodes fine-grained details for generation and captures high-level semantics for understanding. It's compatible with autoregressive generative models (e.g., LlamaGen), multimodal understanding models (e.g., LLaVA), and unified MLLMs (e.g., Chameleon and Liquid).18 19Built upon UniTok, we construct an MLLM capable of both multimodal generation and understanding, which sets a new state-of-the-art among unified autoregressive MLLMs. The weights of our MLLM will be released soon.20 21<p align="center">22<img src="https://github.com/FoundationVision/UniTok/blob/main/assets/samples.png?raw=true" width=93%>23<p>24 25## Performance26 27<table>28    <thead>29        <tr>30            <th>Method</th>31            <th>#Tokens</th>32            <th>rFID &darr;</th>33            <th>Accuracy</th>34        </tr>35    </thead>36    <tbody>37        <tr>38            <td colspan="4"><i>VQVAE Model</i></td>39        </tr>40        <tr align="center">41            <td>VQ-GAN</td>42            <td>256</td>43            <td>4.98</td>44            <td>--</td>45        </tr>46        <tr align="center">47            <td>RQ-VAE</td>48            <td>256</td>49            <td>1.30</td>50            <td>--</td>51        </tr>52        <tr align="center">53            <td>VAR</td>54            <td>680</td>55            <td>0.90</td>56            <td>--</td>57        </tr>58        <tr>59            <td colspan="4"><i>CLIP Model</i></td>60        </tr>61        <tr align="center">62            <td>CLIP</td>63            <td>256</td>64            <td>--</td>65            <td>76.2</td>66        </tr>67        <tr align="center">68            <td>SigLIP</td>69            <td>256</td>70            <td>--</td>71            <td>80.5</td>72        </tr>73        <tr align="center">74            <td>ViTamin</td>75            <td>256</td>76            <td>--</td>77            <td>81.2</td>78        </tr>79        <tr>80            <td colspan="4"><i>Unified Model</i></td>81        </tr>82        <tr align="center">83            <td>TokenFlow &dagger;</td>84            <td>680</td>85            <td>1.37</td>86            <td>--</td>87        </tr>88        <tr align="center">89            <td>VILA-U &dagger;</td>90            <td>256</td>91            <td>1.80</td>92            <td>73.3</td>93        </tr>94        <tr align="center">95            <td>UniTok</td>96            <td>256</td>97            <td>0.39</td>98            <td>70.5</td>99        </tr>100        <tr align="center">101            <td>UniTok &dagger;</td>102            <td>256</td>103            <td>0.38</td>104            <td>78.6</td>105        </tr>106    </tbody>107</table>108 109 110This repo is used for hosting UniTok's checkpoints.111 112For more details or tutorials see https://github.com/FoundationVision/UniTok.113 114 115## Citation116 117```bibtex118@article{unitok,119  title={UniTok: A Unified Tokenizer for Visual Generation and Understanding},120  author={Ma, Chuofan and Jiang, Yi and Wu, Junfeng and Yang, Jihan and Yu, Xin and Yuan, Zehuan and Peng, Bingyue and Qi, Xiaojuan},121  journal={arXiv preprint arXiv:2502.20321},122  year={2025}123}124```