Library-Mutsumi/ccip

1---2datasets:3- deepghs/character_similarity4- deepghs/character_index5metrics:6- f17- adjust_random_score8language:9- en10- ja11- zh12pipeline_tag: zero-shot-image-classification13library_name: dghs-imgutils14tags:15- art16- anime17- character18license: openrail19---20 21# CCIP22CCIP(Contrastive Anime Character Image Pre-Training) is a model to calculuate the visual similarity between anime characters in two images. (limited to images containing only a single anime character). More similar the characters between two images are, higher score it should have.23 24# Usage25Using CCIP with [imgutils](https://dghs-imgutils.deepghs.org/main/tutorials/installation/index.html)26 27![](https://dghs-imgutils.deepghs.org/main/_images/ccip_small.plot.py.svg)28Calculuate character similarity between images:29```30from imgutils.metrics import ccip_batch_differences31 32ccip_batch_differences(['ccip/1.jpg', 'ccip/2.jpg', 'ccip/6.jpg', 'ccip/7.jpg'])33array([[6.5350548e-08, 1.6583106e-01, 4.2947042e-01, 4.0375218e-01],34       [1.6583106e-01, 9.8025822e-08, 4.3715334e-01, 4.0748104e-01],35       [4.2947042e-01, 4.3715334e-01, 3.2675274e-08, 3.9229470e-01],36       [4.0375218e-01, 4.0748104e-01, 3.9229470e-01, 6.5350548e-08]],37      dtype=float32)38```39 40[More detailed instruction](https://dghs-imgutils.deepghs.org/main/api_doc/metrics/ccip.html)41 42# Performence43|                Model                |  F1 Score  |  Precision  |  Recall  |  Threshold  |  Cluster_2  |  Cluster_Free  |44|:-----------------------------------:|:----------:|:-----------:|:--------:|:-----------:|:-----------:|:--------------:|45|        ccip-caformer_b36-24         |  0.940925  |  0.938254   | 0.943612 |  0.213231   |   0.89508   |    0.957017    |46|   ccip-caformer-24-randaug-pruned   |  0.917211  |  0.933481   | 0.901499 |  0.178475   |  0.890366   |    0.922375    |47|       ccip-v2-caformer_s36-10       |  0.906422  |  0.932779   | 0.881513 |  0.207757   |  0.874592   |    0.89241     |48| ccip-caformer-6-randaug-pruned_fp32 |  0.878403  |  0.893648   | 0.863669 |  0.195122   |  0.810176   |    0.897904    |49|        ccip-caformer-5_fp32         |  0.864363  |   0.90155   | 0.830121 |  0.183973   |  0.792051   |    0.862289    |50|        ccip-caformer-4_fp32         |  0.844967  |  0.870553   | 0.820842 |   0.18367   |  0.795565   |    0.868133    |51|       ccip-caformer_query-12        |  0.823928  |  0.871122   | 0.781585 |  0.141308   |  0.787237   |    0.809426    |52|    ccip-caformer-23_randaug_fp32    |  0.81625   |  0.854134   | 0.781585 |  0.136797   |  0.745697   |     0.8068     |53| ccip-caformer-2-randaug-pruned_fp32 |  0.78561   |  0.800148   | 0.771592 |  0.171053   |  0.686617   |    0.728195    |54|        ccip-caformer-2_fp32         |  0.755125  |  0.790172   | 0.723055 |  0.141275   |   0.64977   |    0.718516    |55 56* The calculation of `F1 Score`, `Precision`, and `Recall` considers "the characters in both images are the same" as a positive case. `Threshold` is determined by finding the maximum value on the F1 Score curve.57* `Cluster_2` represents the approximate optimal clustering solution obtained by tuning the eps value in DBSCAN clustering algorithm with min_samples set to `2`, and evaluating the similarity between the obtained clusters and the true distribution using the `random_adjust_score`.58* `Cluster_Free` represents the approximate optimal solution obtained by tuning the `max_eps` and `min_samples` values in the OPTICS clustering algorithm, and evaluating the similarity between the obtained clusters and the true distribution using the `random_adjust_score`.59 60![operations benchmark](https://dghs-imgutils.deepghs.org/main/_images/ccip_benchmark.plot.py.svg)61 62# Citation63```bibtex64@misc{CCIP,65  title={Contrastive Anime Character Image Pre-Training},66  author={Ziyi Dong and narugo1992},67  year={2024},68  howpublished={\url{https://huggingface.co/deepghs/ccip}}69}70```