Modelpublic

5CD-AI/visobert-14gb-corpus

sourceHugging Faceupdated 2y agoView on Hugging Face

59likes121downloads

Model Card

5CD-AI/visobert-14gb-corpus

Overview

We continually pretrain uitnlp/visobert on a merged 14GB dataset, the training dataset includes:

—Crawled data (100M comments and 15M posts on Facebook)
—UIT data, which is used to pretrain uitnlp/visobert
—MC4 ecommerce

Here are the results on 4 downstream tasks on Vietnamese social media texts, including Emotion Recognition(UIT-VSMEC), Hate Speech Detection(UIT-HSD), Spam Reviews Detection(ViSpamReviews), Hate Speech Spans Detection(ViHOS): <table> <tr align="center"> <td rowspan=2>Model</td> <td rowspan=2>Avg MF1</td> <td colspan=3>Emotion Recognition</td> <td colspan=3>Hate Speech Detection</td> <td colspan=3>Spam Reviews Detection</td> <td colspan=3>Hate Speech Spans Detection</td> </tr> <tr align="center"> <td>Acc</td> <td>WF1</td> <td>MF1</td> <td>Acc</td> <td>WF1</td> <td>MF1</td> <td>Acc</td> <td>WF1</td> <td>MF1</td> <td>Acc</td> <td>WF1</td> <td>MF1</td> </tr> <tr align="center"> <td align="left">viBERT</td> <td>78.16</td> <td>61.91</td> <td>61.98</td> <td>59.7</td> <td>85.34</td> <td>85.01</td> <td>62.07</td> <td>89.93</td> <td>89.79</td> <td>76.8</td> <td>90.42</td> <td>90.45</td> <td>84.55</td> </tr> <tr align="center"> <td align="left">vELECTRA</td> <td>79.23</td> <td>64.79</td> <td>64.71</td> <td>61.95</td> <td>86.96</td> <td>86.37</td> <td>63.95</td> <td>89.83</td> <td>89.68</td> <td>76.23</td> <td>90.59</td> <td>90.58</td> <td>85.12</td> </tr> <tr align="center"> <td align="left">PhoBERT-Base </td> <td>79.3</td> <td>63.49</td> <td>63.36</td> <td>61.41</td> <td>87.12</td> <td>86.81</td> <td>65.01</td> <td>89.83</td> <td>89.75</td> <td>76.18</td> <td>91.32</td> <td>91.38</td> <td>85.92</td> </tr> <tr align="center"> <td align="left">PhoBERT-Large</td> <td>79.82</td> <td>64.71</td> <td>64.66</td> <td>62.55</td> <td>87.32</td> <td>86.98</td> <td>65.14</td> <td>90.12</td> <td>90.03</td> <td>76.88</td> <td>91.44</td> <td>91.46</td> <td>86.56</td> </tr> <tr align="center"> <td align="left">ViSoBERT</td> <td>81.58</td> <td>68.1</td> <td>68.37</td> <td>65.88</td> <td>88.51</td> <td>88.31</td> <td>68.77</td> <td>90.99</td> <td>90.92</td> <td>79.06</td> <td>91.62</td> <td>91.57</td> <td>86.8</td> </tr> <tr align="center"> <td align="left">visobert-14gb-corpus</td> <td>82.2</td> <td>68.69</td> <td>68.75</td> <td>66.03</td> <td>88.79</td> <td>88.6</td> <td>69.57</td> <td>91.02</td> <td>90.88</td> <td>77.13</td> <td>93.69</td> <td>93.63</td> <td>89.66</td> </tr> </div> </table>

Usage (HuggingFace Transformers)

Install transformers package:

pip install transformers

Then you can use this model for fill-mask task like this:

python

from transformers import pipeline

model_path = "5CD-AI/visobert-14gb-corpus"
mask_filler = pipeline("fill-mask", model_path)

mask_filler("shop làm ăn như cái <mask>", top_k=10)

Fine-tune Configuration

We fine-tune 5CD-AI/visobert-14gb-corpus on 4 downstream tasks with transformers library with the following configuration:

—seed: 42
—gradientaccumulationsteps: 1
—weight_decay: 0.01
—optimizer: AdamW with betas=(0.9, 0.999) and epsilon=1e-08
—training_epochs: 30
—modelmaxlength: 128
—learning_rate: 1e-5
—metricforbest_model: wf1
—strategy: epoch

And different additional configurations for each task: | Emotion Recognition | Hate Speech Detection | Spam Reviews Detection | Hate Speech Spans Detection | | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | |\- trainbatchsize: 64 \- lrschedulertype: linear | \- trainbatchsize: 32 \- lrschedulertype: linear | \- trainbatchsize: 32 \- lrschedulertype: cosine | \- trainbatchsize: 32 \- lrschedulertype: cosine |