devnote5676/schwartz-values-classifier

1---2license: mit3datasets:4- webis/Touche23-ValueEval5language:6- en7metrics:8- f19tags:10- social-values11---12# Schwartz Value Classifier13This classifier is intended to predict the existence of social values from text snippets.14 15*Disclaimer: this is not the official repo published by the authors of the paper, and may not truly replicate the performance described in the original study* 16 17## Value dimensions18In this model we follow the 10-dimensional categorization of the Schwartz values. [link](https://en.wikipedia.org/wiki/Theory_of_basic_human_values)19 201. security – safety, harmony, and stability of society, of relationships, and of self212. power – social status and prestige, control or dominance over people and resources223. achievement – personal success through demonstrating competence according to social standards234. hedonism – pleasure or sensuous gratification for oneself245. stimulation – excitement, novelty and challenge in life256. self-direction – independent thought and action—choosing, creating, exploring267. universalism – understanding, appreciation, tolerance, and protection for the welfare of all people and for nature278. benevolence – preserving and enhancing the welfare of those with whom one is in frequent personal contact (the 'in-group')289. conformity – restraint of actions, inclinations, and impulses likely to upset or harm others and violate social expectations or norms2910. tradition – respect, commitment, and acceptance of the customs and ideas that one's culture or religion provides30    31## Datasets32This model is finetuned on two datasets: ValueNet (A New Dataset for Human Value Driven Dialogue System, Qiu et al. 2021) and Touche23-ValueEval (The Touché23-ValueEval Dataset for Identifying Human Values behind Arguments, Mirzakhmedova et al., 2023).33 34We follow the original paper to convert both datasets into a binary classification task for each dimension.35- ValueNet36  - A sentence has a positive label if the original label contains 1 (positive) or -1 (negative), and 0 if the original label is 0.37- ValueEval38  - A sentence is assigned a positive label if the original label vector is marked 1 for that dimension.39  - Since the original paper follows a 20-dimension refined categorization, we map them back to 10 dimensions. Therefore, the same sentence appears ten times, once for each dimension.40 41## How to use42Start your sentence with a label that indicates which dimension to measure. An example would be:43 44- \<power> [SEP] staying out late after telling my girlfriend I could be home early45 46Please make sure to follow the exact format "<value\_name>" at the beginning of the sentence as this is a special token in the tokenizer: any spaces or different formats will not be encoded correctly.47 48## Performances49- macro F1 score50  - on ValueNet: 0.64851  - on ValueEval: 0.74452  - Combined: 0.75953  54- ROC-AUC55  - on ValueNet: 0.73656  - on ValueEval:0.84757  - Combined: 0.85558 59## Training details60- Base model: bert-base-uncased61- Epochs: 10 w/ early stopping after no F1 increase in 3 epochs62- Learning rate: 5e-5 w/ warmup for 0.03 steps and subsequent linear decay63- Batch size: 3264- Upsampled training set to maintain 1:1 balance for pos:neg labels.65 66## References67- Do Differences in Values Influence Disagreements in Online Discussions? (EMNLP'23) [link](https://aclanthology.org/2023.emnlp-main.992/)