Modelpublic

ola-owo/distilbert-bigfive-sentence-transformer-lora

sourceHugging Faceupdated 4mo agoView on Hugging Face

0likes

Model Card

distilBERT-based Big-5 personality scorer

This is a sentence-transformers model finetuned from distilbert/distilbert-base-multilingual-cased on the ola-owo/big-five-personality-traits dataset. It maps sentences & paragraphs to a 5-dimensional dense vector space and can be used for feature extraction.

Model Details

Model Description

—Model Type: Sentence Transformer
—Base model: distilbert/distilbert-base-multilingual-cased
—Maximum Sequence Length: 512 tokens
—Output Dimensionality: 5 dimensions
—Similarity Function: Cosine Similarity
—Supported Modality: Text
—Training Dataset:
—ola-owo/big-five-personality-traits
—Languages: en, es

Model Sources

—Documentation: Sentence Transformers Documentation
—Repository: Sentence Transformers on GitHub
—Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'DistilBertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 5, 'bias': True, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

bash

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ola-owo/distilbert-bigfive-sentence-transformer")
# Run inference
sentences = [
    'Can be organized but not rigidly so.',
    'Se adapta bien a nuevas ideas, sin renunciar a la practicidad.',
    'Valora las relaciones y se esfuerza por ser considerado.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 5]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.2455, -0.3998],
#         [ 0.2455,  1.0000, -0.3603],
#         [-0.3998, -0.3603,  1.0000]])

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

ola-owo/big-five-personality-traits

—Dataset: ola-owo/big-five-personality-traits
—Size: 2,250 training samples
—Columns: <code>sentence</code> and <code>label</code>
—Approximate statistics based on the first 100 samples: | | sentence | label | |:---------|:-----------------------------------------------------------------------------------|:-----------------------------------| | type | string | list | | modality | text | | | details | <ul><li>min: 12 tokens</li><li>mean: 22.37 tokens</li><li>max: 47 tokens</li></ul> | <ul><li>size: 5 elements</li></ul> |
—Samples: | sentence | label | |:------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------| | <code>Presenta signos evidentes de inquietud o depresión cuando no tiene una interacción social y atención frecuentes.</code> | <code>[-1.0, -1.0, 1.0, -1.0, -1.0]</code> | | <code>Normalmente, son personas corteses y dispuestas a llegar a acuerdos, aunque no a costa de sus principios fundamentales o de la justicia.</code> | <code>[-1.0, -1.0, -1.0, 0.5, -1.0]</code> | | <code>Sus emociones van y vienen, y algunos días son más difíciles que otros.</code> | <code>[-1.0, -1.0, -1.0, -1.0, 0.5]</code> |
—Loss: <code>_main_.MultiLabelBCEWithLogitsLoss</code>

Evaluation Dataset

ola-owo/big-five-personality-traits

—Dataset: ola-owo/big-five-personality-traits
—Size: 250 evaluation samples
—Columns: <code>sentence</code> and <code>label</code>
—Approximate statistics based on the first 100 samples: | | sentence | label | |:---------|:----------------------------------------------------------------------------------|:-----------------------------------| | type | string | list | | modality | text | | | details | <ul><li>min: 8 tokens</li><li>mean: 22.07 tokens</li><li>max: 37 tokens</li></ul> | <ul><li>size: 5 elements</li></ul> |
—Samples: | sentence | label | |:-------------------------------------------------------------------------------------------------------------------|:-------------------------------------------| | <code>Demonstrates a practical approach to goals with occasional lapses in consistency.</code> | <code>[-1.0, 0.5, -1.0, -1.0, -1.0]</code> | | <code>Equilibra el deseo de orden con la capacidad de adaptarse a los cambios inesperados.</code> | <code>[-1.0, 0.5, -1.0, -1.0, -1.0]</code> | | <code>Equilibra la tradición con la curiosidad, y está abierto a nuevas ideas, siempre que sean razonables.</code> | <code>[0.5, -1.0, -1.0, -1.0, -1.0]</code> |
—Loss: <code>_main_.MultiLabelBCEWithLogitsLoss</code>

Training Hyperparameters

Non-Default Hyperparameters

—per_device_train_batch_size: 64
—num_train_epochs: 10
—learning_rate: 0.0004510471199446068
—warmup_steps: 0.1
—per_device_eval_batch_size: 64
—hub_revision: v2

All Hyperparameters

<details><summary>Click to expand</summary>

—per_device_train_batch_size: 64
—num_train_epochs: 10
—max_steps: -1
—learning_rate: 0.0004510471199446068
—lr_scheduler_type: linear
—lr_scheduler_kwargs: None
—warmup_steps: 0.1
—optim: adamwtorchfused
—optim_args: None
—weight_decay: 0.0
—adam_beta1: 0.9
—adam_beta2: 0.999
—adam_epsilon: 1e-08
—optim_target_modules: None
—gradient_accumulation_steps: 1
—average_tokens_across_devices: True
—max_grad_norm: 1.0
—label_smoothing_factor: 0.0
—bf16: False
—fp16: False
—bf16_full_eval: False
—fp16_full_eval: False
—tf32: None
—gradient_checkpointing: False
—gradient_checkpointing_kwargs: None
—torch_compile: False
—torch_compile_backend: None
—torch_compile_mode: None
—use_liger_kernel: False
—liger_kernel_config: None
—use_cache: False
—neftune_noise_alpha: None
—torch_empty_cache_steps: None
—auto_find_batch_size: False
—log_on_each_node: True
—logging_nan_inf_filter: True
—include_num_input_tokens_seen: no
—log_level: passive
—log_level_replica: warning
—disable_tqdm: False
—project: huggingface
—trackio_space_id: None
—trackio_bucket_id: None
—trackio_static_space_id: None
—per_device_eval_batch_size: 64
—prediction_loss_only: True
—eval_on_start: False
—eval_do_concat_batches: True
—eval_use_gather_object: False
—eval_accumulation_steps: None
—include_for_metrics: []
—batch_eval_metrics: False
—save_only_model: False
—save_on_each_node: False
—enable_jit_checkpoint: False
—push_to_hub: False
—hub_private_repo: None
—hub_model_id: None
—hub_strategy: every_save
—hub_always_push: False
—hub_revision: v2
—load_best_model_at_end: False
—ignore_data_skip: False
—restore_callback_states_from_checkpoint: False
—full_determinism: False
—seed: 42
—data_seed: None
—use_cpu: False
—accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
—parallelism_config: None
—dataloader_drop_last: False
—dataloader_num_workers: 0
—dataloader_pin_memory: True
—dataloader_persistent_workers: False
—dataloader_prefetch_factor: None
—remove_unused_columns: True
—label_names: None
—train_sampling_strategy: random
—length_column_name: length
—ddp_find_unused_parameters: None
—ddp_bucket_cap_mb: None
—ddp_broadcast_buffers: False
—ddp_static_graph: None
—ddp_backend: None
—ddp_timeout: 1800
—fsdp: None
—fsdp_config: None
—deepspeed: None
—debug: []
—skip_memory_metrics: True
—do_predict: False
—resume_from_checkpoint: None
—warmup_ratio: None
—local_rank: -1
—prompts: None
—batch_sampler: batch_sampler
—multi_dataset_batch_sampler: proportional
—router_mapping: {}
—learning_rate_mapping: {}

</details>

Training Logs

Epoch	Step	Training Loss	Validation Loss
1.0	36	0.6733	0.5827
2.0	72	0.5326	0.4780
3.0	108	0.4272	0.4340
4.0	144	0.3851	0.4006
5.0	180	0.3598	0.4137
6.0	216	0.3426	0.3949
7.0	252	0.3375	0.3993
8.0	288	0.3341	0.3944
9.0	324	0.3273	0.3922
10.0	360	0.3225	0.3917

Training Time

—Training: 7.5 minutes

Framework Versions

—Python: 3.13.11
—Sentence Transformers: 5.5.1
—Transformers: 5.12.0
—PyTorch: 2.12.0+cu132
—Accelerate: 1.14.0
—Datasets: 2.19.1
—Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->