nphach/jp-parallel-gloss

jp-parallel-gloss

jp-parallel-gloss makes predictions on similarity of Japanese-to-English glosses (definitions). This is a sentence-transformers model fine-tuned using a dataset of 4M+ parallel/non-parallel gloss pairs from the JMDict database and antonym/synonym pairs from WordNet. The base model used is cross-encoder/ms-macro-MiniLM-L-6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. See its application in Kotoba Tag

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: sentence-transformers/all-MiniLM-L6-v2 <!-- at revision fa97f6e7cb1a59073dff9e6b13e2715cf7475ac9 -->
• —Maximum Sequence Length: 256 tokens
• —Output Dimensionality: 384 dimensions
• —Similarity Function: Cosine Similarity
• —Language: English

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'dearest',
    'to become verminous',
    "having an (overly) strong attachment to one's mother",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Binary Classification

• —Evaluated with <code>BinaryClassificationEvaluator</code>

Training Details

• —Size: 4,404,844 training samples
• —Columns: <code>text1</code>, <code>text2</code>, and <code>label</code>
• —Approximate statistics based on the first 1000 samples: | | text1 | text2 | label | |:--------|:---------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:------------------------------------------------------| | type | string | string | int | | details | <ul><li>min: 3 tokens</li><li>mean: 5.65 tokens</li><li>max: 27 tokens</li></ul> | <ul><li>min: 3 tokens</li><li>mean: 5.64 tokens</li><li>max: 31 tokens</li></ul> | <ul><li>False: ~91.80%</li><li>True: ~8.20%</li></ul> |
• —Samples: | text1 | text2 | label | |:-------------------------------|:--------------------------------------|:-------------------| | <code>based on</code> | <code>making up (a deficiency)</code> | <code>False</code> | | <code>folk (esp. music)</code> | <code>if possible</code> | <code>False</code> | | <code>to start</code> | <code>to die</code> | <code>False</code> |
• —Loss: <code>CoSENTLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }

Evaluation

• —Size: 550,605 evaluation samples
• —Columns: <code>text1</code>, <code>text2</code>, and <code>label</code>
• —Approximate statistics based on the first 1000 samples: | | text1 | text2 | label | |:--------|:---------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|:------------------------------------------------------| | type | string | string | int | | details | <ul><li>min: 3 tokens</li><li>mean: 5.74 tokens</li><li>max: 28 tokens</li></ul> | <ul><li>min: 3 tokens</li><li>mean: 5.7 tokens</li><li>max: 32 tokens</li></ul> | <ul><li>False: ~91.60%</li><li>True: ~8.40%</li></ul> |
• —Samples: | text1 | text2 | label | |:----------------------------------------------------------------------------------|:-------------------------------------|:-------------------| | <code>taking one's children along (to an event, into a new marriage, etc.)</code> | <code>disconnect</code> | <code>False</code> | | <code>to thunder</code> | <code>sheet</code> | <code>False</code> | | <code>throwing event (e.g. javelin, discus, shot put)</code> | <code>extinctive prescription</code> | <code>False</code> |
• —Loss: <code>CoSENTLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }

Training Hyperparameters

Non-Default Hyperparameters

• —eval_strategy: steps
• —per_device_train_batch_size: 128
• —per_device_eval_batch_size: 32
• —weight_decay: 0.01
• —num_train_epochs: 8
• —warmup_ratio: 0.1

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: steps
• —prediction_loss_only: True
• —per_device_train_batch_size: 128
• —per_device_eval_batch_size: 32
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 5e-05
• —weight_decay: 0.01
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 8
• —max_steps: -1
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.1
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: True
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: False
• —fp16: False
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: False
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamw_torch
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —dispatch_batches: None
• —split_batches: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —eval_use_gather_object: False
• —average_tokens_across_devices: False
• —prompts: None
• —batch_sampler: batch_sampler
• —multi_dataset_batch_sampler: proportional

</details>

Training Logs

<details><summary>Click to expand</summary>

</details>

Framework Versions

• —Python: 3.9.21
• —Sentence Transformers: 3.4.0
• —Transformers: 4.48.1
• —PyTorch: 2.5.1
• —Accelerate: 1.3.0
• —Datasets: 3.2.0
• —Tokenizers: 0.21.0