Chandar/sv-subject-based-bert-base-uncased

SentenceTransformer based on google-bert/bert-base-uncased

This is a sentence-transformers model finetuned from google-bert/bert-base-uncased. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• —Model Type: Sentence Transformer
• —Base model: google-bert/bert-base-uncased <!-- at revision 86b5e0934494bd15c9632b12f734a8a67f723594 -->
• —Maximum Sequence Length: 512 tokens
• —Output Dimensionality: 768 dimensions
• —Similarity Function: Cosine Similarity <!-- - Training Dataset: Unknown --> <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Documentation: Sentence Transformers Documentation
• —Repository: Sentence Transformers on GitHub
• —Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Chandar/sv-subject-based-bert-base-uncased")
# Run inference
sentences = [
    'mason-wasp\tis\tsolitary.\nThus\tthe\tprocess\tof\torganic\tevolution\tis\tfar\tfrom\tbeing\tfully\tunderstood.\tWe\tcan\nonly\tsuppose\tthat\tas\tthere\tare\tdevised\tby\thuman\tbeings\tmany\tpuzzles\tapparently\nunanswerable\ttill\tthe\tanswer\tis\tgiven,\tand\tmany\tnecromantic\ttricks\twhich\tseem\nimpossible\ttill\tthe\tmode\tof\tperformance\tis\tshown;\tso\tthere\tare\tapparently\nincomprehensible\tresults\twhich\tare\treally\tachieved\tby\tnatural\tprocesses.\tOr,\notherwise,\twe\tmust\tconclude\tthat\tsince\tLife\titself\tproves\tto\tbe\tin\tits\tultimate\nnature\tinconceivable,\tthere\tis\tprobably\tan\tinconceivable\telement\tin\tits\tultimate\nworkings.\nEND\tOF\tVOL.\tI.',
    '11.4 The Flea on Schr ¨odinger’s Cat 455\nwhere M = Nm is the total mass of the system, for simplicity we assumed V to be\nanalytic (it will even be taken to be polynomial), and we abbreviated\nfk(ρ)=\n(\n− 1√\nN′\nN\n∑\nl=1\nρl\n)k\n+\nN\n∑\nn=1\n(\n√\nN′ρn − 1√\nN′\nN\n∑\nl=1\nρl\n)k\n. (11.22)\nNote that f1(ρ)= 0, so that to lowest order (i.e. k = 2) we have\nhAE (Q,ρ)=\n(\n1\n2 N\nN\n∑\nn=1\nρ2\nn −\nN\n∑\nk̸=l\nρkρl\n)\nV ′′(Q)+ ··· (11.23)\nWe pass to the corresponding quantum-mechanical Hamiltonians in the usual way,\nand couple a two-level quantum system to the apparatus through the Hamiltonian\nhSA = μ·σ3 ⊗P, (11.24)\nwhere the object observable s = σ3, acting on HS = C2, is to be measured. The idea\nis that hA is the Hamiltonian of a pointer that registers outcomes by localization on\nthe real line, hE is the (free) Hamiltonian of the “environment”, realized as the in-\nternal degrees of the freedom of the total apparatus that are not used in recording\nthe outcome of the measurement, and hAE describes the pointer-environment inter-\naction. The classical description of the apparatus then involves two approximations:\n• Ignoring all degrees of freedom except those of A, which classically are (P,Q);\n• Taking the classical limit of hA, here realized as N →∞ (in lieu of ¯h →0).\nThe measurement ofs is now expected to unfold according to the following scenario:\n1. The apparatus is initially in a metastable state (this is a very common assump-\ntion), whose wave-function is e.g. a Gaussian centered at the origin.\n2. If the object state is “spin up”, i.e., ψS =( 1,0), then it kicks the pointer to the\nright, where it comes to a standstill at the bottom of the double well. If spin is\ndown, likewise to the left. IfψS =( 1,1)/\n√\n2, the pointer moves to a superposition\nof these, which is close to the ground state of V displayed in Figure 11.2.\n3. In the last case, the Flea mechanism of §10.2 comes into play: tiny asymmetric\nperturbations irrelevant for small N localize the ground state as N →∞.\n4. Mere localization of the ground state of the perturbed (apparatus) Hamiltonian in\nthe classical regime is not enough: there should be a dynamical transition from\nthe ground state of the original (unperturbed) Hamiltonian (which has become\nmetastable upon perturbation) to the ground state of the perturbed one. This dy-\nnamical mechanism in question should also recover the Born rule.\nThus the classical description of the apparatus is at the same time the root of the\nmeasurement problem and the key to its solution: it creates the problem because at\nfirst sight a Schr¨odinger Cat state has the wrong classical limit (namely a mixture),\nbut it also solves it, because precisely in the classical limit Cat states are destabilized\neven by the tiniest (asymmetric) perturbations and collapse to the “right” states.',
    'tration,p.115).Thebaby\'sparentshavealreadyattainedtheirfullgrowth.Theamountoffoodthatapersonneedstomakeupfortheheatradiatedfromthesurfaceofthebodyvariesw^iththesizeandalsow^iththeshapeofthebody(seeillustrationopposite).Thesmallerachild,themoresurfacehehasinproportiontohisbodyweight,andhencehelosesrelativelymoreheat.Byactualmeasurement,aone-year-oldchildneedsapproximatelytwiceasmuchenergyperpoundofbody-weightasdoesanadult.Energyneedsareindirectlyrelatedtosex.Girlsandwomenhaveathickerlayeroffattytissuebeneaththeskinthanboysandmen.Thisfatpreventsrapidradiationofheatfromthebody.Itisinterestingtorecallthatmostlong-distanceswimmingrecordsareheldbywomenratherthanbymen.Exposurealsoaffectsthebody\'slossofheat.Thebodylosesheatfasterinacold,dry,windyclimatethaninawarm,moistclimate.Cloth-ingandshelterare,ofcourse,factorsinthelossofheat.Circulationoftheblood,breathing,andotherprocessesarecontinuallygoingonwhenthebodyisatrest."Warm-blooded"animalsmaintainaconstanttemperature.Theheatcontinuallyradiatingfromthesurfaceisconstantlybeingreplaced.Muscularmovementsarecontinuallytakingplaceinthedigestiveorgans,andenergyisusedinvariousotherwayswithinthebody.From40to50percentofthebodyismadeupofmus-culartissue.Thebulkofthistissueisattachedtotheskeletonandisusedinstandingaswellasinlocomotionandothervoluntaryactions.Atalltimes,evenwhenthesemusclesarerelaxed,energyisusedinkeepingthemsomewhatonthestretch.AbovetheBaseLineTheamountofenergythatthebodyuses,evenwhileitis"doingnothing",isconstantlyinfluencedbytwosetsoffactors.Digestingfoodinvolvesameasurableamountofenergy.Thusthebodyusesabout6percentmoreenergysoonafterameal,whenthedigestiveorgansaremostactive,thanjustbeforeameal,whendigestionispracti-callyatastandstill.Whenyouaresittingandreading,orwhenyouarestandingquietly,yourbodyusesaboutone-and-a-thirdtimesasmuchenergyasitdoeswhilesleeping.Walkingatamoderatepaceusesabouttwo-and-a-halftimesasmuch;runningusesaboutseventimesandstair-climbingaboutfifteentimesasmuch.UnitofEnergyTomeasuretheenergyexpendedbythelivingbody,weuseaunitdevelopedbyengineers.ThisistheCalorie(Cal),and,likethemorefamiliarfoot-pound(ft-lb)usedinmeasuringwork,itiscomposedoftwofactors.Wemeasureworkasifitalwaysconsistedofsomequantityofmatter(pounds)movingacertaindistance(feet).Inasimilarwaywemeasureheatasaquantityofmatter,forexample,1kilogram(kg)ofwater,beingheatedacertain"distance"(1degreeonthecentigradescale).116',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!--

Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details> -->

<!--

Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details> -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

Evaluation

Metrics

Binary Classification

• —Evaluated with <code>BinaryClassificationEvaluator</code>

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Dataset

Unnamed Dataset

• —Size: 1,124,250 training samples
• —Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code>
• —Approximate statistics based on the first 1000 samples: | | sentence1 | sentence2 | label | |:--------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:------------------------------------------------| | type | string | string | int | | details | <ul><li>min: 127 tokens</li><li>mean: 127.0 tokens</li><li>max: 127 tokens</li></ul> | <ul><li>min: 22 tokens</li><li>mean: 421.72 tokens</li><li>max: 512 tokens</li></ul> | <ul><li>0: ~50.10%</li><li>1: ~49.90%</li></ul> |
• —Samples: | sentence1 | sentence2 | label | |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------| | <code>mason-wasp is solitary.<br>Thus the process of organic evolution is far from being fully understood. We can<br>only suppose that as there are devised by human beings many puzzles apparently<br>unanswerable till the answer is given, and many necromantic tricks which seem<br>impossible till the mode of performance is shown; so there are apparently<br>incomprehensible results which are really achieved by natural processes. Or,<br>otherwise, we must conclude that since Life itself proves to be in its ultimate<br>nature inconceivable, there is probably an inconceivable element in its ultimate<br>workings.<br>END OF VOL. I.</code> | <code>difficulties which my own hypothesis avoids? If, as I have argued, the germ-<br>plasm consists of substantially similar units (having only those minute<br>differences expressive of individual and ancestral differences of structure), none<br>of the complicated requirements which Dr. Romanes emphasizes exist; and the<br>alleged inconceivability disappears.<br>Here I must end: not intending to say more, unless for some very urgent reason;<br>and leaving others to carry on the discussion. I have, indeed, been led to suspend<br>for a short time my proper work, only by consciousness of the transcendent<br>importance of the question at issue. As I have before contended, a right answer<br>to the question whether acquired characters are or are not inherited, underlies<br>right beliefs, not only in Biology and Psychology, but also in Education, Ethics,<br>and Politics.<br>III.<br>As a species of literature, controversy is characterised by a terrible fertility. Each<br>proposition becomes the parent of half a <br>dozen; so that a few replie...</code> | <code>1</code> | | <code>mason-wasp is solitary.<br>Thus the process of organic evolution is far from being fully understood. We can<br>only suppose that as there are devised by human beings many puzzles apparently<br>unanswerable till the answer is given, and many necromantic tricks which seem<br>impossible till the mode of performance is shown; so there are apparently<br>incomprehensible results which are really achieved by natural processes. Or,<br>otherwise, we must conclude that since Life itself proves to be in its ultimate<br>nature inconceivable, there is probably an inconceivable element in its ultimate<br>workings.<br>END OF VOL. I.</code> | <code>RNA Editing in Trypanosomes<br>The trypanosomes are a group of protozoa that include the pathogenTrypanosoma<br>brucei, which causes sleeping sickness in humans (Figure 15.12). Trypanosomes, and<br>virtually all other eukaryotes, have organelles called mitochondria that supply the cell with<br>chemical energy. Mitochondria are organelles that express their own DNA and are believed<br>to be the remnants of a symbiotic relationship between a eukaryote and an engulfed<br>prokaryote. The mitochondrial DNA of trypanosomes exhibit an interesting exception to<br>The Central Dogma: their pre-mRNAs do not have the correct information to specify a<br>functional protein. Usually, this is because the mRNA is missing several U nucleotides.<br>The cell performs an additional RNA processing step calledRNA editing to remedy this.<br>Figure 15.12 Trypanosoma brucei is the causative agent of sleeping sickness in humans.<br>The mRNAs of this pathogen must be modified by the addition of nucleotides before protein<br>synthesis can occur. (cr...</code> | <code>1</code> | | <code>mason-wasp is solitary.<br>Thus the process of organic evolution is far from being fully understood. We can<br>only suppose that as there are devised by human beings many puzzles apparently<br>unanswerable till the answer is given, and many necromantic tricks which seem<br>impossible till the mode of performance is shown; so there are apparently<br>incomprehensible results which are really achieved by natural processes. Or,<br>otherwise, we must conclude that since Life itself proves to be in its ultimate<br>nature inconceivable, there is probably an inconceivable element in its ultimate<br>workings.<br>END OF VOL. I.</code> | <code>plane subtends with the axis an angle of 89° 59′, we have an ellipse which no<br>human eye, even when aided by an accurate pair of compasses, can distinguish<br>from a circle. Decreasing the angle minute by minute, this closed curve becomes<br>perceptibly eccentric, then manifestly so, and by and by acquires so immensely<br>elongated a form so as to bear no recognizable resemblance to a circle. <br>By<br>continuing this process the ellipse changes insensibly into a parabola. On still<br>further diminishing the angle, the parabola becomes an hyperbola. And finally, if<br>the cone be made gradually more obtuse, the hyperbola passes into a straight line<br>as the angle of the cone approaches 180°. Here then we have five different<br>species of line—circle, ellipse, parabola, hyperbola, and straight line—each<br>having its peculiar properties and its separate equation, and the first and last of<br>which are quite opposite in nature, connected together as members of one series,<br>all producible by a single process of insensible...</code> | <code>1</code> |
• —Loss: <code>CoSENTLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }

Training Hyperparameters

Non-Default Hyperparameters

• —per_device_train_batch_size: 16
• —per_device_eval_batch_size: 32
• —learning_rate: 2e-05
• —weight_decay: 0.01
• —max_steps: 2000

All Hyperparameters

<details><summary>Click to expand</summary>

• —overwrite_output_dir: False
• —do_predict: False
• —eval_strategy: no
• —prediction_loss_only: True
• —per_device_train_batch_size: 16
• —per_device_eval_batch_size: 32
• —per_gpu_train_batch_size: None
• —per_gpu_eval_batch_size: None
• —gradient_accumulation_steps: 1
• —eval_accumulation_steps: None
• —torch_empty_cache_steps: None
• —learning_rate: 2e-05
• —weight_decay: 0.01
• —adam_beta1: 0.9
• —adam_beta2: 0.999
• —adam_epsilon: 1e-08
• —max_grad_norm: 1.0
• —num_train_epochs: 3.0
• —max_steps: 2000
• —lr_scheduler_type: linear
• —lr_scheduler_kwargs: {}
• —warmup_ratio: 0.0
• —warmup_steps: 0
• —log_level: passive
• —log_level_replica: warning
• —log_on_each_node: True
• —logging_nan_inf_filter: True
• —save_safetensors: True
• —save_on_each_node: False
• —save_only_model: False
• —restore_callback_states_from_checkpoint: False
• —no_cuda: False
• —use_cpu: False
• —use_mps_device: False
• —seed: 42
• —data_seed: None
• —jit_mode_eval: False
• —use_ipex: False
• —bf16: False
• —fp16: False
• —fp16_opt_level: O1
• —half_precision_backend: auto
• —bf16_full_eval: False
• —fp16_full_eval: False
• —tf32: None
• —local_rank: 0
• —ddp_backend: None
• —tpu_num_cores: None
• —tpu_metrics_debug: False
• —debug: []
• —dataloader_drop_last: False
• —dataloader_num_workers: 0
• —dataloader_prefetch_factor: None
• —past_index: -1
• —disable_tqdm: False
• —remove_unused_columns: True
• —label_names: None
• —load_best_model_at_end: False
• —ignore_data_skip: False
• —fsdp: []
• —fsdp_min_num_params: 0
• —fsdp_config: {'minnumparams': 0, 'xla': False, 'xlafsdpv2': False, 'xlafsdpgrad_ckpt': False}
• —fsdp_transformer_layer_cls_to_wrap: None
• —accelerator_config: {'splitbatches': False, 'dispatchbatches': None, 'evenbatches': True, 'useseedablesampler': True, 'nonblocking': False, 'gradientaccumulationkwargs': None}
• —deepspeed: None
• —label_smoothing_factor: 0.0
• —optim: adamw_torch
• —optim_args: None
• —adafactor: False
• —group_by_length: False
• —length_column_name: length
• —ddp_find_unused_parameters: None
• —ddp_bucket_cap_mb: None
• —ddp_broadcast_buffers: False
• —dataloader_pin_memory: True
• —dataloader_persistent_workers: False
• —skip_memory_metrics: True
• —use_legacy_prediction_loop: False
• —push_to_hub: False
• —resume_from_checkpoint: None
• —hub_model_id: None
• —hub_strategy: every_save
• —hub_private_repo: None
• —hub_always_push: False
• —hub_revision: None
• —gradient_checkpointing: False
• —gradient_checkpointing_kwargs: None
• —include_inputs_for_metrics: False
• —include_for_metrics: []
• —eval_do_concat_batches: True
• —fp16_backend: auto
• —push_to_hub_model_id: None
• —push_to_hub_organization: None
• —mp_parameters:
• —auto_find_batch_size: False
• —full_determinism: False
• —torchdynamo: None
• —ray_scope: last
• —ddp_timeout: 1800
• —torch_compile: False
• —torch_compile_backend: None
• —torch_compile_mode: None
• —include_tokens_per_second: False
• —include_num_input_tokens_seen: False
• —neftune_noise_alpha: None
• —optim_target_modules: None
• —batch_eval_metrics: False
• —eval_on_start: False
• —use_liger_kernel: False
• —liger_kernel_config: None
• —eval_use_gather_object: False
• —average_tokens_across_devices: False
• —prompts: None
• —batch_sampler: batch_sampler
• —multi_dataset_batch_sampler: proportional

</details>

Training Logs

Framework Versions

• —Python: 3.12.9
• —Sentence Transformers: 4.1.0
• —Transformers: 4.53.0
• —PyTorch: 2.7.1
• —Accelerate: 1.8.1
• —Datasets: 3.6.0
• —Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->