CoolFace
Modelpublic

Netta1994/setfit_baai_newrelic_gpt-4o_cot-few_shot_only_reasoning_1726750062.370463

sourceHugging Faceupdated 2y agoView on Hugging Face
0likes6downloads
Model Card

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

  1. 1.Fine-tuning a Sentence Transformer with contrastive learning.
  2. 2.Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

  • —Model Type: SetFit
  • —Sentence Transformer body: BAAI/bge-base-en-v1.5
  • —Classification head: a LogisticRegression instance
  • —Maximum Sequence Length: 512 tokens
  • —Number of Classes: 2 classes <!-- - Training Dataset: Unknown --> <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

Model Labels

LabelExamples
0<ul><li>'The answer provided is a general approach to saving money, offering advice on spending wisely, making good investments, and considering personal goals. It also suggests discussing with a lead or ORGANIZATION_2 for guidance and highlights the importance of health and setting priorities.\n\nHowever, the answer lacks direct grounding in the provided document, which primarily discusses budgeting for studies and spending on learning and development. The document emphasizes evaluating expenditures based on their benefits to personal and organizational goals but does not offer explicit general financial saving tips.\n\nDue to this lack of specific correlation and grounding in the document, the evaluation is:\n\nBad'</li><li>'The answer provided reads incoherently due to the insertion of "Cassandra Rivera Heather Nelson" and other names in inappropriate places throughout the text, making it difficult to assess its alignment with the original document.\n\nFurthermore, the inserted names disrupt the meaning of the text, making it unclear if all relevant points from the document are covered accurately. The structure of the sentences becomes disjointed and presents an overall lack of clarity. \n\nGiven the nature of these errors, it\'s impossible to fairly evaluate whether the answer strictly adheres to the details provided in the source documents. Therefore, based on the clarity, coherence, and alignment to the original text, the final result is:\n\nBad'</li><li>"The provided answer does not satisfactorily respond to the question about accessing the company's training resources. Instead, it contains unrelated information about document management, security protocols, feedback processes, and learning budget requests. The relevant information about accessing training resources is clearly missing or obscure.\n\nReasoning:\n\n1. Irrelevant Details: The answer includes details about the usage of a password manager, secure sharing of information, and expense reimbursement, none of which pertain to accessing training resources.\n\n2. Lack of Specificity: No explicit method or platform for accessing training resources is mentioned, which is the core inquiry.\n\n3. Missed Key Point: The document points towards systems used for personal documents and reimbursement requests but fails to highlight training resource access points.\n\nFinal evaluation: Bad"</li></ul>
1<ul><li>'The answer demonstrates a clear connection to the provided document, outlining key tips and principles for giving feedback as requested by the question. The response includes the importance of timing, focusing on the situation rather than the person, being clear and direct, and the goal of helping rather than shaming. It also mentions the importance of appreciation and receiving feedback with an open mind. \n\nHowever, there are some typographical errors and misplaced words (e.g., "emichelle James Johnson MDamples") that detract slightly from the clarity. Despite these minor issues, the content provided accurately reflects the information in the source document and comprehensively addresses the question. Therefore, the final evaluation is:\n\nGood'</li><li>"The given answer provides a general explanation of why it is important to proactively share information from high-level meetings, but it lacks grounding in the provided document. \n\nWhile the answer discusses the benefits of transparency, alignment with the organization's vision and mission, and fostering an open work environment, it does not directly quote or refer to specific points in the document. This weakens the argument, as it seems more like an independent explanation rather than an answer strictly based on the provided material.\n\nThe document mentions the importance of employees knowing why they are doing what they do and the necessity of closing the information gap by proactively sharing what was discussed in high-level meetings. This specific point from Document 4 could have been directly referenced to make the answer more aligned with the source material.\n\nThus, the answer, although conceptually correct, does not appropriately leverage the provided document.\n\nFinal result: Bad"</li><li>"The answer is partially correct since it provides the most important guidance on how to report car travel expenses for reimbursement. However, it contains inaccuracies and omissions that could cause confusion. \n\n1. The email addresses cited in the answer don’t match those in the document.\n2. There’s mention of requesting a parking card for a specific date (2004-04-14), which implies an inaccurate, irrelevant detail that might mislead the reader.\n3. The answer doesn't explicitly suggest that travel cost reimbursement is handled monthly, which is a crucial piece of information.\n\nConsidering these elements, the evaluation is as follows:\n\nEvaluation:\nThe essential details related to car travel reimbursement are present, but incorrect email addresses and irrelevant details might mislead or cause inconvenience for employees.\n\nThe final evaluation: Bad"</li></ul>

Evaluation

Metrics

LabelAccuracy
all0.6269

Uses

Direct Use for Inference

First install the SetFit library:

bash
pip install setfit

Then you can load this model and run inference.

python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_cot-few_shot_only_reasoning_1726750062.370463")
# Run inference
preds = model("The answer succinctly addresses the question by stating that finance@ORGANIZATION_2.<89312988> should be contacted for questions about travel reimbursement. This is correctly derived from the provided document, which specifies that questions about travel costs and reimbursements should be directed to the finance email.

Final evaluation: Good")

<!--

Downstream Use

List how someone could finetune this model on their own dataset. -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Set Metrics

Training setMinMedianMax
Word count3085.7538210
LabelTraining Sample Count
032
133

Training Hyperparameters

  • —batch_size: (16, 16)
  • —num_epochs: (5, 5)
  • —max_steps: -1
  • —sampling_strategy: oversampling
  • —num_iterations: 20
  • —bodylearningrate: (2e-05, 2e-05)
  • —headlearningrate: 2e-05
  • —loss: CosineSimilarityLoss
  • —distancemetric: cosinedistance
  • —margin: 0.25
  • —endtoend: False
  • —use_amp: False
  • —warmup_proportion: 0.1
  • —l2_weight: 0.01
  • —seed: 42
  • —evalmaxsteps: -1
  • —loadbestmodelatend: False

Training Results

EpochStepTraining LossValidation Loss
0.006110.2304-
0.3067500.2556-
0.61351000.244-
0.92021500.1218-
1.22702000.0041-
1.53372500.0022-
1.84053000.0017-
2.14723500.0017-
2.45404000.0015-
2.76074500.0014-
3.06755000.0013-
3.37425500.0013-
3.68106000.0012-
3.98776500.0012-
4.29457000.0012-
4.60127500.0012-
4.90808000.0012-

Framework Versions

  • —Python: 3.10.14
  • —SetFit: 1.1.0
  • —Sentence Transformers: 3.1.0
  • —Transformers: 4.44.0
  • —PyTorch: 2.4.1+cu121
  • —Datasets: 2.19.2
  • —Tokenizers: 0.19.1

Citation

BibTeX

bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->