Frazic/udever-bloom-3b-sentence

Model Card for udever-bloom

<!-- Provide a quick summary of what the model is/does. -->

udever-bloom-3b is finetuned from bigscience/bloom-3b via BitFit on MS MARCO Passage Ranking, SNLI and MultiNLI data. It is a universal embedding model across tasks, natural and programming languages. (From the technical view, udever is merely with some minor improvements to sgpt-bloom)

<div align=center><img width="338" height="259" src="https://user-images.githubusercontent.com/26690193/277643721-cdb7f227-cae5-40e1-b6e1-a201bde00339.png" /></div>

Model Details

Model Description

• —Developed by: Alibaba Group
• —Model type: Transformer-based Language Model (decoder-only)
• —Language(s) (NLP): Multiple; see bloom training data
• —Finetuned from model : bigscience/bloom-3b

Model Sources

<!-- Provide the basic links for the model. -->

• —Repository: github.com/izhx/uni-rep
• —Paper : Language Models are Universal Embedders
• —Training Date : 2023-06

How to Get Started with the Model

Use the code below to get started with the model.

import torch
from transformers import AutoTokenizer, BloomModel

tokenizer = AutoTokenizer.from_pretrained('izhx/udever-bloom-3b')
model = BloomModel.from_pretrained('izhx/udever-bloom-3b')

boq, eoq, bod, eod = '[BOQ]', '[EOQ]', '[BOD]', '[EOD]'
eoq_id, eod_id = tokenizer.convert_tokens_to_ids([eoq, eod])

if tokenizer.padding_side != 'left':
    print('!!!', tokenizer.padding_side)
    tokenizer.padding_side = 'left'


def encode(texts: list, is_query: bool = True, max_length=300):
    bos = boq if is_query else bod
    eos_id = eoq_id if is_query else eod_id
    texts = [bos + t for t in texts]
    encoding = tokenizer(
        texts, truncation=True, max_length=max_length - 1, padding=True
    )
    for ids, mask in zip(encoding['input_ids'], encoding['attention_mask']):
        ids.append(eos_id)
        mask.append(1)
    inputs = tokenizer.pad(encoding, return_tensors='pt')
    with torch.inference_mode():
        outputs = model(**inputs)
        embeds = outputs.last_hidden_state[:, -1]
    return embeds

encode(['I am Bert', 'You are Elmo'])

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

• —MS MARCO Passage Ranking, retrieved by (https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/msmarco/trainbi-encoder_mnrl.py#L86)
• —SNLI and MultiNLI (https://sbert.net/datasets/AllNLI.tsv.gz)

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Preprocessing

MS MARCO hard negatives provided by (https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/msmarco/trainbi-encoder_mnrl.py#L86). Negatives for SNLI and MultiNLI are randomly sampled.

Training Hyperparameters

• —Training regime: tf32, BitFit
• —Batch size: 1024
• —Epochs: 3
• —Optimizer: AdamW
• —Learning rate: 1e-4
• —Scheduler: constant with warmup.
• —Warmup: 0.25 epoch

Evaluation

Table 1: Massive Text Embedding Benchmark MTEB

Table 2: CodeSearchNet

Table 3: Chinese multi-domain retrieval Multi-cpr

More results refer to paper section 3.

Technical Specifications

Model Architecture and Objective

• —Model: bigscience/bloom-3b.
• —Objective: Constrastive loss with hard negatives (refer to paper section 2.2).

Compute Infrastructure

• —Nvidia A100 SXM4 80GB.
• —torch 2.0.0, transformers 4.29.2.

Citation

BibTeX:

@article{zhang2023language,
  title={Language Models are Universal Embedders},
  author={Zhang, Xin and Li, Zehan and Zhang, Yanzhao and Long, Dingkun and Xie, Pengjun and Zhang, Meishan and Zhang, Min},
  journal={arXiv preprint arXiv:2310.08232},
  year={2023}
}