Chengfengke/herbert

1---2license: apache-2.03base_model:4- google-bert/bert-base-chinese5metrics:6- accuracy7language:8- en9- zh10pipeline_tag: fill-mask11---12# Herbert: Pretrained Bert Model for Herbal Medicine13 14**Herbert** is a pretrained model for herbal medicine research, developed based on the `bert-base-chinese` model. The model has been fine-tuned on domain-specific data from 675 ancient books and 32 Traditional Chinese Medicine (TCM) textbooks. It is designed to support a variety of TCM-related NLP tasks.15 16---17 18## Introduction19 20This model is optimized for TCM-related tasks, including but not limited to:21- Herbal formula encoding22- Domain-specific word embedding23- Classification, labeling, and sequence prediction tasks in TCM research24 25Herbert combines the strengths of modern pretraining techniques and domain knowledge, allowing it to excel in TCM-related text processing tasks.26 27---28 29## Model Config30 31```json32{33  "hidden_size": 1024,34  "max_position_embeddings": 512,35  "model_type": "bert",36  "num_attention_heads": 16,37  "num_hidden_layers": 24,38  "torch_dtype": "float32",39  "vocab_size": 2112840}41### requirements42"transformers_version": "4.45.1"43 44###  Quickstart45 46#### Use Huggingface47```python48from transformers import AutoTokenizer, AutoModel49 50# Replace "Chengfengke/herbert" with the Hugging Face model repository name51model_name = "Chengfengke/herbert"52 53# Load tokenizer and model54tokenizer = AutoTokenizer.from_pretrained(model_name)55model = AutoModel.from_pretrained(model_name)56 57# Input text58text = "中医理论是我国传统文化的瑰宝。"59 60# Tokenize and prepare input61inputs = tokenizer(text, return_tensors="pt", truncation=True, padding="max_length", max_length=128)62 63# Get the model's outputs64with torch.no_grad():65    outputs = model(**inputs)66 67# Get the embedding (sentence-level average pooling)68sentence_embedding = outputs.last_hidden_state.mean(dim=1)69 70print("Embedding shape:", sentence_embedding.shape)71print("Embedding vector:", sentence_embedding)72```73 74 75#### LocalModel76```python77from transformers import BertTokenizer, BertForMaskedLM78 79# Load the model and tokenizer80model_name = "Chengfengke/herbert"81tokenizer = BertTokenizer.from_pretrained(model_name)82model = BertForMaskedLM.from_pretrained(model_name)83inputs = tokenizer("This is an example text for herbal medicine.", return_tensors="pt")84outputs = model(**inputs)85```86 87## Citation88 89If you find our work helpful, feel free to give us a cite.90 91```bibtex92@misc{herbert-embedding,93  title = {Herbert: A Pretrain_Bert_Model for TCM_herb and downstream Tasks as Text Embedding Generation},94  author = {Yehan Yang,Xinhan Zheng},95  month = {December},96  year = {2024}97}98 99@article{herbert-technical-report,100  title={Herbert: A Pretrain_Bert_Model for TCM_herb and downstream Tasks as Text Embedding Generation},101  author={Yehan Yang,Xinhan Zheng},102  institution={Beijing Angopro Technology Co., Ltd.},103  year={2024},104  note={Presented at the 2024 Machine Learning Applications Conference (MLAC)}105}106