cbdb/MetaDis

1---2language:3- zh4tags:5- SequenceClassification6- MetaDis7- 古文8- 文言文9- ancient10- classical11- Biography12- 古代人物传记13license: cc-by-nc-sa-4.014---15 16# <font color="IndianRed"> MetaDis (Classical Chinese Biographical Metadata Disambiguation)</font>17[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1UcyhdfX5_NuZ87XR1fUmMACJ7qY-nn-P#scrollTo=cd-iH6OLpIeV)18 19 20 21Download <font color="IndianRed">template excel sheet</font> from here: https://huggingface.co/cbdb/MetaDis/blob/main/template.xlsx 22 23 24---25Welcome to the repository for MetaDis, a specialized model designed for disambiguating biographical metadata within Classical Chinese texts.26 27At the core of the problem MetaDis aims to solve is a common issue researchers encounter when studying historical texts - the identification of individuals sharing the same name. Are these instances referring to the same person or two different people? This is the question MetaDis seeks to answer.28 29MetaDis is based on the `AutoModelForNextSentencePrediction` architecture, a machine learning model that processes two sequences of data as its input. It then outputs a 0 or 1 - a binary representation indicating whether or not the two sequences refer to the same person. Here, 0 represents 'not the same person', and 1 indicates 'the same person'.30 31---32 33### <font color="IndianRed">Input Data Formatting </font>34 35In order to ensure the highest accuracy and performance of the MetaDis model, we've specifically designed an input format based on the data the model was originally trained on. This is crucial as it allows the model to accurately interpret and process your data.36 37To assist you in this process, we've provided a template Excel (.xlsx) file. We recommend downloading this template and inputting your data directly into it, ensuring your data matches the same format as the model's training data. 38 39To download our Excel data template, please click [here](https://huggingface.co/cbdb/MetaDis/blob/main/template.xlsx).40 41--- 42 43### <font color="IndianRed">Code Demonstration: Loading and Using MetaDis Model </font>44 45The following section demonstrates how to directly load the MetaDis model and use it for predicting whether two sets of biographical information refer to the same person or not.46 47Please ensure that you have the `transformers` library installed in your Python environment. If not, you can install it using pip:48 49```python50pip install transformers51```52 53Now, let's load our model and make some predictions:54 55```python56# Import necessary libraries from HuggingFace Transformers57from transformers import AutoTokenizer, AutoModelForNextSentencePrediction58import torch59 60# Load our tokenizer and model61tokenizer = AutoTokenizer.from_pretrained("cbdb/MetaDis")62model = AutoModelForNextSentencePrediction.from_pretrained("cbdb/MetaDis")63 64# Define our sentences to compare65sentence1 = ['first biographical information of person name A', 'first biographical information of person name B']66sentence2 = ['second biographical information of person name A', 'first biographical information of person name B']67 68# Loop through each sentence pair69for s1, s2 in zip(sentence1, sentence2):70    # Prepare the inputs71    encoding = tokenizer(s1, s2, truncation=True, padding=True, return_tensors='pt')72 73    # Move the inputs to the device where the model is74    for key in encoding:75        encoding[key] = encoding[key].to(model.device)76 77    # Make the prediction78    outputs = model(**encoding)79 80    # Extract the prediction81    logits = outputs.logits82    preds = torch.argmax(logits, dim=-1)83 84    # Display the results85    if preds.item() == 1:86        print('Same person')87        print(s1, s2)88    else:89        print('Different person')90        print(s1, s2)91```92 93This code demonstration shows how you can load our MetaDis model, prepare inputs in the necessary format, and extract predictions to determine if the biographical details refer to the same person or different individuals. Remember to replace the example sentences with your own data.94 95Remember to include a link or instructions on how users can install the `transformers` library if they don't already have it installed.96 97---98 99### <font color="IndianRed">Authors </font>100Queenie Luo (queenieluo[at]g.harvard.edu)101<br>102Hongsu Wang103<br>104Peter Bol105<br>106CBDB Group107 108### <font color="IndianRed">License </font>109Copyright (c) 2023 CBDB110 111Except where otherwise noted, content on this repository is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).112To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/ or113send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.