Datasetpublic

stefan-it/span-marker-base-model-detection

SpanMarker Base Model Detection It is relative simply to determine base model of a fine-tuned SpanMarker model: import os from huggingface_hub import login, HfApi hf_token = os.environ.get("HF_TOKEN") login(token=hf_token, add_to_git_credential=True) api = HfApi() Please make sure that HF_TOKEN is set as environment variable. After that, list of all SpanMarker models can be retrieved and configuration file is parsed. Please make sure that span-marker library is installed:… See the full description on the dataset page: https://huggingface.co/datasets/stefan-it/span-marker-base-model-detection.

sourceHugging Facemitupdated 3y agoView on Hugging Face

0likes6downloads

Dataset Card

SpanMarker Base Model Detection

It is relative simply to determine base model of a fine-tuned SpanMarker model:

python

import os

from huggingface_hub import login, HfApi

hf_token = os.environ.get("HF_TOKEN")
login(token=hf_token, add_to_git_credential=True)
api = HfApi()

Please make sure that HF_TOKEN is set as environment variable.

After that, list of all SpanMarker models can be retrieved and configuration file is parsed. Please make sure that span-marker library is installed:

python

from span_marker import SpanMarkerConfig

f_out = open("span_marker_base_model_detection.csv", "wt")

f_out.write("Nr,Model ID,Base Model ID\n")

counter = 1
for span_marker_model in api.list_models(filter="span-marker"):
    try:
        config = SpanMarkerConfig.from_pretrained(span_marker_model.modelId)

        base_model = config.encoder["_name_or_path"]
        
        f_out.write(f"{counter},{span_marker_model.modelId},{base_model}\n")
        
        counter +=1
    except Exception as e:
        print(e)
f_out.close()