plaguss/argilla_sdk_docs_raw_unstructured

Dataset info This dataset contains documentation chunks from repositories (ADD REPOS). Postprocessing After some inspection, some chunks contain text too short to be meaningful, so we decided to remove those by removing chunks whose number of tokens (computed with the same tokenizer of the model to be used for the embeddings) is lower or equal to the 5%: from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("BAAI/bge-base-en-v1.5") df =… See the full description on the dataset page: https://huggingface.co/datasets/plaguss/argilla_sdk_docs_raw_unstructured.