Datasetpublic

Programmer-RD-AI/sfia-9-chunks

sfia-9-chunks Dataset Overview The sfia-9-chunks dataset is a derived dataset from sfia-9-scraped. It uses sentence embeddings and hierarchical clustering to split each SFIA-9 document into coherent semantic chunks. This chunking facilitates more efficient downstream tasks like semantic search, question answering, and topic modeling. Chunking Methodology We employ the following procedure to generate chunks: from sentence_transformers import… See the full description on the dataset page: https://huggingface.co/datasets/Programmer-RD-AI/sfia-9-chunks.

sourceHugging Facecc-by-nc-4.0updated 1y agoView on Hugging Face

1likes6downloads

Dataset Card

sfia-9-chunks Dataset

Overview

The sfia-9-chunks dataset is a derived dataset from `sfia-9-scraped`. It uses sentence embeddings and hierarchical clustering to split each SFIA-9 document into coherent semantic chunks. This chunking facilitates more efficient downstream tasks like semantic search, question answering, and topic modeling.

Chunking Methodology

We employ the following procedure to generate chunks:

python

from sentence_transformers import SentenceTransformer
from sklearn.cluster import AgglomerativeClustering

MODEL_NAME = "all-MiniLM-L12-v2"

def get_chunks(document: str) -> list[str]:
    model = SentenceTransformer(MODEL_NAME)
    sentences = document.split(". ")
    embs = model.encode(sentences)
    clustering = AgglomerativeClustering(n_clusters=None, distance_threshold=1.0).fit(
        embs
    )
    chunks = []
    for label in set(clustering.labels_):
        group = [
            sentences[i]
            for i in range(len(sentences))
            if clustering.labels_[i] == label
        ]
        chunks.append(". ".join(group))
    return chunks

—Model: all-MiniLM-L12-v2 for efficient sentence embeddings.
—Clustering: Agglomerative clustering with a distance threshold of 1.0 to dynamically determine the number of semantic groups.

Dataset Structure

This dataset consists of a flat list of semantic text chunks derived from the SFIA-9 documents. When loaded, each example is an object with a single field:

—chunks (string): One coherent semantic chunk extracted via hierarchical clustering.

Usage Example

python

from datasets import load_dataset

dataset = load_dataset("Programmer-RD-AI/sfia-9-chunks")

for example in dataset:
    print(example["chunks"])

License

This dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. You can view the full license at: https://creativecommons.org/licenses/by/4.0/

Citation

If you use this dataset in your research, please cite:

bibtex

@misc{ranuga_disansa_gamage_2025,
	author       = {Ranuga Disansa Gamage},
	title        = {sfia-9-chunks (Revision 035dc41)},
	year         = 2025,
	url          = {https://huggingface.co/datasets/Programmer-RD-AI/sfia-9-chunks},
	doi          = {10.57967/hf/5747},
	publisher    = {Hugging Face}
}