saidsef/knowledge-base-docs

Knowledge Base Documentation Dataset

A comprehensive, pre-processed and vectorized dataset containing documentation from 25+ popular open-source projects and cloud platforms, optimized for Retrieval-Augmented Generation (RAG) applications.

📊 Dataset Overview

This dataset aggregates technical documentation from leading open-source projects across cloud-native, DevOps, machine learning, and infrastructure domains. Each document has been chunked and embedded using the all-MiniLM-L6-v2 sentence transformer model.

Dataset ID: saidsef/knowledge-base-docs

🎯 Sources

The dataset includes documentation from the following projects:

📋 Dataset Schema

Each row in the dataset contains the following fields:

🔧 Dataset Creation Process

1. Data Collection

• —Shallow clone of 25+ GitHub repositories
• —Extraction of documentation files (.md, .mdx, .rst)

2. Content Processing

• —Removal of YAML frontmatter
• —Conversion to LLM-friendly markdown format
• —Stripping of scripts, styles, and media elements
• —Code block preservation with proper formatting

3. Text Chunking

• —Chunk size: 500 words
• —Overlap: 50 words
• —Ensures semantic continuity across chunks

4. Vectorization

• —Model: all-MiniLM-L6-v2
• —Embedding dimensions: 384
• —Normalization: Enabled for cosine similarity
• —Pre-computed embeddings for fast retrieval

5. Storage Format

• —Format: Apache Parquet
• —Compression: Optimized for query performance
• —File: knowledge_base.parquet

💻 Usage Examples

Loading the Dataset

import pandas as pd
from datasets import load_dataset

# From Hugging Face Hub
dataset = load_dataset("saidsef/knowledge-base-docs")
df = dataset['train'].to_pandas()

# From local Parquet file
df = pd.read_parquet("knowledge_base.parquet", engine="pyarrow")

Semantic Search / RAG Implementation

import numpy as np
from sentence_transformers import SentenceTransformer

# Load the same model used for embedding
model = SentenceTransformer('all-MiniLM-L6-v2', trust_remote_code=True)

def retrieve(query, df, k=5):
    """Retrieve top-k most relevant documents using cosine similarity"""
    # Encode the query
    query_vec = model.encode(query, normalize_embeddings=True)
    
    # Convert embeddings to matrix
    embeddings_matrix = np.vstack(df['embeddings'].values)
    
    # Calculate cosine similarity
    norms = np.linalg.norm(embeddings_matrix, axis=1) * np.linalg.norm(query_vec)
    scores = np.dot(embeddings_matrix, query_vec) / norms
    
    # Add scores and sort
    df['score'] = scores
    return df.sort_values(by='score', ascending=False).head(k)

# Example query
results = retrieve("How do I configure an nginx ingress controller?", df, k=3)
print(results[['content', 'score']])

Building a RAG Pipeline

from transformers import pipeline

# Load a question-answering model
qa_pipeline = pipeline("question-answering", model="distilbert-base-cased-distilled-squad")

def rag_answer(question, df, k=3):
    """RAG: Retrieve relevant context and generate answer"""
    # Retrieve relevant documents
    context_rows = retrieve(question, df, k=k)
    context_text = " ".join(context_rows['content'].tolist())
    
    # Generate answer
    result = qa_pipeline(question=question, context=context_text)
    return result['answer'], context_rows

answer, sources = rag_answer("What is a Kubernetes pod?", df)
print(f"Answer: {answer}")

📈 Dataset Statistics

# Total chunks
print(f"Total chunks: {len(df)}")

# Average chunk length
df['chunk_length'] = df['content'].apply(lambda x: len(x.split()))
print(f"Average chunk length: {df['chunk_length'].mean():.0f} words")

# Embedding dimensionality
print(f"Embedding dimensions: {len(df['embeddings'].iloc[0])}")

🚀 Use Cases

• —RAG Applications: Build retrieval-augmented generation systems
• —Semantic Search: Find relevant documentation across multiple projects
• —Question Answering: Create technical support chatbots
• —Documentation Assistant: Help developers navigate complex documentation
• —Learning Resources: Train models on high-quality technical content
• —Comparative Analysis: Compare documentation approaches across projects

🔍 Performance Considerations

• —Pre-computed embeddings: No need for runtime encoding
• —Optimized retrieval: Matrix multiplication for fast cosine similarity
• —Parquet format: Efficient storage and query performance
• —Chunk overlap: Better context preservation across boundaries

🛠️ Requirements

pandas>=2.0.0
numpy>=1.24.0
sentence-transformers>=2.0.0
pyarrow>=12.0.0
datasets>=2.0.0

📝 License

This dataset is a compilation of documentation from various open-source projects. Each source maintains its original license:

• —Most projects use Apache 2.0 or MIT licenses
• —Refer to individual project repositories for specific licensing terms

🤝 Contributing

To add new sources or update existing documentation:

1. 1.Add the source configuration to the sites list
2. 2.Run the data collection pipeline
3. 3.Verify content processing and embedding quality
4. 4.Submit a pull request with updated dataset

📧 Contact

For questions, issues, or suggestions, please open an issue on the GitHub repository or contact the maintainer.

🙏 Acknowledgments

Special thanks to all the open-source projects that maintain excellent documentation, making this dataset possible.

Last Updated: December 2025 Version: 1.0 Embedding Model: all-MiniLM-L6-v2 Total Sources: 25+