huzey/claude-skills-chunk

huzey/claude-skills-chunk

Rule-based Markdown chunks derived from huzey/claude-skills (source file: claude-skills.gpt_summary.parquet).

Settings

• —--include-heading-path-in-text: true
• —--include-frontmatter-unit: true
• —--primary-level-strategy: highest
• —--max-primary-level: 3
• —--max-chars: 4000
• —--min-chars: 800
• —--tldr: true (synthetic TL;DR chunk at chunk_index_in_doc=0)

Columns Added From Source Dataset

These columns are copied from the source dataset huzey/claude-skills (per skill, then repeated for every chunk row):

• —skills_sh_id (string): stable id, typically <repo>/<slug>
• —github_stars (int)
• —skills_sh_total_installs (int, may be null)
• —skills_sh_weekly_installs (int, may be null)

Important: the HF dataset page metrics (downloads/likes) are not used here. The stats above come from the data columns in huzey/claude-skills.

Export Notes

The source parquet contains exact duplicate document rows and also multiple variants for the same (repo, name). To keep identifiers stable and unique:

• —doc_uid is a SHA1 over (repo, name, split, domain_category, description, sha1(full_content)).
• —Exact duplicate rows (same inputs above) are skipped.
• —chunk_uid is a SHA1 over (doc_uid, chunk_id, chunk_index_in_doc).

Run summary:

• —input_rows=22862
• —unique_docs=21017
• —skipped_dup_docs=1845
• —chunks=368513
• —shards=4

Files

Parquet shards are stored under data/ as train-00000-of-000NN.parquet.

Columns (Per Chunk Row)

Key fields:

• —name, repo, skills_sh_id
• —github_stars, skills_sh_total_installs, skills_sh_weekly_installs
• —domain_category, split, description
• —doc_uid, chunk_uid, chunk_id, chunk_index_in_doc, unit_kind
• —heading_path_titles, heading_path_levels
• —body_start_line, body_end_line
• —char_len, text

TL;DR Format

For unit_kind=tldr rows, the chunk text is:

TL;DR
<Title>：<What>

Embeddings

• —qwen3emb_chunk_text (fixedsizelist[float16], dim=4096): embedding of the per-chunk text field using Qwen/Qwen3-VL-Embedding-8B.
• —instruction (system prompt): Embed this text from a subsection of an AI agent skill.md file
• —max_length: 2048
• —maxtextchars (pre-truncate, head-only): 50000
• —torch dtype: fp16 (compute), stored as fp16

• —qwen3emb_chunk_text_with_tldr (fixedsizelist[float16], dim=4096): embedding of per-chunk text augmented with the per-document TL;DR (for non-TL;DR rows).
• —instruction (system prompt): Embed this text from a subsection of an AI agent skill.md file, an overview of this skill is provided in TL;DR
• —input text: TL;DR + "\n\n" + chunk_text (if unit_kind=tldr, use chunk_text only)
• —max_length: 2048
• —maxtextchars (pre-truncate, head-only): 50000
• —torch dtype: fp16 (compute), stored as fp16

• —qwen3emb_level2_chunk_text (struct): embedding representing the level2 section that a row belongs to.
• —fields: ref_chunk_uid (string), emb (list[float16], length=4096 when present; null otherwise)
• —semantics:
• —if a level2 section has multiple level3 chunks: concatenate those level3 text values and embed; emb is stored on the first level3 row, other rows point to it via ref_chunk_uid.
• —otherwise: no new embedding is computed; ref_chunk_uid points to the row itself, meaning you should reuse its existing qwen3emb_chunk_text.
• —qwen3emb_level2_chunk_text_with_tldr (struct): same as above, but using TL;DR-augmented inputs (reusing qwen3emb_chunk_text_with_tldr when no new level2 embedding is computed).