Datasetpublic

bleugreen/typescript-chunks

typescript-chunks A dataset of TypeScript snippets, processed from the typescript subset of the-stack-smol. Processing Each source file is parsed with the TypeScript AST and queried for 'semantic chunks' of the following types. FunctionDeclaration ---- 8205 ArrowFunction --------- 33890 ClassDeclaration ------- 5325 InterfaceDeclaration -- 12884 EnumDeclaration --------- 518 TypeAliasDeclaration --- 3580 MethodDeclaration ----- 24713 Leading comments are added… See the full description on the dataset page: https://huggingface.co/datasets/bleugreen/typescript-chunks.

sourceHugging Faceupdated 3y agoView on Hugging Face

3likes60downloads

Dataset Card

typescript-chunks

A dataset of TypeScript snippets, processed from the typescript subset of the-stack-smol.

Processing

—Each source file is parsed with the TypeScript AST and queried for 'semantic chunks' of the following types.

FunctionDeclaration ---- 8205
ArrowFunction --------- 33890
ClassDeclaration ------- 5325
InterfaceDeclaration -- 12884
EnumDeclaration --------- 518
TypeAliasDeclaration --- 3580
MethodDeclaration ----- 24713

—Leading comments are added to the front of content
—Removed all chunks over max sequence length (2048)
—Deduplicated / cleaned up
—Generated instructions / summaries with gpt-3.5-turbo (in progress)

Dataset Structure

python

from datasets import load_dataset
load_dataset("bleugreen/typescript-chunks")

DatasetDict({
    train: Dataset({
        features: ['type', 'content', 'repo', 'path', 'language'],
        num_rows: 89115
    })
})