Datasetpublic

ashvardanian/StringWars

StringKilla - Small Datasets for String Algorithms Benchmarking The goal of this dataset is to provide a fairly diverse set of strings to evalute the performance of various string-processing algorithms in StringZilla and beyond. English Texts English Leipzig Corpora Collection 124 MB uncompressed 1'000'000 lines of ASCII 8'388'608 tokens of mean length 5 The dataset was originally pulled from Princeton's website: wget --no-clobber -O leipzig1M.txt… See the full description on the dataset page: https://huggingface.co/datasets/ashvardanian/StringWars.

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face

1likes136downloads

Dataset Card

StringKilla - Small Datasets for String Algorithms Benchmarking

The goal of this dataset is to provide a fairly diverse set of strings to evalute the performance of various string-processing algorithms in StringZilla and beyond.

English Texts

English Leipzig Corpora Collection

—124 MB uncompressed
—1'000'000 lines of ASCII
—8'388'608 tokens of mean length 5

The dataset was originally pulled from Princeton's website:

wget --no-clobber -O leipzig1M.txt https://introcs.cs.princeton.edu/python/42sort/leipzig1m.txt

Pull it with:

huggingface-cli download ashvardanian/StringKilla leipzig1M.txt

Hutter Prize Compression Dataset

—1 GB uncompressed, 0.3 GB originally
—13'147'025 lines of ASCII
—67'108'864 tokens of mean length 6

The dataset was originally Matt Mahoney's page:

wget --no-clobber -O enwik9.zip http://mattmahoney.net/dc/enwik9.zip
unzip enwik9.zip && rm enwik9.zip && mv enwik9 enwik9.txt

Pull it with:

huggingface-cli download ashvardanian/StringKilla enwik9.txt

Multilingual Texts

UTF8 Decoder Stress Test

—14 KB uncompressed
—204 lines of UTF8

The dataset was originally pulled from an Martin Dürst's HTML adaptation of Markus Kuhn's UTF8 stress test:

wget --no-clobber -O utf8.txt https://www.w3.org/2001/06/utf-8-test/UTF-8-demo.html

Pull it with:

huggingface-cli download ashvardanian/StringKilla utf8.txt

XLSum Extractive Summarization Benchmark

—4.7 GB uncompressed, 1.7 GB originally
—1'004'598 lines of UTF8
—268'435'456 tokens of mean length 8

The dataset was originally aggregates from "Bangladesh University of Engineering and Technology" GitHub repo:

wget --no-clobber -O xlsum.csv.gz https://github.com/ashvardanian/xl-sum/releases/download/v1.0.0/xlsum.csv.gz
gzip -d xlsum.csv.gz

Pull it with:

huggingface-cli download ashvardanian/StringKilla xlsum.csv

Bioinformatics

DNA Strings

Filename	String Length	Lines Count
acgt_100.txt	100	100'000
acgt_1k.txt	1'000	100'000
acgt_10k.txt	10'000	100'000
acgt_100k.txt	100'000	1000
acgt_1m.txt	1'000'000	1200
acgt_10m.txt	10'000'000	1200

Pull them with:

huggingface-cli download ashvardanian/StringKilla acgt_100.txt
huggingface-cli download ashvardanian/StringKilla acgt_1k.txt
huggingface-cli download ashvardanian/StringKilla acgt_10k.txt
huggingface-cli download ashvardanian/StringKilla acgt_100k.txt
huggingface-cli download ashvardanian/StringKilla acgt_1m.txt
huggingface-cli download ashvardanian/StringKilla acgt_10m.txt