hyper-efficient-system-llc/bosnian-corpus-v1

Bosnian Corpus v1.0 (cleaned)

This dataset provides a cleaned and genre-annotated corpus of contemporary Bosnian, designed for quantitative analysis of language entropy, “language energy”, and modern NLP tasks.

The canonical release of this corpus is published on Zenodo: DOI: 10.5281/zenodo.17757098

Corpus composition

The corpus is built from three publicly available resources released via the CLARIN.SI repository:

1. 1.Sarajevo Corpus of SMS Messages in Bosnian 1.1
2. 2.Bosnian Web Corpus bsWaC 1.1
3. 3.Bosnian Web Corpus CLASSLA-web.bs 1.0

All sources were converted to plain text, cleaned, normalized, partially deduplicated, and merged into a single consistent dataset.

Size and statistics

• —Total size: ~6.18 GB (≈ 6,182,905,888 bytes)
• —Lines: ~46,258,935
• —Tokens: ~942,515,845

Genre structure

The web portion of the corpus is organized into the following “super-genres”:

• —News
• —Opinion
• —Forum / Chat
• —Info / HowTo
• —Legal / Administrative
• —Literature
• —Ads / Promo
• —Mix / Other

For each super-genre, a separate text file is provided, along with one global file concatenating all genres. The global file is intended for entropy estimation and language-model training.

Cleaning and normalization

Cleaning focuses on removing technical noise that would bias frequency distributions and entropy estimates, while preserving the linguistic signal:

• —Unicode normalization (UTF-8, NFC)
• —Correction of common mojibake artifacts
• —Removal of URLs, e-mail addresses, file names, boilerplate, CMS/navigation lines
• —Filtering of lines with a high proportion of non-letter characters
• —Optional digit normalization and lowercasing
• —Language filtering to keep primarily Bosnian text

Files

• —bosnian_corpus_all.txt — full corpus (all genres combined)
• —Per-genre text files:
• —news
• —opinion
• —forum_chat
• —info_howto
• —legal_admin
• —literature
• —ads_promo
• —mix_other
• —README.txt — dataset description

Associated research papers (Bosnian and English) are published separately on Zenodo.

• —Download: data/bosnian-corpus-1.0.zip

Code availability

Preprocessing, cleaning, and entropy-calculation scripts are available on GitHub:

https://github.com/H4sK0/bosnian-corpus-pipeline

License

This dataset is released under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Users must:

• —credit this Zenodo record and the original source corpora
• —distribute derivative datasets under the same or a compatible license

Citation

@dataset{kahrimanovic2025bosniancorpus,
  title   = {Bosnian Corpus (v1.0): Cleaned Web and SMS Text for Entropy and NLP Research},
  author  = {Kahrimanović, Hasan},
  year    = {2025},
  publisher = {Zenodo},
  doi     = {10.5281/zenodo.17757098}
}