protibimbo/ManikKatha

ManikKatha — Manik Bandopadhyay Bengali Literary Corpus

Bengali literary prose by Manik Bandopadhyay (1908-1956).

Seven novels: Padma Nadir Majhi (1936), Dibaratrir Kabya (1936), Chatushkon (1942), Majhir Chhele (1959), and Sahartali, Ahingsa and Darpan, the three collected in the 1965 selection Sera Manik. Novels are split into their printed chapters.

Sixty-one short stories: all fifty-eight of Uttarkaler Galpa-Sangraha (2nd edition, 1964) in their published order, plus Atmahatyar Adhikar, Pragaitihasik and Sarisrip from Sera Manik. Each story is one whole record, and no story appears twice.

Every record carries its work, chapter or story title, the collection it was gathered from, the year of first publication where the source volumes record it, and the page-level OCR confidence scores that produced the text, so consumers can filter to the quality level their use case needs.

At a glance

Each record is one chapter of a novel or one complete short story, so records carry whole narrative arcs rather than fragments.

Text is uncorrected OCR. Rather than silently shipping the errors, every record carries the OCR confidence scores that produced it, so you can filter to whatever quality level your use case needs — see Filtering by quality below.

Contents

Long-form works

Short stories

One record each; collection_bn / collection_en name the volume they were printed in.

Record size distribution

Sources

Every record traces back to a scanned printed edition on the Internet Archive.

Schema

conf_* columns are null, never 0.0, when confidence is genuinely unknown — a fabricated zero would read as worst-possible quality and silently drop good text.

Filtering by quality

from datasets import load_dataset

ds = load_dataset("protibimbo/ManikKatha", split="train")

# 1. drop whole records that are mostly noise
good = ds.filter(lambda r: (r["conf_mean"] or 0) >= 75.0)

# 2. or excise only the bad pages, keeping the rest of the chapter
def drop_bad_pages(rec, floor=70.0):
    keep = [rec["text"][p["char_start"]:p["char_end"]]
            for p in rec["pages"] if (p["conf_mean"] or 0) >= floor]
    return {"text": "".join(keep)}

clean = ds.map(drop_bad_pages)

The pages spans are contiguous and tile text exactly, so slicing and rejoining them never loses or duplicates characters.

Intended use

Continued pretraining and LoRA/PEFT style tuning on next-token prediction — the records are raw prose, not instruction pairs. Text is not pre-chunked to a token window: packing to 2048/4096 with an EOS at document boundaries is a training-time decision, and pre-chunking would sever sentences and take that choice away. n_tokens_est is provided so you can plan packing without tokenizing first.

There is a single train split. For style evaluation, hold out a work yourself (filtering on work_id) so the held-out text is genuinely unseen.

# hold out one novel for evaluation
train = ds.filter(lambda r: r["work_id"] != "aranyak")
eval_ = ds.filter(lambda r: r["work_id"] == "aranyak")

How the text was produced

1. 1.Extract — text and per-word confidence are read from the Internet Archive hOCR derivative of each scan. The PDF's own invisible text layer carries the same characters but no confidence, so hOCR is preferred wherever an item exists.
2. 2.Normalise — Unicode NFC. Zero-width joiners are preserved: in Bengali they carry the reph / ya-phala distinction, so stripping them would change words.
3. 3.Strip page furniture — running headers and page numbers are removed. The same running title is OCR'd differently on nearly every page, so headers are detected by fuzzy-clustering candidate lines rather than by exact match.
4. 4.Reflow — printed line breaks are rejoined into paragraphs, including across page boundaries. Paragraphs are separated by \n\n.
5. 5.Segment — split into works by verified page ranges, then into chapters by printed headings.

Limitations

• —OCR is uncorrected (tesseract 5.0.0-alpha-20201231-10-g1236, tesseract 5.3.0-6-g76ae). Mean record confidence is 84.0/100, and 0 of 105 records fall below 75. Expect wrong conjuncts, stray Latin characters and mangled punctuation. The text is good enough to learn style and syntax from, and not a reliable edition to quote from.
• —Front matter, editorial prefaces and critical appendices were excluded by page range, but header and page-number removal is heuristic and a few survive mid-text.
• —Paragraph boundaries are reconstructed from printed line breaks and are approximate; dialogue segments more reliably than continuous narration.
• —Chapter splits come from printed headings. Books whose headings did not survive OCR appear as a single long record rather than being split arbitrarily.
• —Scans are of mid-century reprints, not first editions, so spelling reflects the printing house's conventions of the day.
• —Single author and single genre: this is literary prose from one writer, and is not a general-purpose Bengali corpus.

Provenance and rights

Scans and OCR come from the Internet Archive; see ia_identifier and source_url on each record, and the Sources table above.

Cataloguing note: Internet Archive/DLI author fields for this collection are not always reliable. Attribution here was checked against each scan's own title page, not taken from the metadata record.

Public domain in India: the author died in 1956, so copyright expired in 2016 under life + 60. US status is unresolved. Manik's career began in 1928 and every work here was first published in 1929 or later, so the URAA may have restored US copyright in all of it — there is no date filter that separates a clear subset from a risky one, as there is for this project's Tagore corpus. This is published as a known and accepted risk, not as a cleared position; confirm before relying on it. first_published is set on 62 of 105 records. For the four standalone novels it is taken from the edition's own imprint. For the 58 Uttarkaler Galpa-Sangraha stories it is the year the story first appeared in book form, taken from the গল্প পরিচয় in that volume; a story may have appeared in a periodical earlier. It is null for Sahartali, Ahingsa, Darpan and the three stories taken from Sera Manik, whose scan records no dates.