damo-da/oag-nepal-audit-reports

OAG Nepal Audit Reports — Nepali transcripts and ruled tables

Machine-readable transcripts of 6,234 publications of the Office of the Auditor General of Nepal (महालेखा परीक्षकको कार्यालय, OAG) — the annual audit reports of local governments, provinces and central bodies, plus the OAG's own bulletins, journals and financial statements.

The OAG publishes these as PDFs whose text layer is, for most documents, legacy pre-Unicode Devanagari: fonts like Preeti and Fontasy Himali that map Nepali glyphs onto Latin code points, so an ordinary extractor returns Latin-looking gibberish. This dataset is the decoded, page-anchored text, plus the ruled audit tables as a cell grid — the itemised findings, voucher numbers and rupee amounts that are the substance of an audit report.

338,257 pages. 532,921,563 Devanagari characters. 443,344 ruled tables over 28,348,869 cells, 5,630,443 of them parsed numerics.

What is in it

Plus markdown/ — the 6,234 transcripts themselves, which are the primary artifact — pdf/ — the 6,236 source PDFs they were made from, 19.36 GiB — and MANIFEST.json, listing every file with its size and SHA-256.

By publication type

Local-level reports by province

Most-represented fiscal years (BS)

🛑 How to read the quality of this data

This is a decoded legacy-font corpus, not a clean digital text. Read these five limits before using it for anything that depends on a number being right.

1. Accuracy is an AGREEMENT rate against a second instrument, not a truth claim

The corpus was measured against an independent vision-OCR read of 390 pages sampled across quality strata — a reader that never touches the text layer, so it fails differently from font-table decoding. Corpus-weighted word agreement 76.2%. On the 171 of 390 pages where both instruments produced comparable amounts of text, clean-verdict pages agree at about 93%.

⚠️ Neither figure is "accuracy". Where the two instruments agree both are almost certainly right; where they disagree one is wrong and the pair only localises the error. The gap between 76% and 93% is mostly the corpus over-producing text, which is the defect described next.

🛑 Those figures were measured on `markdown-quality-v17-d22f13e13bfe9d0d`, NOT on this build's tree (`markdown-quality-v19-pii-redacted-r3`), and have not been re-measured since. They therefore predate whatever this build changed, and the de-duplication described below should move them upward — the same measurement predicted clean agreement near 93% once over-production was removed. Read 76.2% as a floor established on an earlier tree, not as this build's measured agreement.

2. Some pages still contain their own content twice

1 of 338,257 pages (0.000%) contain their own content twice — v1.3, published 2026-08-27 had 1. Duplicated Devanagari is well-formed Devanagari, so no character-level quality axis can see it — use the structural test (is the page's token stream exactly its own first half twice?) if it matters to you.

The residual is not unexplained. On 198 of 200 examined, a coarse detected table holds figures the finer grid lost, so de-duplicating would delete real rupee amounts. Leaving the page doubled is the cheaper error.

3. Digits are never folded across scripts, and you should not fold them either

The source typesetting mixes scripts: this build carries 45,722,539 Devanagari digits and 14,745,722 ASCII digits. २०७९ and 2079 are kept distinct everywhere, text is always verbatim, and value_num is filled only where a cell parses cleanly as a scalar — a fiscal year (२०७९।८०), a voucher number with a trailing danda, or a range is left null rather than guessed.

4. Per-page provenance: not every page came from the PDF's text layer

⚠️ A vision read substitutes whole consonants invisibly — a wrong character is still valid Devanagari, with no replacement character and no ratio change to give it away. Treat vision_ocr pages as the lower-confidence arm.

5. A few documents were never decoded at all, and you must be able to see which

17 documents contain zero Devanagari characters, and 13 of them score `clean`. That is not a contradiction and it is the clearest illustration on this card of what a verdict is worth: the corpus holds two entirely different populations that both come out with no Devanagari.

• —Most are English by design — OAG publishes unofficial English translations of its annual report summaries, and those are correct, complete and rightly clean.
• —0 are documents where likhit produced no usable text, so the markitdown fallback shipped the undecoded legacy bytes: text that looks like v n s T i s rather than Nepali. The audit catches these — they fail legacy_ascii and structure — but documents.engine reads markitdown+likhit for them, the same value as a document likhit decoded successfully.

So documents.engine_fallback (bool) and engine_fallback_reason are shipped to separate them. Filter on `engine_fallback == False` if you want text that was actually decoded.

🛑 These are not spread evenly, and in the worst-scoring collections they are the whole story:

The audit verdicts, as shipped

🛑 A clean verdict means no axis fired, which is weaker than "correct". The 11 axes that ran are canonical_words, duplication, legacy_ascii, matra_damage, mojibake, numeric_damage, ocr_commentary, repetition, repha_loss, spacing, structure — every one an internal well-formedness check on damaged conjuncts, replacement characters, spacing or numeric shape. None of them compares a transcript against the page it came from, so a systematically wrong but well-formed decode scores clean.

This corpus has shipped exactly that: a known malformed conjunct-ra family runs to thousands of occurrences across hundreds of documents while each document's own rate stays under every threshold, so the axis reports it and the verdict dilutes it away. The quality frame therefore ships the axis counts in *_detail_json, not only the verdicts, so you can set your own threshold instead of inheriting ours.

🛑 How much the verdict table flatters this corpus — two measurements

1. The clean rate is a property of which detectors were switched on. The audit ships with several calibrated detectors disabled, because enabling them relabels a published tree and that is the generation owner's call. Re-run over these exact bytes with every calibrated detector on (--mojibake-pua --mark-after-space --repha-malformed --repha-extended --duplication --page-refusal), the same 6,234 documents score 3,083 clean / 2,999 suspect / 152 garbled — 49.5%, against the 92.4% in the table above. 2,676 documents leave `clean` and none enters it. The largest single cause is private-use characters: 511,159 of them across 2,822 documents, counted in mojibake.private_use_chars on every row and not scored. The whole re-audit costs 43 seconds, and the counts you need are already in the quality frame.

2. A hand read of 50 `clean` documents found 1 faithful. Stratified random sample of 50 documents that score clean, drawn with weights summing to the full clean population, read against the source PDFs: 49 of 50 carry a defect that changes a word or a number, and 13 of 50 contain runs of prose that cannot be recovered at all — लेखापरीक्षण shipping as लेखापरी ण, on audit-opinion pages. Independently, of 21 unambiguously destructive mutations applied to a real transcript — deleting 80% of its pages, dropping every table row, shifting every digit, shuffling reading order — 18 still score `clean` with every detector enabled.

Neither result is a reason not to use this corpus; it is the best machine-readable form of these reports that exists. It is a reason to treat `clean` as "no axis fired" and nothing more, to prefer the raw counts over the verdicts, and to verify against pdf/ anything you are going to publish a number from.

The source materials

pdf/ carries all 6,236 source PDFs, 19.36 GiB, laid out as pdf/<bucket>/<file_id>__<name>.pdf. Each documents row points at its own with pdf_relpath, so a transcript and the paper it came from are one join apart.

Every staged PDF was checked byte-for-byte against the documents row that describes it — matched on documents.pdf_sha256 — so these are the exact bytes the transcripts were made from, not a re-download that might differ. source_url is also carried per row if you want the publisher's copy: 6,234 of 6,234 rows have a pdf_sha256 to check it against.

⚠️ The layout uses the file_id prefix rather than the bare file name because 229 of the documents share a file name with another document. A name-keyed layout silently overwrites them.

⚠️ pdf/ holds 6,236 files but documents has 6,234 rows. The difference is 2 PDFs with no transcript: publication_financial-statement/11356__BVm-Financial Report 2075-76.pdf, publication_financial-statement/12514__आ. वि. २०८०।८१.pdf. Shipped anyway, and named here rather than quietly dropped — it is a scan with no text layer that no paid vision read has covered, so there is nothing to transcribe from.

Figures — and why the number is 15,585 and not 181,920

figures has 181,920 rows, one per raster image placed on a page. Most of them are not figures. The frame says which is which instead of letting the row count imply an answer:

kind is derived, and the thresholds are conventions — page_frac >= 0.6 is a page scan, identical bytes in >= 20 documents is furniture, and below page_frac 0.005 is a fragment. Both inputs ship per row (page_frac, n_documents_sharing_image), so re-cut them if your use needs a different split. Neither has a clean gap behind it: page_frac is bimodal but its trough is populated, and the sharing count has no gap at all.

🛑 The fragment threshold is the one that changes the answer rather than trimming it. Without it 103,880 placements read as inline images — but that class had a median page fraction of 0.0, a median area of 153 pixels, and two documents supplying 76.8% of it. Requiring half a percent of the page leaves 15,585 candidates in 3,995 documents with the largest single document down to 1.6%.

What is deliberately not here

• —`figure_series_data` — no chart was read; CIAA's figure_data came from a paid vision read and this corpus has had none
• —`per_page_markdown_files` — 335,392 loose files; the identical text is in pages.parquet
• —`extracted_image_files` — 181,920 raster placements, mostly repeated seals and full-page scans; the source PDFs are shipped instead

A page anchor is a BLOCK START, not a page boundary

Inside a long ruled table the extractor emits the whole run under the anchor of the page where the table began. So a page's own section can be short while its text sits under an earlier anchor, and pages.chars / pages.n_table_blocks are per section. pages.printed_page_no carries the page number printed in the publisher's own footer where the page has one, which is the reliable way to cite a page.

Duplicates: the same report is sometimes published twice

work_id groups the rows that are the same underlying work, so deduplicate on work_id rather than on doc_id if you are counting reports. republication marks a document republished under a second file id; attribution_conflict marks rows that must not be collapsed, because two municipalities each claim the report and the conflict is the finding.

⚠️ This grouping was derived on the index named in Provenance below, not re-derived on this build's tree. Which documents are the same work is a property of the source PDFs and transfers cleanly; the attribution_conflict rows were separated on content and have not been re-checked here.

Loading it

from datasets import load_dataset

pages = load_dataset("damo-da/oag-nepal-audit-reports", "pages", split="train")
cells = load_dataset("damo-da/oag-nepal-audit-reports", "table_cells", split="train")

# every numeric cell of one province's reports
docs = load_dataset("damo-da/oag-nepal-audit-reports", "documents", split="train")
karnali = {d["doc_id"] for d in docs if d["province"] == "Karnali Province"}
amounts = [c for c in cells if c["doc_id"] in karnali and c["is_numeric"]]

Provenance

Transcription is by likhit, an open-source Nepali PDF text extractor, at the revision named above.

Personal data in these transcripts

The source reports name individuals. Identifiers judged unambiguously personal and of no research value were removed from the transcripts, and from every frame built from them, by 3 passes:

Each removed value is replaced in place by a [REDACTED:...] marker naming what it was, so a row is never silently short. 3,405 values were replaced in total; every one is journalled, and the journals record no matched digit.

🛑 This does not mean the dataset is free of personal data. The policy is deliberately narrow, because over-redacting an audit report destroys the accountability content it exists for. Specifically:

• —Names are not removed. Individuals are named throughout, including in beneficiary tables alongside beneficiary IDs.
• —Disclosed rather than removed, by policy: institutional email addresses (letterhead); office landline numbers; company PAN / permanent account numbers; entity bank account numbers; personal names. Each was adjudicated against a masked sample and judged either not personal data (an institution's published contact, a company identifier) or too entangled with audit content to remove safely.
• —The source PDFs in `pdf/` are not redacted at all. They are the unmodified published files, byte-verified against pdf_sha256. Identifiers removed from a transcript remain readable in the PDF it came from.
• —Some candidates were deliberately refused (citizenship_refused_ambiguous_row 1, citizenship_refused_digit_count 1, citizenship_refused_length 39, date_of_birth_refused_ambiguous_row 17). These are precision guards, not misses: the match window ran past the identifier, or a row held two candidate values and nothing established which one the label meant. Widening either would have removed audit figures.

If you need a stronger guarantee than this, filter or re-derive from pdf/ yourself rather than assuming the markdown is clear.

Licence and intended use

The underlying documents are public records published by the Office of the Auditor General of Nepal. This derived dataset is released under CC BY-NC 4.0 for research, journalism and civic accountability work.

It is a research artifact, not an official record. A figure that matters should be checked against the source PDF, which source_url and pdf_sha256 let you do. Do not present a transcript as the OAG's own text.