ERCDiDip/charter-diplomatic-segmentation

1{2 "cells": [3  {4   "cell_type": "markdown",5   "metadata": {},6   "source": [7    "# Quickstart — `segmenter_tiny_v4` diplomatics segmenter\n",8    "\n",9    "This notebook shows you **exactly** how to load the model and segment a medieval charter into its diplomatic parts (`INVOCATIO`, `INTITULATIO`, `PUBLICATIO`, `NARRATIO`, `DISPOSITIO`, `DATATIO`, ..., `CORROBORATIO`).\n",10    "\n",11    "**You don't need to know anything about Hugging Face.** If you just downloaded this repository, every code cell below runs as-is (the first one installs the two required libraries)."12   ]13  },14  {15   "cell_type": "markdown",16   "metadata": {},17   "source": [18    "## 1. Install the two required libraries\n",19    "\n",20    "Run this cell once. It installs PyTorch and Transformers if they are missing."21   ]22  },23  {24   "cell_type": "code",25   "execution_count": null,26   "metadata": {},27   "outputs": [],28   "source": [29    "import subprocess, sys, importlib.util\n",30    "\n",31    "def _ensure(pkg, module=None):\n",32    "    module = module or pkg\n",33    "    if importlib.util.find_spec(module) is None:\n",34    "        subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", pkg])\n",35    "\n",36    "_ensure(\"torch\")\n",37    "_ensure(\"transformers\")\n",38    "print(\"dependencies ready\")"39   ]40  },41  {42   "cell_type": "markdown",43   "metadata": {},44   "source": [45    "## 2. Load the model\n",46    "\n",47    "The cell below loads the model **from this repository's local folder** (`segmenter_tiny_v4/`). If you prefer to use it from the Hugging Face hub instead, set `USE_HUB = True` — then it downloads the same model from `ERCDiDip/charter-diplomatic-segmentation`."48   ]49  },50  {51   "cell_type": "code",52   "execution_count": null,53   "metadata": {},54   "outputs": [],55   "source": [56    "import os\n",57    "from transformers import AutoTokenizer, AutoModelForTokenClassification\n",58    "\n",59    "USE_HUB = False                 # False -> use the local ./segmenter_tiny_v4/ folder\n",60    "MODEL = \"ERCDiDip/charter-diplomatic-segmentation\"   # (used only if USE_HUB = True)\n",61    "\n",62    "if USE_HUB:\n",63    "    tok = AutoTokenizer.from_pretrained(MODEL)\n",64    "    model = AutoModelForTokenClassification.from_pretrained(MODEL)\n",65    "else:\n",66    "    local = os.path.join(\"segmenter_tiny_v4\")\n",67    "    assert os.path.isdir(local), f\"Nem találom a modell mappát: {local}\"\n",68    "    tok = AutoTokenizer.from_pretrained(local)\n",69    "    model = AutoModelForTokenClassification.from_pretrained(local)\n",70    "\n",71    "model.eval()\n",72    "id2label = {int(k): v for k, v in model.config.id2label.items()}\n",73    "print(f\"Loaded. {len(id2label)} labels, label example: {list(id2label.values())[:3]}\")"74   ]75  },76  {77   "cell_type": "markdown",78   "metadata": {},79   "source": [80    "## 3. A ready-to-use segmentation function\n",81    "\n",82    "This is the whole point of the notebook: paste any charter text and get back, for every word, which diplomatic part it belongs to. It uses **windowed inference** (512 tokens), so long charters also work."83   ]84  },85  {86   "cell_type": "code",87   "execution_count": null,88   "metadata": {},89   "outputs": [],90   "source": [91    "import torch\n",92    "\n",93    "MAXLEN = 512        # model context window (in subword tokens)\n",94    "WINDOW   = 450      # safety margin so words never fall off the edge\n",95    "\n",96    "def predict(text: str):\n",97    "    \"\"\"Segment any charter text.\n",98    "    Returns a list of dicts: {word, label (full BIO), section, conf}.\n",99    "    \"\"\"\n",100    "    words = text.split()\n",101    "    if not words:\n",102    "        return []\n",103    "\n",104    "    # score accumulation per word position\n",105    "    scores = [torch.zeros(len(id2label)) for _ in words]\n",106    "    seen   = [0.0] * len(words)\n",107    "\n",108    "    with torch.no_grad():\n",109    "        i = 0\n",110    "        while i < len(words):\n",111    "            chunk = words[i:i + WINDOW]\n",112    "            enc = tok(chunk, is_split_into_words=True, truncation=True,\n",113    "                      max_length=MAXLEN, return_tensors=\"pt\")\n",114    "            logits = model(**enc).logits[0]             # [T, 22]\n",115    "            probs  = torch.softmax(logits, dim=-1)\n",116    "            wids   = enc.word_ids()\n",117    "\n",118    "            # first-subword softmax votes per word, added to global scores\n",119    "            last = None\n",120    "            for t in range(len(wids)):\n",121    "                w = wids[t]\n",122    "                if w is None or w == last:\n",123    "                    continue      # skip special tokens and continuation subwords\n",124    "                last = w\n",125    "                g = i + w\n",126    "                if g < len(words):\n",127    "                    scores[g] = scores[g] + probs[t].cpu()\n",128    "                    seen[g]   = 1.0\n",129    "\n",130    "            i += WINDOW\n",131    "\n",132    "    out = []\n",133    "    for w, s in enumerate(words):\n",134    "        if seen[w] == 0:\n",135    "            continue\n",136    "        lab_id   = int(scores[w].argmax())\n",137    "        label    = id2label[lab_id]\n",138    "        section  = label[2:] if label[1] == \"-\" and len(label) > 2 else label\n",139    "        conf     = float(scores[w].max() / scores[w].sum())\n",140    "        out.append({\"word\": words[w], \"label\": label,\n",141    "                    \"section\": section, \"conf\": conf})\n",142    "    return out\n",143    "\n",144    "def to_spans(pred):\n",145    "    \"\"\"Collapse consecutive words with the same section into readable spans.\"\"\"\n",146    "    spans = []\n",147    "    for r in pred:\n",148    "        if spans and spans[-1][\"section\"] == r[\"section\"]:\n",149    "            spans[-1][\"words\"].append(r[\"word\"])\n",150    "            spans[-1][\"conf\"] = min(spans[-1][\"conf\"], r[\"conf\"])\n",151    "        else:\n",152    "            spans.append({\"section\": r[\"section\"], \"words\": [r[\"word\"]], \"conf\": r[\"conf\"]})\n",153    "    for s in spans:\n",154    "        s[\"text\"] = \" \".join(s[\"words\"])\n",155    "    return spans"156   ]157  },158  {159   "cell_type": "markdown",160   "metadata": {},161   "source": [162    "## 4. Try it on a sample charter\n",163    "\n",164    "Run the next cell and look at the printed structure. The latin text below is a real medieval charter opening — `INVOCATIO` (invocation of God), then `INTITULATIO` (who is speaking), then `PUBLICATIO` (to everyone), then the substance."165   ]166  },167  {168   "cell_type": "code",169   "execution_count": null,170   "metadata": {},171   "outputs": [],172   "source": [173    "sample = (\"In nomine sancte et individue trinitatis. Ego Lupoldus dei gratia dux \"\n",174    "          \"Austrie universis Christi fidelibus presentibus et futuris salutem. \"\n",175    "          \"Notum sit omnibus tam presentibus quam futuris qualiter ego dedi \"\n",176    "          \"terram iuxta fluvium Danubii in proprietatem ecclesie sancte Marie. \"\n",177    "          \"Factum est autem anno dominice incarnationis MCXLVIII. \"\n",178    "          \"Actum est Vienne. Datum per manum cancellarii nostri.\")\n",179    "\n",180    "pred = predict(sample)\n",181    "spans = to_spans(pred)\n",182    "\n",183    "for s in spans:\n",184    "    print(f\"\\n### {s['section']}  (conf {s['conf']:.3f})\")\n",185    "    print(s[\"text\"])"186   ]187  },188  {189   "cell_type": "markdown",190   "metadata": {},191   "source": [192    "## 5. Put your own charter in a variable\n",193    "\n",194    "Replace the text below with any Latin or Middle High German charter and re-run. The model does not care about punctuation or line breaks — it works on `text.split()` (rough word level)."195   ]196  },197  {198   "cell_type": "code",199   "execution_count": null,200   "metadata": {},201   "outputs": [],202   "source": [203    "my_charter = \"\"\"\n",204    "In nomine domini nostri Ihesu Christi. Nos Otto dei gratia rex Romanorum\n",205    "omnibus fidelibus in perpetuum. Inter cetera religiosorum loca praecipua\n",206    "devotione diligentes, monasterium sancti Petri in Monasterio speciali\n",207    "favore prosequi disposuimus. Quapropter notum facimus universis presentis\n",208    "pagine inspectoribus quod praefatam ecclesiam sub tuitionem nostram\n",209    "suscepimus. Acta sunt hec anno dominice incarnationis millesimo\n",210    "ducentesimo primo, indictione quarta, data per manum cancellarii nostri.\n",211    "\"\"\"\n",212    "\n",213    "for s in to_spans(predict(my_charter)):\n",214    "    print(f\"\\n### {s['section']}  (conf {s['conf']:.3f})\")\n",215    "    print(s[\"text\"])"216   ]217  },218  {219   "cell_type": "markdown",220   "metadata": {},221   "source": [222    "## 6. Word-by-word view (optional)\n",223    "\n",224    "If you prefer the raw per-word BIO labels instead of collapsed sections."225   ]226  },227  {228   "cell_type": "code",229   "execution_count": null,230   "metadata": {},231   "outputs": [],232   "source": [233    "for r in predict(sample)[:40]:\n",234    "    print(f\"{r['word']:>18}  {r['label']:>16}  section={r['section']:>14}  conf={r['conf']:.3f}\")"235   ]236  },237  {238   "cell_type": "markdown",239   "metadata": {},240   "source": [241    "## Notes, limitations and citation\n",242    "\n",243    "- **What it is:** a `BertForTokenClassification` head over Multilingual-MiniLM, distilled from a larger XLM-R teacher. It recognises **11 diplomatic sections** via a 22-label BIO scheme (`B-`/`I-` prefix per section).\n",244    "- **Language scope:** trained on medieval Latin and German. Behaviour on other languages is untested and likely poor.\n",245    "- **`NARRATIO` caveat:** on Middle High German charters the model tends to merge `NARRATIO` into `DISPOSITIO` (a documented intrinsic limit of the teacher's signal).\n",246    "- **Length:** context window 512 tokens; this notebook slides a window over longer texts automatically.\n",247    "- **Labels:** read them from `model.config.id2label` (alphabetical order) — never hard-code the index order.\n",248    "- **License:** `cc-by-nc-4.0` (non-commercial). If you use the model, please cite:\n",249    "\n",250    "```bibtex\n",251    "@misc{charter-diplomatic-segmentation,\n",252    "  title={Distilled Weakly-Supervised Multi-Head Segmentation of Medieval Charters},\n",253    "  author={Kovács, Tamás; Nicolaou, Anguelos; Atzenhofer-Baumgartner, Florian; Renet, Nicolas; Consolo, Giuseppe, Tscherne Niklas; Decker, Franziska; and Vogeler, Georg},\n",254    "  year         = { 2026 },\n",255    "  url          = { https://huggingface.co/ERCDiDip/charter-diplomatic-segmentation },\n",256    "  doi          = { 10.57967/hf/10291 },\n",257    "  publisher    = { Hugging Face }\n",258    "}\n",259    "```"260   ]261  }262 ],263 "metadata": {264  "kernelspec": {265   "display_name": "Python 3",266   "language": "python",267   "name": "python3"268  },269  "language_info": {270   "name": "python",271   "version": "3.10"272  }273 },274 "nbformat": 4,275 "nbformat_minor": 4276}277