moebiusT7/book-ocr-studio
0
1<!doctype html>2<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>12B / 26B C1-OCR comparison and limitations</title><style>body{margin:0;background:#f5f4ef;color:#18342f;font:17px/1.65 system-ui,sans-serif}main{max-width:900px;margin:auto;padding:35px 24px}a{color:#215947}h1{line-height:1.2}pre{background:#e8ebe3;padding:18px;overflow-x:auto;white-space:pre-wrap;overflow-wrap:anywhere}code{font-size:.88em}table{border-collapse:collapse;display:block;overflow:auto}td,th{padding:9px;border:1px solid #becbbf;text-align:left}nav{margin-bottom:32px}</style></head><body><main><nav><a href="index.html">Book OCR Studio</a> · <a href="install.html">Install</a> · <a href="benchmarks.html">Comparison</a> · <a href="release-notes.html">Release notes</a></nav><h1>12B / 26B C1-OCR comparison and limitations</h1>3 4<h2>Latest test: current pipeline, 100 synthetic cases</h2>5 6<p>On 2026-09-23, the shipped <code>c1_ocr_v3</code> review pipeline was tested on the same7100 short synthetic images (50 Japanese, 50 English) with three seeds per8model, sequentially on one RTX 5070 Ti 16 GB. Both models used native Ollama,9temperature 0 and an 8,192-token context. Each image had one corrupted OCR10field and one initially correct field. Scores include image re-verification.</p>11 12<table>13<thead>14<tr>15 <th>Metric</th>16 <th style="text-align:right;">12B + C1</th>17 <th style="text-align:right;">26B + C1</th>18</tr>19</thead>20<tbody>21<tr>22 <td>Corrected fields, exact match / 300</td>23 <td style="text-align:right;">165 (55.0%)</td>24 <td style="text-align:right;">74 (24.7%)</td>25</tr>26<tr>27 <td>Errors left unchanged / 300</td>28 <td style="text-align:right;">129</td>29 <td style="text-align:right;">226</td>30</tr>31<tr>32 <td>Changed but still not exact / 300</td>33 <td style="text-align:right;">6</td>34 <td style="text-align:right;">0</td>35</tr>36<tr>37 <td>Originally correct fields changed / 300</td>38 <td style="text-align:right;">3</td>39 <td style="text-align:right;">0</td>40</tr>41<tr>42 <td>Mean review time per case, excluding loading</td>43 <td style="text-align:right;">3.021 s</td>44 <td style="text-align:right;">2.519 s</td>45</tr>46</tbody>47</table>48 49<p>The 300 observations per model repeat 100 cases across three seeds; they are50not 300 independent images. Exact matching includes whitespace and punctuation.51The three changes to initially correct fields in 12B were whitespace removal.52This larger test did <strong>not</strong> support a general quality advantage for 26B suggested53by the older six-image results below. It supports retaining <strong>12B + C1 as the54default</strong>, with 26B optional, rather than promising higher accuracy from 26B.</p>55 56<p>A separate exploratory follow-up added the same explicit JSON-schema instruction57to both models' prompts: exact corrections were 47/100 for 12B and 41/100 for 26B.58This prompt change is <strong>not shipped</strong>; those results are not pooled with the table.59The paired net-correction difference's 95% interval included zero in that follow-up.</p>60 61<p>These are short horizontal synthetic cases with shared templates, not 100 books62or a guarantee for real books, vertical text, ruby, complex layouts or languages63beyond those tested. They measure the complete review stage, not capture, OCR or64export time, and do not isolate raw model capability from review-gate behavior.65The newer experiment's raw logs and fixtures are not bundled; this public summary66is not an independently reproducible benchmark package.</p>67 68<h2>Historical six-image test: earlier C1 prompt</h2>69 70<p><strong>Exploratory measurements, not a claim about the current release's overall71speed or accuracy.</strong> Measured on 2026-09-23 using an earlier OCR-specific C172prompt. The current application uses <code>c1_ocr_v3</code>, with additional review73steps. This is not a new benchmark of that pipeline.</p>74 75<p>Six authored images each contained three erroneous and three correct OCR76fields. Each arm used seeds 101, 202 and 303 at temperature 0: 18 requests,7754 repeated erroneous fields and 54 repeated correct fields per arm.78These are repetitions, not 54 independent samples. The 26B+C1 arm was added79later using the same images and scoring; the earlier arms were not rerun.</p>80 81<table>82<thead>83<tr>84 <th>Metric</th>85 <th style="text-align:right;">12B + C1</th>86 <th style="text-align:right;">26B + C1</th>87</tr>88</thead>89<tbody>90<tr>91 <td>Sum of 18 review-request wall times</td>92 <td style="text-align:right;">52.291 s</td>93 <td style="text-align:right;">40.022 s</td>94</tr>95<tr>96 <td>Corrected fields, exact match / 54</td>97 <td style="text-align:right;">51</td>98 <td style="text-align:right;">45</td>99</tr>100<tr>101 <td>Errors left unchanged / 54</td>102 <td style="text-align:right;">3</td>103 <td style="text-align:right;">0</td>104</tr>105<tr>106 <td>Changed but still not exact / 54</td>107 <td style="text-align:right;">0</td>108 <td style="text-align:right;">9</td>109</tr>110<tr>111 <td>Originally correct fields changed / 54</td>112 <td style="text-align:right;">0</td>113 <td style="text-align:right;">0</td>114</tr>115<tr>116 <td>Corrected fields after colon/adjacent-space normalization / 54</td>117 <td style="text-align:right;">51</td>118 <td style="text-align:right;">54</td>119</tr>120</tbody>121</table>122 123<p>All nine exact-match failures in 26B+C1 were in three technical fields124across three repeats: the erroneous letters/numbers were corrected, but125full-width colons became half-width colons with different spacing. The last126row is a supplementary, post-hoc sensitivity analysis, not the primary127predefined score. Normalization does not establish perfect source fidelity.12812B left one authored name error in all three repeats.</p>129 130<p>In this small test, 26B's summed request time was approximately <strong>23.5% lower</strong>.131This does not include the complete capture/OCR/export workflow or establish132a cold-start/loading comparison. No confidence interval, significance test,133held-out generalization or training-contamination clearance is claimed.13426B may make a whole job slower if limited VRAM forces OCR and review to135alternate. Both observed model artifacts were Q4_0; exact tags, digests,136sizes and individual request measurements are in137<a href="benchmarks/historical-c1.json">the measurement data</a>.</p>138 139<p>The release therefore defaults to <strong>12B + C1</strong>. <strong>26B + C1 is optional</strong> for140users able to provide more VRAM. These observations suggest a possible141benefit on some errors, accompanied by a punctuation-fidelity tradeoff;142they do not guarantee a speed or quality improvement on a user's books.</p>143 144<p>The JSON contains only numeric measurements and synthetic case identifiers.145Source-image fixtures and the old benchmark harness are not included, so146this package supports checking the reported arithmetic, not rerunning the147entire historical experiment. Source-file SHA-256 values identify the local148records used; hashes alone are not independent verification of the experiment.</p>149<hr><p><a href="BENCHMARKS.md" download>Download original text</a></p></main></body></html>150