Better text extraction could unlock cleaner training data
A new benchmarking effort from Hugging Face and EleutherAI argues that one of the most practical ways to improve open language models is not a larger model architecture or a new training recipe, but better source text. Their FineBooks project evaluates whether modern open-weight optical character recognition systems can repair a longstanding weakness in public-domain book datasets: text that was extracted years ago from scans with error-prone OCR tools.
The problem matters because a large share of openly licensed historical writing reaches AI developers in noisy form. Libraries digitized huge collections long before today’s generative AI boom, and many of those archives were processed with older OCR pipelines that introduced missing characters, bad spacing, mistranscribed words, and structural errors. Once those mistakes enter training corpora, they reduce the quality of the material used to teach language models.
FineBooks is an attempt to measure how much the situation has improved. The project tested 14 open-weight OCR models on 2,165 pages from historical books and published the results as a leaderboard. According to the supplied source text, the best systems produced character accuracy above 97% while costing less than two U.S. dollars per thousand pages. That combination is notable because it suggests large-scale cleanup may now be operationally realistic for open-data projects that cannot afford expensive proprietary processing.
Why OCR quality matters for language models
The source text points to an earlier result from the Talkie project that quantified the downside of bad OCR. In that comparison, a language model trained on OCR-derived text learned at only 30% of the efficiency of a model trained on human transcriptions of the same books. That finding frames OCR errors not as a minor archival nuisance, but as a direct bottleneck on model training quality.
For open-model developers, the implication is straightforward. If public-domain books are a core source of licensed or license-compatible text, then improving their transcription quality can raise the value of a dataset without needing to negotiate new content deals. That is especially relevant for communities trying to build strong models from fully open inputs.
The FineBooks authors focus on historical books because the scale is large and the upside is immediate. The source text notes that Common Pile, described there as the largest openly licensed training corpus released last year, includes about 300,000 public-domain books whose text came from older OCR runs. Reprocessing even part of that collection with better tools could change the quality profile of an important open training resource.
Smaller models did unexpectedly well
One of the more striking findings in the source text is that smaller OCR models often beat larger ones. That runs against the common assumption that more parameters automatically translate to better performance. In practice, OCR quality depends heavily on layout handling, training data, language coverage, and how well a model deals with degraded scans, tables of contents, illustrations, and unusual typography. Size alone is not enough.
The top model named in the candidate metadata is dots.mocr, which reportedly reached 97.6% character accuracy. FineBooks also emphasizes cost efficiency, another important variable for organizations contemplating mass reprocessing. If strong OCR can be run cheaply, the economics of historical-text cleanup begin to look different. Instead of treating archival rescanning as a specialized, expensive curation task, developers may be able to incorporate it into ordinary dataset preparation pipelines.
The benchmark dataset itself appears designed to reflect real complications in book scanning. The source text describes examples that include a single-column English natural history page, a multilingual table of contents, and an illustration plate where the complete transcription is only a line of artist credit. Those examples matter because historical books are not uniform blocks of clean prose. OCR systems have to cope with multilingual pages, decorative formatting, sparse text, and visual artifacts that can confuse recognition.
A path to stronger open datasets, with limits
The largest near-term consequence of the FineBooks work may be for open-data maintenance rather than end-user products. Reprocessing millions of pages from archives such as the Biodiversity Heritage Library could improve the text quality feeding future open models. The source text says the library alone contains more than 300,000 digitized natural history documents totaling more than 64 million pages, which shows why automation is essential. Human transcription at that scale is unrealistic.
That said, the researchers do not present current OCR as a complete solution for every use case. The source text says the best outputs are already good enough for AI training purposes, but remain too error-prone for scholarly or scientific applications. That distinction is important. A language model can tolerate some noise spread across billions of tokens, especially if the objective is broad pattern learning. A historian, librarian, or scientist working with exact citations cannot.
So the practical conclusion is narrower than “OCR is solved.” FineBooks suggests OCR has improved enough to materially upgrade training corpora, not that it can replace careful human review where textual fidelity is critical. Developers and researchers should resist collapsing those standards into one. Training usability and archival accuracy are related goals, but they are not identical.
Why this matters beyond one benchmark
FineBooks fits into a larger shift in AI infrastructure: improving the inputs rather than only optimizing the models. In recent years, public discussion around AI progress has focused heavily on chips, parameters, and benchmark scores. But open-model builders also face a quieter constraint: the quality of the text they are legally and practically able to use. Better OCR expands the value of public-domain material already sitting in digital archives.
If the benchmark holds up and the tooling is adopted widely, the result could be a meaningful quality lift for future open models trained on historical literature, scientific texts, and archival collections. That would not eliminate the need for better curation, deduplication, and filtering, but it would remove one clear source of preventable degradation.
The broader lesson is that data infrastructure still offers large gains. A relatively inexpensive OCR improvement can ripple forward into dataset quality, training efficiency, and model behavior. For open AI ecosystems operating under budget and licensing constraints, that may be one of the more consequential upgrades available right now.
This article is based on reporting by The Decoder. Read the original article.
Originally published on the-decoder.com
