I have been trying, on and off for the last five years, to digitise the 8-volume Kannada Nighantu, a dictionary that itself took more than 50 years to finish.
I started this when I was reading a Halegannada text. I kept coming across unfamiliar words and wanted an easy way to look them up. With English, I can search for a word and quickly compare definitions across multiple dictionaries. For Kannada, especially when dealing with older texts and obscure vocabulary, that experience simply did not exist. Finding and exploring such words was often difficult and frustrating.
We also have physical copies of this dictionary at home. Once I started going through the volumes seriously, it felt sad that so much work was still mostly locked inside printed books. For people unfamiliar with this dictionary, Publishing Perspectives has a short introduction, and Scroll has a longer article about the 9,000-page, 8-volume dictionary that took more than 50 years.
Five long years of OCR rabbit holes
I started the OCR work with scans from archive.org. It turned out to be much harder than I expected. The pages are dense. The print has an old letterpress look and all the artefacts that come with it. It uses letter forms and older Kannada characters that are no longer used today. In fact, the characters change across the volumes! On top of that, every page is full of abbreviations, symbols, subscripts, reference marks, etymology notes, and dictionary-specific conventions. OCR systems do not usually see this kind of page, so they do not work very well on it.
I am a software engineer, not an ML engineer or researcher, so much of this was trial and error. I went down many rabbit holes: image preprocessing, page segmentation, line extraction, trying different OCR engines, creating training data by hand, generating synthetic data, and fine-tuning models. Each improvement would solve one problem and expose two more. Even after fine-tuning gave me a much better OCR model than before, there were still too many mistakes for review to be efficient.
In the last six months, the work has moved much faster because of newer VLMs and LLM coding agents. The newer models gave me a much better starting point than I had earlier, especially Chitrapathak-2. The coding agents also helped a lot with experiments, tooling, cleanup, and keeping the training and evaluation loop moving.
The final result did not come from one model. What worked was a mix: fine-tuned OCR models, combined OCR data, and post-processing based on the structure of the dictionary itself. Things like page headers, word order, repeated patterns, and common dictionary conventions turned out to be very useful for catching and fixing errors.
There are a lot of technical details behind all this. I can share them separately if anyone is interested.
Where it stands now
The output is finally good enough that human review feels practical.
I did a small held-out check on four manually reviewed pages from Volume 1. In that sample, the current pipeline gives about a 0.45% character error rate. In simpler terms, around 87% of the lines are already exactly correct after basic normalization. On a typical 96-line page, this means roughly 84 lines need no correction, and around 12 lines need one or two character corrections.
This is a small sample, so I am not presenting it as a full benchmark.
| System | Character error rate (CER) | Exact lines |
|---|---|---|
| Current dictionary-tuned pipeline (fine-tuned) | 0.45% | 87.43% |
| Earlier hybrid pipeline (fine-tuned) | 0.49% | 86.65% |
| Best fine-tuned Chitrapathak-2 model | 0.88% | 79.06% |
| Stock Chitrapathak-2 | 7.74% | 15.97% |
| Tesseract with kan_best | 8.41% | 10.99% |
| Surya | 9.66% | 19.11% |
| Google Document AI | 18.50% | 3.40% |
(I tested many models, this is a small subset)
I have now run this on Volume 1. The text is not final. It still needs review. But it is no longer a situation where someone has to retype everything from scratch.
I also ended up building many tools around this work: scripts for preprocessing, evaluation, model comparison, line review, dictionary-aware checks, rendering, and so on. One of those tools is a review workbench, because checking OCR output in plain text files was painful.
The Typeset view helps compare the corrected text against the original page structure.
Help Needed
As far as I understand, this dictionary is not in the public domain. The rights may be with Kannada Sahitya Parishat, but I am not fully sure. I do not want to publish or distribute the digitised text without proper clarity or permission.
Does anyone here know how to approach this properly?
I am looking for help with questions like these:
-
Who is the right person or institution to contact?
-
Has anyone worked with KaSaPa or similar institutions for digitisation permissions?
-
How should this be framed so that it is taken seriously as a preservation and access effort?
-
Are there examples from other Indian-language projects where permissions were handled well?
If the permissions side can be sorted out, the next step is review and correction. If we can make correction manageable so that reviewers can check lines, fix mistakes, and approve good text instead of typing everything from scratch, then it is doable. I already have a process and tooling for this, which I built for myself, and it could become useful for broader community review.
I am also interested in collaboration beyond this one dictionary. Eventually, I want to open-source the tools, evaluation setup, and datasets I have built. My experience has been that, for old printed material, especially dictionaries and reference works, base OCR is not enough. Fine-tuning, layout handling, and domain-specific correction make a huge difference, and we should push OCR to its limits to reduce the manual effort, which is going to be the real bottleneck anyway. If anyone is interested in discussing this or sharing notes, please reach out.