0.9m06 Pt 2 - FGbz 2; Table OCRs; Text Pages 2

[maple7-7-7]

This discussion is a fresh extension of v0.9m06 Part 1.
It begins with testing Alex's patched test binaries for DjvuLibre Win32 and Win64.
The patch is to improve upon the integration and description of FGbz chunks in minidjvu-mod.
This post to be updated.
Update: Alex's binaries and my test results:
https://github.com/trufanov-nok/minidjvu-mod/files/7363837/patched_djvumake3.5.28_x64_vs2013.zip
https://github.com/trufanov-nok/minidjvu-mod/files/7363838/patched_djvumake3.5.28_x32_vs2013.zip
For any problems with older machines, Alex suggests this link to Visual C++ Redistributable Packages for Visual Studio 2013:
https://www.microsoft.com/en-us/download/details.aspx?id=40784
Results -- Excellent!
The newly patched DjVuLibre's successfully fix the FGbz chunk problems described in Part 1 of this discussion. I have tested the x64 DjVuLibrePatched in Win 10, and have tested the x32 DjVuLibrePatched in Win 7, and the FGbz patched function for both worked perfectly: 25 bytes and 2 colors instead of 139 bytes and 27 colors for the FGbz chunk in a table test page colorized with 26 color zones of #blue. Thanks, Alex, once again.

[maple7-7-7]

Adding FGbz chunks using Patched DjVuLibre Win64
Comparison: Left Side: Any2DjVu Colorized Table -- Right Side: v0.9m06 Colorized Table (jpg)

any2djvu Sjbz = 2999 bytes; v0.9mo6 Sjbz = 1159 bytes
any2djvu FGbz = 24 bytes; v0.9m06 FGbz = 25 bytes
Both Sjbzs are from respective sets of 20 table pages with a 20-page dictionary and 600dpi.
The v0.9mo6 program uses much better table component analysis and compression.
These enhanced functions were introduced by Alex with this 0.9m06 version of minidjvu-mod.
The script for making the minidjvu FGbz chunk:

SCRIPT_TABLE_TEMPLATE_2.txt

Coordinates were chosen using the advanced feature in DjView 4 which allows one to select a given area on a page and save it as maparea coordinates.
Note that after doing the horizontal lines, it was not necessary to determine mapareas for the vertical lines, as they became colorized in the process of adding the others.
The FGbz script was tested on another table with the same layout and dpi, and worked fine. So this makes such a script function as a kind of table template.
Thanks again, Alex.

[maple7-7-7]

Here are two 20-page 600dpi DjVus with the same-looking content, as two sets of Colorized Tables.
One set is from the any2djvu server, which processed it as a PDF in color, and the other set is from minidjvu-mod0.9m06 using the DjVuLibrePatchedx64 (Win) program. Minidjvu-mod processed it as PBMs and then the resulting b&w DjVu was colorized using djvumake in the patched DjVuLibre with command-line FGbz chunks. A b&w PDF version of the document processed by any2djvu gave the same Sjbzs and dictionary in the resulting DjVu as the color PDF.

File Sizes: any2djvu 85.3 KB -- minidjvu-mod 55.0 KB
Also included is the script to integrate FGbz chunks into the minidjvu-mod DjVus.

COLORIZED_TABLES_any2djvu_minidjvu-mod0.9m06djvu_and_Script.zip

In almost all instances, after horizontal-line mapareas were used to colorize the tables, it was not necessary to also include the mapareas of the vertical lines.

My experience at any2djvu is that if the table lines are colorized and also some of the characters, this does not affect the segmentation of the page. The Sjbz stays the same, as if it were a black and white file. If on the other hand a solid block of color is in a table cell, then there may be some segmentation of that color, where some of the color is placed in the foreground as Sjbz, and some is placed in the background as BG44. Or it may be totally kept in the Sjbz. This is one example of where the Threshold option in the program DjVuDigital can be useful when working with PDFs in Linux, although such a program lacks shared dictionaries.

[maple7-7-7]

OCR - Any2DjVu vs Minidjvu-mod0.9m06 DjVu Table Pages - Use the pdf2djvu OCR instead
Because the Sjbzs for the two programs are so different when it comes to sorting data, it seemed a good question to ask whether two identical-looking DjVus that have quite different shared dictionaries might be treated differently by the OCR system at any2djvu.
We may commonly use the any2djvu server to generate DjVus, but it also offers the option to submit a DjVu for Optical Character Recognition, to generate a hidden text layer.
I tested the two 20-page DjVus, and there was virtually no difference in the hidden text display. The OCR was virtually useless in that there were crazy symbols for many characters. The largest digits were not OCR'd at all, and it was basically not worth the trouble. I then made the DjVu in the program pdf2djvu, which gave a huge file size, but a much more accurate and far less outlandish hidden text display. I recommend that one use the OCR from a parallel DjVu in pdf2djvu to add a hidden text layer to a table-rich DjVu produced with any2djvu or minidjvu-mod. Even then, there will be quite a few corrections to make. Unlike the FGbz chunks, the OCR layer can be placed on any page, as long as the page dimensions are the same as the original OCR'd page. You can transfer a pdf2djvu-based OCR to other programs' DjVus.

[trufanov-nok]

he OCR was virtually useless in that there were crazy symbols for many characters. The largest digits were not OCR'd at all, and it was basically not worth the trouble.

I don't know which OCR engine is used by any2djvu, but suppose it's very old. The best OCR engine would be ABBYY FineReader, but it's commertial. I'm using opensource Tesseract OCR engine.

one use the OCR from a parallel DjVu in pdf2djvu to add a hidden text layer to a table-rich DjVu produced with any2djvu or minidjvu-mod.

Do you process a "color.pdf" made from "color.tmd"?

[maple7-7-7]

Hi Alex,
Thanks for the information about other OCR possibilities.
Yes. Most of my color.pdf's are from color.tmd's.
One reason the pdf2djvu DjVu was so big was that the program commonly creates huge BG44 chunks when dealing with color pdfs. Any2djvu handles this much better, of course. I have often let the OCR from pdf2djvu "rescue" a poorly made OCR from the other programs, when dealing with tables.
It is interesting that one can really customize the OCR layer if one is willing to put in the time. Then the table OCR's can be perfectly accurate. And so for instance when distinguishing a 3-line "equals sign" from a standard 2-line equals sign, you could add 3 hyphens in a row --- to symbolize that sign. Then you could search for the sign that way. The sign itself will look fine of course when highlighted, because DjVu is basically written in stone.
I have played around with the OCR using djvused and it is a place to get very creative. For instance, if you have a document where instead of searching for a phrase you are looking instead for all of the verbs, you could tag a verb OCR entry with something like "/v", actually replace the verb itself in the OCR, and then enter /v in the search box in DjView and find all the verbs. I am presuming that such an entry is not considered a special character in the system. There are sometimes adjustments needed in the OCR based on how one tags, but it is a very interesting area to be creative in.
Another example is if you want to find all the names of birds in a Nature book, you could manually place the expression "brd" in a set of coordinates holding the name of the bird, and that bird's name would be highlighted. Of course the hidden text layer if displayed would show "brd". You could also replace in the OCR the word "warm" with the word "cool" and of course when you search for the word "cool" you would have the word "warm" highlighted. Or for more dramatic effect, you could be reading Moby Dick and the hidden layer if displayed shows War and Peace.
It has been some time since I played with the OCR, but I recall that one could even put two references in the same set of coordinates. I would have to check how that was done. What I mean is two OCR words as separate entries for one DjVu word.
Another thing was to find only the instances of a given term specific to text, or to tables, or to title pages, or to picture captions. OCR can also serve as an annotations system. For instance, have an OCR entry "Title Page" and when you have that expression added to each Title Page as an OCR entry, then you can search for Title Pages, even though you would not see that expression itself on the page.
Or let's say you are dealing with weather in the U.S. with the Fahrenheit system F. You could say to find only those temperatures in the range of 35-40 F. What you would do is create an OCR entry for each instance of a temperature in that range in the document. The entry could say "35-40" and then the search function finds all of those temperatures. Very simple yet lots of possibilities.

[maple7-7-7]

Basic Text Pages - Any2djvu vs MiniDjVu-mod0.9m06
For both programs, the DjVus look identical and the file sizes tend to be within about 1% of each other when using standard 10pg and 20pg dictionaries. Any2djvu is limited to 10pg dictionaries from a submitted zip file, and is limited to 20pg dictionaries from a pdf.
Any2djvu uses pre-LizardTech encoders said to function almost as well as the commercial encoders. In my experience, the program DjVuSolo3.1, which offers the commercial encoders but for limited use, encodes text with the same look and efficiency as any2djvu. So we can think of any2djvu encoding of text as commercial-level quality.
MiniDjVu-mod offers the advantageous option of using bigger dictionaries than 20pgs. I find that one can reduce file size by another 10-15% depending on how large the dictionary is beyond 20 pages.
MiniDjVu-mod is a program that acts as a bridge between the cjb2 program in DjVuLibre, which lacks the shared dictionary option, and the commercial version, which does. This has always been my chief reason for wanting to use it.
What are the file size reductions like when using the shared dictionary option in minidjvu-mod?
Using a basic set of 200 text pages, with no enhancements like borders or corners or embellished text characters, here is a sample set of results. This set uses the options -m for match, and the option "Use Prototypes." These two options avoid any degradation to the pages and yet allow for good compression.
MiniDjVu-Mod v0.9m06
200 basic text pages - 5100x6600 - 600 dpi - from basic PBMs
Using the minidjvu-mod GUI:
No shared dictionaries: 879 KB (Enter a "1" for the GUI dictionary pages option)
10 pg dictionaries: 533 KB - an almost 40% reduction in file size
20 pg dictionaries: 490 KB - a 44% reduction
40 pg dictionaries: 462 KB - a 47% reduction
50 pg dictionaries: 458 KB - a 48% reduction
100 pg dictionaries: 441 KB - an almost 50% reduction
200 pg dictionary: 432 KB - a 51% reduction
Larger dictionary DjVus tend to open more slowly and to be slower in initial larger page jumps.
The 40 or 50 pg dictionary option is probably the sweet spot for good compression and responsiveness on a newer computer. I tested things using a newer computer with 16 GB RAM.

[maple7-7-7]

This is an apology to Alex. I had found earlier that the Win version of minidjvu-mod would not process much beyond 460 text-based pages at a time. I went back and checked the 903 pbms I was attempting to process, and in the 460s there are several pgms instead of pbms, although they are named with the extension .pbm. I had initially batch converted the 903 pages and should have checked the look of each converted page afterwards.
Using Alex's newest update, which enhances the ability of his program to work with larger numbers of files in Windows, and having converted the pgms back to pbms, I was able to encode the 903 pages with his minidjvu-mod. I have had similar success using 903 bmps. Thanks.