0.9m05 - Border Graphics; Tables 1

[maple7-7-7]

Hi, Alex,
Here are the results of testing v09.5 vs v09.3. For all pages of a 30-page set of 4250x5500 pbms converted to 500 dpi DjVus using 09.5, the right border was unexpectedly offset a bit to the left. The 09.3 version was fine. This is shown in the two jpgs, which are converted from the respective first page DjVus. A 30-page dictionary was used for both versions.
Also included is a zip of the 30 pbms and a script of the 30 pbm filenames.
Thanks.
test_30_pbms_script_for_500dpi.txt
test_30_PBMs_(01-30)_4250x5500_emerson_for_500dpi_09.3_and_09.5.zip
test_30_pbms_mdm09 3_gui_500dpi_emerson_30pgdict_pg1
test_30_pbms_mdm09 5_gui_500dpi_emerson_30pgdict_pg1

[trufanov-nok]

Very good, this problem was in 0.9m3 too. Once I faced with a page which right border of a rectangle that touched the edge of the image disappeared after encoding. That time I just increased image width by a few pixels to walkaround this problem as I had no time to investigate this. And later I couldn't recall which image had a problem. Now I can finally trace it.

[maple7-7-7]

Good. Here is a jpeg of a DjVu page encoded with v09.5, this time using a pbm that is 2500x3300 for a 300dpi DjVu setup. I include the zipped pbm here. All of the encoded 300dpi DjVu pages have the same type of right-hand border. Hopefully this will help you trace further the nature of the problem.

test_pbm_for_09 5_emerson_2550x3300_for_300dpi_djvu_then_converted_to_jpg

test_pbm_zipped_for_09.5_emerson_2550x3300_for_300dpi_djvu.zip

It appears that this site does not accept DjVu or pbm files directly.

[maple7-7-7]

(I submitted a ticket to GitHub asking if they could add the ability to accept DjVu and pbm formats in attachments.)

[trufanov-nok]

Ok, I've traced it down and fixed in sources. It was about image smoothing operation.
Wait a bit - I'll regenerate binaries. I think I won't change version number - will just replace binaries.

[maple7-7-7]

Thanks. I am so happy for everyone's sake that you were able to isolate the problem. When encoding, I used all default settings except the 300dpI (and 30-pg dictionary). I think I will rename the new binaries 09.5b and put them in a separate subfolder, as I like to watch the evolution of this program.
Thanks again for the update, Alex.

[trufanov-nok]

Ok, everything (encoder and encoder bundled with gui) should be updated now. Please redownload the binaries.

[maple7-7-7]

Thanks. About to do so.

[maple7-7-7]

Hi, Alex,
Here are the results of comparing 4 programs as DjVus at 500 and 300 dpi. (8)
v09.5b GUI 0.2.3b v0.9.3 DjVuSolo3.1
Same set of 10 4250x5500 pbms, tested at 500dpi and 300dpi
10pg dictionaries
Some Results:

1. Smaller file sizes with 500dpi
2. GUI 0.2.3b outperforms all the others, even 0.95b. (?)

Conclusion
Your newest version works best as a GUI, and outperforms DjVuSolo3.1.

The attachment includes side-by-side DjVu data comparison as a png.
Most of the differences between 0.9.5b and GUI are in the pages and not the dictionary,
Thanks.
Stephen
minidjvu-mod-0.9.5b_vs_gui-0.2.3b_vs_0.9.3_vs_DjVuSolo3.1.zip

[maple7-7-7]

Hi Alex,
This is minidjvu_gui_0.2.3(b) vs DjVuSolo vs any2djvu (a2d) as Tables-only DjVus.
I use silly random-like characters for many letters and numbers.
Best comparisons are with 200/300/600, as 400 is "variable dpi" at a2d. No 500's.
Note: 600 means 5100x6600; 300 means 2550x3300, etc.
I took 20 basic PBMs with Tables only and tested 01-10 and 11-20 pgs.
Workflow:
Textmaker 20 Table pgs TMD (color) > PDF (color) > a2d DjVu 200/300/400/600 (color)
a2d DjVu 600 etc (color) --> WinDjVuExtended export as 20 BMPs 600 etc (color)
Overall 80 BMPs (color) --> 80 IrfanView PBMs (binary) (200/300/400/600)
Your GUI - 01-10 PBMs to DjVu; 11-20 PBMs to DjVu (200/300/400/600)
All 10 pg dictionaries. At a2d I used normal fg compression. No OCR.
Some Results: For 200/300/600, file sizes (6 for each) - Pages look good.
minidjvu-mod-gui-0.2.3(b) File Size total = 342 KB
DjVuSolo3.1 File Size total = 200 KB
any2djvuserver File Size total = 199 KB
Most characters are the font FreeSerif. The + sign and 3-line equals sign are Liberation Sans. The = sign is Century Schoolbook L.
I kept the color in the initial files because it was associated with the line art and was tedious to remove. So that was saved for IrfanView converting to PBMs.
Thanks,
Stephen
TABLE_TESTS_DJVUS_minidjvu-mod_0.2.3b_DjVuSolo_any2djvu_200-300-400-600.zip
TABLE_TEST_PBMS_200-300-400-600.zip

[trufanov-nok]

Thank you, I'll check it!

[trufanov-nok]

Well, I've investigated the djvu dictionaries with my djvudict and found out a couple of problems.
It seems that minidjvu's image splitting (to letters) algorithm isn't sophisticated enough to detect all symbols. After splitting it's using "walking along contours" approach that can't detect symbols inside symbols. Like,

is a one big component for it.
Also, minidjvu marks too big components with a "not a letter" flag to exclude the from lossy compression as they are part of the graphic. And it marks the components that are touching them with this flag too even they are small bcs they might be sliced by splitting algo while being a parts of a whole graphic image. They problem is that this algo isn't sophisticated enough too and detecting boundary boxes detection instead of real counters detection. That means this:

Will be split into 4 components, but all of them will be marked with a "not a letter" flag, bcs one of them is too big and it's boundary box includes the other. And this is interpreted as "touching".

So, it seems the old algorithm shall be replaced with something like that.
I think I'll just adapt the Connected Components detection algo from DjVuLibre's cjb2 encoder. I suspect that this is exactly the same algorithm that commercial encoder uses as they all limit size of the component to 500 pixels max.

Good news is that it seems none of encoders are trying to compress the "too big" components with some lossless method and keep putting pieces of the grid to the Sjbz as unique bitmaps even they are exactly the same. This is a possibility to achieve even better page compression, at least for your test case.

[maple7-7-7]

Hi Alex,
Thank you for all the details in your newest post.
I understand at least 30% of it!
I think I am going to need a Ph.D. to understand the Seed Filling reference page!
I am quite happy to keep using any2djvu for generating DjVus with Tables in them. I did not know about your djvudict project.
Your explanation on the project page is very helpful. I had this idea that each page had its own dictionary but was not quite sure. It would need that for compression, whether shared across pages or not. I guess you could say each page has a "shared paragraphs" dictionary.
I see that ScanTailor is largely about dealing with a variety of compression methods for a given complex page. For me, I also deal with variety, but usually in terms of many pages of one content type at a time. I usually work with 4 basic types of pages. I am usually dealing with a stretch of Text pages, a stretch of Table pages, a stretch of JPG pages, and scattered Title/Subtitle pages. Obviously I try to keep the basic type of content within a given page of one type in order to maximize compression efficiency. So when I create a DjVu ebook, I try to work with clusters of the same page content. This is one reason that your new ability to select discontinuous pages for a given dictionary is so valuable to me.
There are times I will have a Table Title page on the left side, and the corresponding Table on the right. But I will encode the two types as parts of separate sets of pages, after making two separate PDFs, then split the resulting any2DjVu pages and merge them in an interleafing pattern. I remember the first time I tried this, thinking that the merged final DjVu would "confuse" the viewer and would not work. But I was overjoyed when it did work.
I had originally thought that a given shared dictionary would only work across consecutive pages, especially when that is how shared dictionaries are usually encoded.
As mentioned, I have another type of DjVu page that I have been working with, in addition to Text pages, Table pages, and JPEG pages. It is the Title Page (and similar-looking Subtitle Pages), usually for each Chapter/Subchapter/Table in an ebook. Each of these pages uses large areas of solid color. I think each page is the type that is useful for the cpaldjvu function. But I found that I could get much smaller file sizes than the cpaldjvu ones if I first encode such a page as a cjb2 and then add in a few carefully positioned FGbz chunks. These smaller file sizes occur even without using the shared dictionary.
Would you be interested in seeing a few Title/Subtitle pages some time and deciding if they are being encoded optimally? We could try a few more comparisons. Then, for me at least, we would have covered all the key bases. I see that cpaldjvu does not allow for shared dictionaries. But this making of a cjb2 version and then adding extensive FGbz coloration areas makes for a kind of pseudo-cpaldjvu.
Thanks again.
Stephen

[trufanov-nok]

Hi Stephen,
Yes I'm aware about FGbz chunks. There are a few method which allow to get lossy encoded but still colored text. And Fgbz is a most simple of them. Basically Fgbz chunk contains only a set of rectanle coordinates and colors for them. DjVu renderer must paint text which is inside such boxes or touches them with a specified color. Thus this method of text coloring has a very little overhead. If you are fine with solid color text and your color text not connected with elements of different color - it's the best.
FGbz chunks might be set via DjVuLibre's djvumake console tool. Unfortunately this require djvu page disassembling and assembling again.

Btw, I didn't tried but it looks like you can get the proper coordinates of the rectangles for djvumake in DjView4 viewer. You shall enable "Enable advanced features in menus" in Settings\Preferences\ Advanced tab. Then hold Ctrl and select the rect on page. There will be "Copy Maparea" command in context menu that pops up when you release mouse button. It will copy into the clipboard something like:

(maparea "url" "comment" (rect 1572 2642 359 318))

This is actually for setting clickable zones and popup comments via djvused, but the rect part should contain the right coordinates for color FGbz zone too.

Also, I've already implement the setting of color zones for FGbz chunk (via djvumake) in the upcoming ScanTailor 0.3.0 which will support assembling of processed pages into the bundled DjVu document. It's in a pretty same way as it is done in DjView4 as I'm reusing it's controls to display djvu pages. The ScanTailor 0.3.0 is supposed to be based on a stable minidjvu-mod release.

(A page with 3 color zones set)

(Foreground tab paints the rects of the FGbz zones which change color for every character touching them)

I would not add FGbz encoding directly into minidjvi-mod bcs this would imply re-encoding of the whole shared dictionary at every color or zone coordinates change. While it requires only FGbz chunk replacement. It would have much more sense if DjVuLibre's djvused could do this. Still the current approach with djvuextract/djvumake is not so difficult.

[maple7-7-7]

Thanks, Alex!
A very detailed and pretty post. Almost like watching a movie!
Thank you also for the tip on how to quickly set coordinates to be used with colorizing Sjbz data. I was always amazed at how much gets done with so few bytes with the FGbz colorization process.
I seem to recall when I experimented with FGbz that even if two pages looked identical, you could not always transfer a newly created FGbz chunk to the other identical-looking page unless everything else lined up, too, especially the dictionaries. It is a much different setup than just adding a hidden text layer on top, even though it looks similar in how you might do that.

[trufanov-nok]

I've updated minidjvu-mod to ver 0.9m06. It must be better now with tables encoding and encoding "corners" you use.
Still there are issues with bold font detection which is highlighted in another thread. Hope to fix this in further releases.

[maple7-7-7]

Thanks, Alex!
By the way, I was at a translated page of this address:
www.linux.org.ru/news/opensource/16412492/page1#comments
and some of the commenters seem to get angry about the fact that people are trying to improve DjVu, like it is the end of the world or something!
It is like they put $100 of Anger into $5 of DjVu issues!
Thanks again.