0.9m06 Pt 3 - 10x Smaller than cpaldjvu - any2djvu

[maple7-7-7]

Some new findings.
Take a Title Page with 2 dominant colors that makes such a page a candidate for the DjVuLibre program cpaldjvu. Create 100 such pages called "Section 1 . . . Section 100" as Black and White and also as Color. For minidjvu-mod and any2djvu, pages are colorized with FGbz chunks using djvumake.
Compare the results of 3 approaches to making DjVus:

1. Process with cpaldjvu (-colors 2 -bgwhite) no dicts available;
2. Submit to any2djvu (20pg dicts);
3. Process with minidjvu-mod0.9m06 with 20pgdicts and a 100pgdict.
   (for color pages, a BG of 63 bytes as a common denominator)
   Results: Extraordinary 10:1 Compression offered by minidjvu-mod.
   100 Title Pages: Color: 416 KB cpaldjvu no dict available (see PNG - on the left)
   100 Title Pages: Colorized 38 KB minidjvu-mod 100pg dict (see PNG - on the right)

4.8:1 - 7.4:1 Compression compared with Any2DjVu (Color)
100 Title Pages: Colorized 283 KB any2djvu 20pg dicts
100 Title Pages: Colorized 59 KB minidjvu-mod 20pg dicts
100 Title Pages: Colorized 38 KB minidjvu-mod 100pg dict

10:1 Compression compared with Any2DjVu (Black and White)
100 Title Pages: BW 273 KB any2djvu 20pg dicts
100 Title Pages: BW 27 KB minidjvu-mod 100pg dict

100_TITLES_CPALDJVU--ANY2DJVU--MINIDJVU-MOD_0.9m06.zip

[trufanov-nok]

I actually quite impressed with cpaldjvu results. I thought it works in a more simple way.

1. It contains BG44 chunk. That chunk is a background image which size is equal to the size of page and it's always filled with 1 color. So it's just 63 bytes in your case. This color is a lightest color (in your case - white) if -bgwhite is specified, and dominant color otherwise. Why it's needed? To compress the image. If user quantifies the image, like book cover image, which is a solid color + book title on it, it has sense to just fill background with a dominant color and draw text with FGbz color zones other it. I would suggest that compression of a book covers is a main use case that was expected for cpaldjvu user.
   I would not save BG chunk if it's a solid white at all, but authors don't bother about it.

2. It colorize foreground (text) layer as a FGbz chunk. How? In a very interesting manner: it quantifies all image colors to -colors N colorpalette (it will be stored in FGbz). Then for every pixel of a new color from color palette is assigned. And at this point, let's say, the image is split by connected components of the same color. That means if you have a letter V that is colored with 2 colors: \ is red and / is blue - it will be interpreted as a two different characters very close to each other while Sjbz encoding. This makes possible to have a single multicolored letter which is impossible to achieve by setting FGbz after Sjbz encoding is already completed. That is what I didn't expect to find out in cpaldjvu. It also looks like cpaldjvu will ignore characters of the color that match to the color of background at this step. That's how background color allows to compress the page.

3. It seems that the only reason why minidjvu-mod + FGbz chunk is smaller than cpaldjvu is the Sjbz compression. Especially the Sjbz compression of a "too big to be a letter" elements. And if we compress cpaldjvu multipage 100_TITLES_COLOR_CPALDJVU_nodict_600dpi__416_KB.djvu using Djbz we'll get a comparable result with minidjvu-mod. And there is a tool for that!
   Few years ago than I realized that DjVuLibre lacks jb2 encoder that could used shared Djbz dictionaries, I googled out a tool named jb2unify which takes a bundled multipage djvu document and recombines its Sjbz dictionaries to form a Djbz dictionaries. The problem is it was a very straightforward and put into Djbz dictionary only characters that are 100% match. So, it's lossless compression. And this tool didn't get much popularity bcs such lossless Djbz doesn't allow to achieve compression ratio comparable to commercial encoder anyway. But such lossless compression is exactly what minidjvu-mod does with graphics (corners and tables) on your pages.

So, we can compare two bumdled colored image generation chains:

1. minidjvu-mod + manual FGbz setting via djvumake
2. cpaldjvu + djvum bundling + jb2unify

Advantages of minidjvu-mod chain: lossy compression of a text layer
Disadvantages of minidjvu-mod chain: manual setting of color zones, can't represent multicolor letters

Advantages of cpaldjvu+jb2unify chain: no manual work to set color zones, multicolor letters
Disadvantages of cpaldjvu+jb2unify chain: only lossless compression, might have unnecessary BG44 chunk, upd: process whole document at once, no --page-per-dict-like option

The jb2unify project is here: https://github.com/velwant/jb2unify
The related thread on DjVuLibre discussion board: https://sourceforge.net/p/djvu/discussion/103286/thread/7ee1ba44/
And some backstage info that could be auto-translated from chezh: https://wiki.control.fel.cvut.cz/mediawiki/index.php/jb2unify

The jb2unify is an add-on for DjVulibre project and could be built on Linux with:

cd /tmp
git clone https://github.com/barak/djvulibre.git
git clone https://github.com/velwant/jb2unify.git

cp jb2unify/jb2unify.cpp djvulibre/tools/
cd djvulibre/
git apply ../jb2unify/Makefile.am.patch

autoreconf --install
./configure
make

I'll build and attach a windows binary later. Don't know if there is already built windows binary in network. It shall be 15 years old...

I've tried to apply jb2unify to 100_TITLES_COLOR_CPALDJVU_nodict_600dpi__416_KB.djvu and got 39,7 KiB (40 609) document, while minidjvu-mod's one is 37,2 KiB (38 139).

The experimental files are attached. There are pbm with multicolored letter, The result of it's encoding with cpaldjvu. And the jb2unify compressed version of 100_TITLES_COLOR_CPALDJVU_nodict_600dpi__416_KB.djvu
tests.zip

[maple7-7-7]

That is really interesting, Alex. Thank you! And thank you also for the new release.
Your releases are like the excitement I would have when my dad used to bring home for me as a boy the newest "Top 30 Hits" charts that used to be produced in "the good old days."

Paragraph 1: The bg of 63 bytes I actually created several years ago to substitute for the larger auto-generated one that DjVuLibre makes (288 bytes) when adding FGbzs to Sjbzs and no BG44 is specified. This was to save on file sizes. For cpaldjvu, its original BG44 here was 292 bytes white for the 600dpi DjVu. I chose to include the same 63 byte BG44 so that all bgs would be the same for each way of making a DjVu, since all the "action" seemed to be in the Sjbzs and the djbzs, and I wished to focus on those. So that 63 byte bg was used to save a bit on file size and to act as a constant.

Paragraph 2: Cool. It looks like, as you say, cpaldjvu has its own way of analyzing things.
"It also looks like cpaldjvu will ignore characters of the color that match to the color of background at this step. That's how background color allows to compress the page." Does this mean that if I have a background that is the same color as the table lines, or has colored zones that align with the table lines, we can get extra compression? I think the narrow table lines in the foreground are 6 pixels thick at 600 dpi. What about the possibility of somehow making instead of one long table line, several small table lines that touch and that could be recognized for compression better? I don't know. This thinking is getting too "advanced" for me! But maybe instead of asking the word processor to make a table, I manually create a table using lines of hyphens or dashes. (?) Then the table would not be seen as a graphic but as a collection of characters. (?) More work!

Paragraph 3: I had downloaded jb2unify a few months ago after you referred to it at sourceforge, but did not test it after reading that the compression was not that great, and especially now that we have the mod version of minidjvu. Of course I did not know about integrating it with cpaldjvu. It is amazing how you have been able to "turbocharge" cpaldjvu with jb2unify and get comparable results to minidjvu-mod + FGbzs. The limitation of no -p option is not a big deal if one simply works with a chosen number of pages at a time, and then bundles the bundles.

Thank you for the links, the zip file, and how to install jb2unify on Linux.
Stephen

[trufanov-nok]

Does this mean that if I have a background that is the same color as the table lines, or has colored zones that align with the table lines, we can get extra compression? I think the narrow table lines in the foreground are 6 pixels thick at 600 dpi. What about the possibility of somehow making instead of one long table line, several small table lines that touch and that could be recognized for compression better

Nothing of that shall work. Imagine a page with a blue border and black text inside it. Depending on thickness of a border we can consider it a white page with blue border and text (case of a thin enough border) or a blue page with a white rect inside and text in it (thick enough border). csepdjvu may estimate area of each color and choose dominant color as a background, so in second case you'll end up with a white rectangle and black text on a blue page instead of a blue border and text in first case. The problem is - the blue page case can contain artifacts if lossy compression is applied. That's why they need an option to force lightest color usage as a background. But cpaldjvu seems not to use lossy encoding... that better be checked.
Another case there dominant background would be a best option - is a white text on a black page.

[trufanov-nok]

Attached is DjVuLibre with patched djvumake (FGbz optimization) and jb2unify utility for Windows x64.
I couldn't build jb2unify with VS2013 due not full C99 standart support in its compiller (missing Variable Length Arrays). So this is build with MSYS2 (Mingw64). I hope I put all required libs in the package.

djvulibre-jb2unify_mingw64.zip

[maple7-7-7]

Thanks for the explanations and the zipped files.

[maple7-7-7]

Hi Alex,
I was wondering if it is possible to set the maximum size of the analyzing area at 250x250 instead of the usual 500x500 and see if we get something interesting. I could do the research if you think it is worth a try. Thanks.
Stephen

[trufanov-nok]

Hi, I don't think that would have sense. I would expect more fragments and more encoding overheads if this size is decreased. If we consider its increasing then I'm not sure if 500x500 is a technical specification limit or just an optimal max value. But as both commercial encoder and DjVuLibre's cjb2 uses the same I guess it's well tested (not only on tables but over a lot of different types of pages with different dpi) and would not change it.

[maple7-7-7]

Hi Alex, Yes, I remember you saying that the commercial and free versions both had the same 500x500 limits. Thanks.

…

________________________________ From: Alexander Trufanov ***@***.***> Sent: December 26, 2021 11:47 AM To: trufanov-nok/minidjvu-mod ***@***.***> Cc: maple7-7-7 ***@***.***>; Author ***@***.***> Subject: Re: [trufanov-nok/minidjvu-mod] 0.9m06 Pt 3 - 10x Smaller than cpaldjvu - any2djvu (Discussion #23) Hi, I don't think that would have sense. I would expect more fragments and more encoding overheads if this size is decreased. If we consider its increasing then I'm not sure if 500x500 is a technical specification limit or just an optimal max value. But as both commercial encoder and DjVuLibre's cjb2 uses the same I guess it's well tested (not only on tables but over a lot of different types of pages with different dpi) and would not change it. — Reply to this email directly, view it on GitHub<#23 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOBEQDVNN3ZDVGTUJOTIBYLUS36DXANCNFSM5GSGYFOA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[maple7-7-7]

Hi. Alex. I hope you are OK. I would like to resume our interaction but I need a little help with my old Linux Lubuntu 18.10 32-bit, which won't even open a 32-bit deb file, the lazy man's way of installing a package. The new Lubuntu version is 64-bit, which my lapotop cannot accommodate. Thanks.

[trufanov-nok]

Hi, how error message looks like?

[maple7-7-7]

Basically I don't seem to have permissions to do anything whatsoever. Not even updating things. Like a global blockade! Sorry.

[trufanov-nok]

Well, it won't be easy in this case. What dpkg -l "libjemalloc*" outputs?

[maple7-7-7]

Hi, Sorry for all this trouble, Alex. I can always use my Win7 laptop or Win10 desktop. But here is a screenshot for the Lubuntu 32-bit laptop. Most people don't use Linux in the non-Android sense, as you know. So I think it is still good to know how your Linux version of minidjvu-mod is doing.

…

________________________________ From: Alexander Trufanov ***@***.***> Sent: July 22, 2022 2:58 PM To: trufanov-nok/minidjvu-mod ***@***.***> Cc: maple7-7-7 ***@***.***>; Author ***@***.***> Subject: Re: [trufanov-nok/minidjvu-mod] 0.9m06 Pt 3 - 10x Smaller than cpaldjvu - any2djvu (Discussion #23) Well, it won't be easy in this case. What dpkg -l "libjemalloc*" outputs? — Reply to this email directly, view it on GitHub<#23 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOBEQDQ4CVP5CM4XZVTF36DVVKZKFANCNFSM5GSGYFOA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[trufanov-nok]

There is no any screenshots attached...

[maple7-7-7]

Sorry again. The screenshot is in the direct email only. I can't get github to show it even in a preview. Here it is again in the direct email. Maybe we should skip. Hate to waste your time.

…

________________________________ From: Alexander Trufanov ***@***.***> Sent: July 22, 2022 3:21 PM To: trufanov-nok/minidjvu-mod ***@***.***> Cc: maple7-7-7 ***@***.***>; Author ***@***.***> Subject: Re: [trufanov-nok/minidjvu-mod] 0.9m06 Pt 3 - 10x Smaller than cpaldjvu - any2djvu (Discussion #23) There is no any screenshots attached... — Reply to this email directly, view it on GitHub<#23 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOBEQDTZOLHNWTHIKX4S3KDVVK375ANCNFSM5GSGYFOA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[maple7-7-7]

Here is a pasted text version:
stephen@stephen-pc:~$ dpkg -l "libjemalloc*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=======================-================-================-====================================================
ii libjemalloc1 3.6.0-11 i386 general-purpose scalable concurrent malloc(3) implem
ii libjemalloc2:i386 5.1.0-1 i386 general-purpose scalable concurrent malloc(3) implem

[maple7-7-7]

Hi again, Alex.
I was in a discount store and bought a cheap but OK used Toshiba laptop with a dual-core 64-bit Pentium T4200, 3 GB RAM and have installed Lubuntu 22.04 LTS. Should I use the 64-bit Linux debs you have provided me previously to get minidjvu-mod experiments going again? Thanks. I will try to correspond with github.

[trufanov-nok]

Yes, I guess latest 64-bit deb package will work fine on 22.04.

[maple7-7-7]

Hi Alex,
Sorry to neglect you and your great minidjvu-mod program for so long. I think there might be another reason to try an adjustment to the 500x500 analyzed area parameter: the resolution of a given DjVu page might affect the efficiency. Are you up for making one 350x350 version of minidjvu-mod 0.9m06 Win64 that I could personally test? Or maybe just say what line might need changing? Thanks.