Cosmic Text for font rendering

[ElhamAryanpur]

As discussed in #1016 , I have done some testing on cosmic-text.

First of all, they have some docs that I've been exploring. And I was testing their example. They do have no_std and wasm support too according to the Cargo.toml they have.

I've noticed that the way it renders, is by drawing rectangles. Which is something I've seen for the first time to be honest. You can check it here. It gives a x,y,width,height, and color which is usually just color predefined and alpha channel being different for aliasing and stuff.

There is also another method if rectangles aren't possible: Swash Image which basically returns an image bytes to be rendered instead of individually creating rectangles. It requires some things I couldn't implement myself to be honest.

I was able to sort of hack the rectangle method in my engine

It had... not good results to be honest

Although maybe that's on me for having some issues with the engine as their examples do work and work very well. But yeah, that's my findings so far on cosmic text.

[thomaskrause]

I also experimented with cosmic text but went with hacking some completely inefficient and not really generic code in the tessellate_text function of the epaint Tessellator.

pub fn tessellate_text(&mut self, text_shape: &TextShape, out: &mut Mesh) {
        let TextShape {
            pos: galley_pos,
            galley,
            underline,
            override_text_color,
            angle,
        } = text_shape;

        if galley.is_empty() {
            return;
        }

        if galley.pixels_per_point != self.pixels_per_point {
            eprintln!("epaint: WARNING: pixels_per_point (dpi scale) have changed between text layout and tessellation. \
                       You must recreate your text shapes if pixels_per_point changes.");
        }

        for row in &galley.rows {
            let metrics =
                cosmic_text::Metrics::new(galley.job.sections[0].format.font_id.size, row.height());
            let mut buffer = cosmic_text::Buffer::new(&mut self.font_system, metrics);
            let mut buffer = buffer.borrow_with(&mut self.font_system);
            buffer.set_size(row.rect.width(), row.rect.height());

            let attrs = cosmic_text::Attrs::new();

            buffer.set_text(&row.text(), attrs, cosmic_text::Shaping::Advanced);
            buffer.shape_until_scroll();
            let text_color = override_text_color
                .map(|c| cosmic_text::Color::rgba(c.r(), c.g(), c.b(), c.a()))
                .unwrap_or_else(|| cosmic_text::Color::rgb(0xFF, 0xFF, 0xFF));

            buffer.draw(&mut self.swash_cache, text_color, |x, y, w, h, color| {
                let min_pos = *galley_pos + row.rect.min.to_vec2() + vec2(x as f32, y as f32);
                let size = vec2(w as f32, h as f32);
                let output_rect = Rect::from_min_size(min_pos, size);
                let ecolor =
                    Color32::from_rgba_unmultiplied(color.r(), color.g(), color.b(), color.a());
                out.add_colored_rect(output_rect, ecolor);
            });
        }
    }

Using cosmic text would mean to replace the whole pipeline of text layout and not just rendering, which seems to be quite a change, even possible in the public facing API. But it is definitely possible.

[ElhamAryanpur]

Interesting work! Can you type in some RTL text? e.g. (سلامم)

And yes I agree, it'll be a big change. User facing API's could stay the same still in my opinion, but the pipeline will need some change.

[thomaskrause]

Since the text layout part is still original, the example RTL text is placed weirdly and the font rendering is still quite blurry.

[emilk]

Very cool!

It seems like if we switch to cosmic text we need to switch from using a glyph atlas to a string atlas.

egui currently uses a glyph atlas, which is a big image/texture where each glyph (character) is rendered once and then stored so it can be referenced at rendering. Text layout is about generating UV-rects that point into the individual characters in the glyph atlas.
The glyph-atlas in egui can be found in Backend -> Inspection -> Font texture of egui.rs:

Cosmic Text seems to work by rendering a full string of text as a bitmap (the rectangles in buffer.draw are patches of pixels). This improves kerning and anti-aliasing of the text, but is more complex. We need to cache the results of the rendering. The easiest way would be to cache each text string in its own bitmap and stream that to the GPU each frame. That would mean a lot of texture switching in he render backend, which I believe would be quite slow for something like the WebGL2 backend.

15 years or go so I worked on a text renderer for Phun/Algodoo which used a string atlas for this: I combined all the small string-textures into one a few big texture atlases. Back then I did a lot of book-keeping in order to only update the atlas when a new piece of text was added and removed. There is a lot of complexity here, like atlas fragmentation, and handling of text too large to fit in an atlas. Perhaps these days GPU streaming is fast enough that we can construct the atlases once at the start of each frame and stream it to the GPU every frame. That would remove all the book-keeping from the problem.

[ElhamAryanpur]

I do not know font rendering, but can't there be a GPU cache of the bitmaps and be reused without having to create a new texture? that should be fast enough. although problem would be the issue in different sizes, could quickly grow in memory size.. hmm

I'd say GPUs nowadays are fast enough, and given WebGPU is right around the corner, this could be possible. I wonder how did the ICED achieved it.

[ElhamAryanpur]

Since the text layout part is still original, the example RTL text is placed weirdly and the font rendering is still quite blurry.

Oh interesting! ICED had similar issue, they have opened an issue on cosmic-text to define bounding sizes for fonts to solve it. I'm not sure how it works tho-

[emilk]

Oh interesting! ICED had similar issue, they have opened an issue on cosmic-text to define bounding sizes for fonts to solve it. I'm not sure how it works tho-

Do you have a link for that issue?

[ElhamAryanpur]

yes of course! pop-os/cosmic-text#70

[ElhamAryanpur]

related issue that explains further: iced-rs/iced#1877

[dignifiedquire]

maybe this helps as direction for how to solve the caching? pop-os/cosmic-text#26 (comment)

[mikeandmore]

I'm digging (trying...) into this. It looks like we need to draw textures and not polygons... buffer.draw() is terribly slow. It renders pixel by pixel...

[mikeandmore]

Here are some notes I have while reading the code.

1. layout_section() in text_layout.rs calls FontImpl::font_impl_and_glyph_info to look for glyph info for each (unicode?) character.
2. If the glyph isn't found, it'll render via FontImpl::allocate_glyph(). This function first allocates the texture with TextureAtlas::allocate(), and it draws on the FontImage. Interestingly, glyph.draw() is a similar API to the buffer.draw() in cosmic_text, which renders the glyph pixel-by-pixel.
3. TextureAtlas::allocate() returns a position in the FontImage. If not enough, resize the FontImage.
4. FontImpl keeps a reference of TextureAtlas, which owns a FontImage. In update_texture() in render.rs, it'll update upload the actual texture into the GPU.

[ElhamAryanpur]

interesting, I can hardly understand most of them 😅

[emilk]

This might be useful for reference: https://github.com/grovesNL/glyphon

[StratusFearMe21]

For people who need this right now, feel free to use this crate I just created https://github.com/StratusFearMe21/egui-glyphon