Integrating GLMakie plots into CairoMakie

[asinghvi17]

When writing the new CairoMakie rasterization pipeline, I realized that it would be surprisingly easy to integrate CairoMakie with GLMakie for the currently unsupported plot types (volume, with an option for mesh, surface and meshscatter). Essentially, all that one would need to do is to somehow copy the appropriate plots into a sub-Scene of the Axis scene, and then render that to a colorbuffer via GLMakie. After that, one could easily display the rendered image in lieu of the plot.

I think this might take the form of an optional external package (one might call it CairoGLMakie.jl) which simply pirates or overrides CairoMakie's draw_atomic functions.

Challenges would include dynamically resizing the child Scene for better resolution and then deleting it after the rendering process is done (which may lead to performance problems if used in recordings).

With this, we could switch quickly between CairoMakie (performance) and GLMakie (speed and better 3d rendering).

[asinghvi17]

I wrote a basic prototype for volume using GLMakie.scene2image, but there are still a couple of issues and the bounding box is a bit off :D (also the axis spines somehow don't appear, maybe that's a consequence of the image but I doubt it)

volume.pdf

The code

using CairoMakie
using CairoMakie: Cairo, CairoScreen, draw_atomic

using CairoMakie.Makie
using Makie: resize!, pixelarea, ARGB32

using GLMakie: scene2image

function CairoMakie.draw_atomic(scene::Scene, screen::CairoScreen, primitive::Volume, scale::Number = 1)

    w, h = Int.(scene.px_area[].widths)

    # We create a dummy scene to render to, which inherits its parent's
    # transformation and camera
    render_subscene = Scene(
        scene;
        show_axis = false,
        camera = camera(scene),
        lights = scene.lights,
        ssao = scene.ssao,
        backgroundcolor = RGBAf(0,0,0,0) # transparent
    )
    # Store the current pixel area for reset later
    # Otherwise this changes the layout
    old_pixelarea = Makie.pixelarea(render_subscene)[]
    # Resize the rendering subscene to the new size (for high resolution images)
    Makie.resize!(render_subscene, (pixelarea(render_subscene)[].widths .* scale)...)
    # Set the camera to look at the whole scene.
    Makie.update_cam!(render_subscene, cameracontrols(render_subscene), boundingbox(scene))
    # Push the provided primitive to the rendering scene
    push!(render_subscene, primitive)

    img, gl_screen = GLMakie.scene2image(render_subscene)
    save("hi.png", img)

    # Clean up and remove render subscene
    resize!(render_subscene, old_pixelarea.widths...)
    empty!(render_subscene.plots)
    # Makie.disconnect!(render_subscene.camera)
    Makie.update_cam!(render_subscene, EmptyCamera())
    empty!(render_subscene.theme)
    empty!(render_subscene.children)
    empty!(render_subscene.current_screens)
    pop!(scene.children)

    # return img

    surf = Cairo.CairoARGBSurface(CairoMakie.to_uint32_color.(img))

    Cairo.rectangle(screen.context, 0, 0, w, h)
    Cairo.save(screen.context)
    Cairo.translate(screen.context, 0, 0)
    Cairo.scale(screen.context, w / surf.width, h / surf.height)
    Cairo.set_source_surface(screen.context, surf, 0, 0)
    p = Cairo.get_source(screen.context)
    Cairo.pattern_set_extend(p, Cairo.EXTEND_PAD) # avoid blurry edges
    Cairo.pattern_set_filter(p, Cairo.FILTER_NEAREST)
    Cairo.fill(screen.context)
    Cairo.restore(screen.context)

    return

end

[asinghvi17]

Note for the future, let this be a flag for the Scene, and just render the whole thing. Then, CairoMakie can skip that scene in the rendering procedure.

[asinghvi17]

This is blocked by the fact that GLMakie doesn't have transparency in image output. CC @SimonDanisch - how hard would this be to add? I tried a bit by changing some types around, but am really unfamiliar with the GLMakie code.

Because of this issue, any plot drawn "behind" the rendered Scene (in z-order) is blocked by the white background of the Scene.

With transparency, it would be possible to rasterize individual plots instead of trying to hack Scenes as well.

[ffreyer]

how hard would this be to add?

The first step would be to allow RGBA outputs in colorbuffer. That part isn't hard, though it seems like rendering requires an opaque background. Might be all the stencil stuff we do, which is also limiting the number of scenes we can have...

[asinghvi17]

Hmm, interesting - I was not aware of that aspect. I had tried to add RGBA color outputs to GLMakie's colorbuffer a while ago, but it still had the white background. I had looked a little deeper into this at the time and it seems that it is possible to do, but the process is a little involved.

[ffreyer]

I played around a little bit with it. The colorbuffer in OpenGL is alread an RGBA buffer, so you just have to make a few adjustments in screen.jl. Getting things to render with a transparent background is more difficult. If you use a plain Scene with clear = true (default; clear as in clearing the screen) nothing renders (perhaps because stencil buffer doesn't get set correctly?) and with clear = false you do, but it everything ends up fully transparent in colorbuffer(). Maybe there is some blending going wrong in that case.

As far as I remember order independent transparency also requires or assumes an opaque background.