Thoughts on transformations

[jkrumbiegel]

Most of this draws from https://matplotlib.org/users/transforms_tutorial.html, they had this stuff figured out pretty well already.

We probably need (at least?) 4 different levels of transformations in Makie:

• Display (actual pixels)
  • allows you to place anything pixel perfect in the scene
  • allows all kinds of absolute placements or measures that are not supposed to change with scaling of the plot
    • so everything like font sizes, marker sizes, line thickness, absolute margins and paddings, this all needs to go be specified in the display transform
    • if unit support is built in, it also works on this level: pt, mm, dip, px, all these things are in the end convertible to a specific pixel number, so can be used interchangeably
  • allows picking of things that are being clicked on by going down the transform chain from here
• Figure / Scene
  • if just one scene exists this covers the whole image, with the lower left corner being (0, 0) and the upper right (1, 1))
    • because in Makie, subscenes are allowed, this would need to be more flexible to allow nesting, but it's still basically the same thing
  • allows you to place e.g. a title centered at the top if you put it at figure coordinates (0.5, 0.9)
  • allows placement of subplots in figure relative space
• Axis
  • for normal rectangular 2D axes these also go from (0, 0) at lower left to (1, 1) at upper right
  • allows to place text / annotations relative to the axis / subplot
  • for 3D axes it's probably more complicated because the boundaries can change when rotating the axes..
  • there are also logarithmic axes, polar axes, things that are a bit more complicated
  • still it is important for layouting purposes that an axis always separates between its inner part where the actual plot is and the outer part where labels etc are
    • basically, any kind of container that says "i want to be aligned with other container's important edges at these top, right, bottom and left lines" serves the same purpose and can be included in a layout engine, it doesn't have to be an actual axis
    • a layout engine would basically, as far as I understand it, and as far as the Rhea prototype works right now, always deal with the actual display level and try to align things in this space. So it would want to set the left and right edges of a figure at let's say 0px and 1000px of a window that is 1000px wide, the first subplot's left edge would be at 20px, etc. So it should be possible to calculate for all axis decorations how far away from the axis frames they reach (the axis area is the part that's flexible in the layout, the decorations don't change size). Let's say the left axis label is 10pt wide and 20pt away from the left axis spine, then the layout engine knows to reserve 30pt space on the left side of the axis and will adjust axis width accordingly.
• Data
  • this is the level at which the actual data lives, so that you can plot points from 0 to 1,000,0000 in one subplot next to another subplot that goes only from 0.1 to 0.5 but they're both the same size

Things that should be possible:

• create axis label text of 8pt size with 5mm gap to the axis boundary
• create scatter plot markers that are always 3dip wide with a stroke width of 0.5pt
• create scatter plot markers whose width is meaningful in data coordinates (and therefore increases / decreases in size when the axis limits are changed)
• an axis that spans half of the figure width and height with its main area (excluding labels)
• lines that always span the full (or a percentage of) height or width of an axis, but the other dimension is in data coordinates (axvline, axhline)
  • this is a blended transform, where the x and y parts refer to different reference frames (possible as long as the transforms are separable)
• zoom into an axis while leaving all text and lines intact, by simply changing how the data transform maps to the axis transform (which is equal to changing the x and y limits of the axis). right now zooming into an axis just magnifies everything, this could be a useful function sometimes but most of the time not what you want.
• more advanced blends: create a text annotation that is always 1/5th axis width + 20pt to the right of a point at (0.3, 0.3) in figure space
• every object in a scene should have a place in this chain (or tree) of transforms so you can calculate any conversion you want by going up and down this chain
• if these transformations / matrix multiplications all happen at the GPU level, the performance is much better
  • vispy does this, e.g., by building nested transforms as functions into GLSL code that is created dynamically through string interpolation. Only the parameters of these functions are then updated on the GPU. Maybe this is easier to realize with Julia's compilation powers?

Depending on how you think about it, figure and axis could both just be called a container, so you could merge the two and make multiple nested versions possible. This would integrate neatly with arbitrary nested grids from the layout engine.

[asinghvi17]

Thanks a lot, @jkrumbiegel! Just a couple of thoughts:

• As of right now, it wouldn't be too hard to do the axis/figure coordinates (although maybe they would be better off being called Scene coordinates, since Scenes are basically the merged containers you're talking about.

• One thing to be careful of is determining which container the relative coordinate is relative to, since it's Scenes all the way down.

[yakir12]

Just chiming in here, rotation of the camera in a zoomed-in scene should rotate around the new center of the data, not the center of the data before the zoom. Matlab got that fixed only in the last (few?) version.

[ffreyer]

This is quite old now, but still something we should figure out. #1472 is requesting it, and in places where it's needed we currently have hacky and/or specialized code to handle it (e.g, setting projection matrices in DataInspector or the space options in text and scatter) so I'm thinking about working on this next.

With respect to GL backends we already have the transformations for

• data coordinates given by the view and projection matrix in scene.camera
• screen/pixel coordinates given by scene.camera.pixelspace

and scene space is very simple because Mat4f(I) maps to a (-1, 1) x (-1, 1) x (0, 1) space. So it's not hard to render something completely in data, pixel or scene coordinates. (Axis coordinates should be transformed into scene coordinates on the CPU in my opinion.) What is more difficult is handling different spaces/units in different components or across different dimensions.

Some options for solving the former:

• Compile different shaders depending on the space a component/attribute is in. For this we may want shaders to recompile reactively. This should be high performance unless shaders need to recompile, but probably makes them harder to read.
• Add tags for each relevant component and do branching. Probably slow
• Duplicate the relevant components so that there is a data version and a frag space version. I think this would be the best for components that are per plot, but the worst for components that are per instance.

Adjusting shader compilation seems like the best approach to me. With respect to the second problem, I don't think we need to worry about it now. Ultimately we should always be able to project inputs to a unified space on the CPU if necessary. For the front end I really like Gadflys approach. Being able to write 20px or 0.7w looks very natural to me. (Though we probably don't want to export w and h.)

[SimonDanisch]

At least parts of it are addressed in the new scene tutorial on master:
https://makie.juliaplots.org/dev/tutorials/scenes/#interaction_with_axis_layouts

[ffreyer]

I don't generally like the approach of creating a new scene just for a different pixel or scene coordinates. Take DataInspector for example. There I want to create a tool tip when something is hovered. It should be attached to a position in data space, but I also want a background behind text to separate it from the rest of the visualization. The background needs to scale with text size, which is in pixel coordinates.

How am I supposed to handle this? Draw the text (and a highlighter) in the correct scene and the background in another? I'd rather keep all the tool tip plots together. So I manually project and move everything to a pixel scene? Seems rather cumbersome...

Another reason why I want this to work per plot is because I want to eventually try adding a postprocessing pipeline to scenes. I'm thinking of something like

for scene in flattened_scenes
    # clear buffer
    for (stage, postprocessor) in enumerate(scene.render_pipeline)
	render!(scene.plots, stage)
	apply!(postprocessor)
    end
    # copy buffer to main buffer
end
# to screen

which would mean clearing, drawing to and then copying from a temp buffer for each scene. I'm worried that might get expensive when lots of scenes are involved. (Though it might also be easy to combine scenes and if a scene only needs to render (i.e. no fxaa etc) it shouldn't be any more expensive than now.)

On another note I also took a look at the primitive plots in MakieCore. There aren't a lot of attributes that can be transformed. There are linewidths (lines, linesegments and heatmap, though heatmap breaks with it in GLMakie), strokewidth (scatter, text), markersize (meshscatter, scatter) and textsize (text). I'm not sure if there is a use in having transformations for linewidths and strokewidths. I can see markersize for meshscatter being useful though and the rest is alreadyworking. So there isn't a lot missing

[SimonDanisch]

I don't generally like the approach of creating a new scene just for a different pixel or scene coordinates.

Yeah I was thinking we could just have multiple projections per scene... We already kind of have that with camera.pixel_space... Then we can just do plot!(scene, ...; projection=...) and just use the correct projection matrix from the scene

[ffreyer]

projection can also be used to overwrite the projection matrix at the moment (at least in GLMakie). I'd rather keep that available. Maybe we just use space for input arguments and markerspace for textsize and markersize? In a later step we could then introduce units (px, w, mm, etc) and have space and markerspace become backend attributes.

[SimonDanisch]

projection can also be used to overwrite the projection matrix at the moment

But that's pretty much what we'd do by changing the space? You can't set a space and keep the same projection matrix...

[ffreyer]

What I mean is that right now you can do scatter(..., projection = Mat4f(I)). If we make projection = :data/:pixel/:relative you could not pass a matrix anymore unless we specifically check for that.

[SimonDanisch]

If we make projection = :data/:pixel/:relative you could not pass a matrix anymore unless we specifically check for that.

Why not? I think that's exactly what we should do:

projection: either a predefined projection like  `:data/:pixel/:relative` or a custom projection matrix

[SimonDanisch]

Counterpoint:
markspace, space is a much better duo...

[ffreyer]

Also these options would affect the view matrix (and the projectionview matrix) as well. Might be a bit misleading to group it with just the projection matrix