Tracking issue: Documenting Vello's architecture

[PoignardAzur]

In the process of the 0.1 release, I've written an ARCHITECTURE.md document. However, while that document lists files and resources, it only skims the conceptual architecture of Vello.

Right now, as far as I'm aware, the best resources for learning about Vello (besides reading the code and asking questions on Zulip) are the Vello pipeline diagram and the docs page listing Raph's blog articles.

The diagram

I've written my feedback to the diagram in a zulip thread:

• The diagram is very arrow-rich, which makes it hard to read. Any other changes should move towards making arrows fewer and shorter.
• The bump-allocation buffer symbols could probably be put directly into their respective "yellow box". At glance, only two of them are shared between multiple passes. For these you probably want to duplicate the box but have a dashed line between the two to remind the buffer is the same.
• The diagram is fairly horizontal, but you can expect it to show up in pages which will have a vertical layout. As such, it would probably be better to have the difference "stages" to be in a strict vertical order (Inputs -> Input Processing -> Binning -> Coarse Rasterization -> Segment Sort -> Fine Rasterization -> PostScript tiger).
• All boxes should be order in such away that arrows always flow to the right and down. Right now, "Path", "Tile" and "SegmentCount" have left-flowing arrows, which makes the flow hard to follow.
• There should be a legend indicating what the different color boxes mean.

I think this feedback still stands.

Overall, a better architecture diagram could be a great launching point for a documentation effort; but of course, that's easier said than done.

The blog articles

Our doc page mentions these articles about Vello:

• Requiem for piet-gpu-hal
• piet-gpu progress: clipping, Feb 24, 2022
• Fast 2D rendering on GPU, Jun 13, 2020
• A sort-middle architecture for 2D graphics, Jun 12, 2020
• piet-gpu progress report, Jun 1, 2020
• 2D Graphics on Modern GPU, May 8, 2019

And these articles about GPU compute:

• The stack monoid revisited, May 13, 2021
• Prefix sum on portable compute shaders, Nov 17, 2021
• The stack monoid, Sep 5, 2020
• Prefix sum on Vulkan, Apr 30, 2020
• GPU resources, Feb 12, 2020

My recap of the Vello articles

In chronological order:

• 2D Graphics on Modern GPU

  • Describes how a scene graph can be seen as a DAG of transforms, paths and operations.
  • Describes an early pipeline of "tiling" -> "rendering"
  • Covers general compute-related concepts; eg serialization formats, parallel culling, what "GPU threads" mean, etc.

• piet-gpu progress report

  • Introduces piet-gpu-hal.
  • Discusses the notion of render layers, and the introduces the notion of a reusable chunk of scene description (what later became SceneFragment).
  • Discusses notions of performance, and the design goal of moving as much work as possible onto the GPU.

• A sort-middle architecture for 2D graphics

  • Introduces the idea that prefix sum can be used for graphics rendering.
  • Describes an architecture where canvas operations are represented as a single stream, where the monoid is the canvas state that the operation modifies.
  • Describes the general pipeline of "binning" -> "coarse rasterization into tiles" -> "sort each tile" -> "fine rasterization". (Though it's rather light on details about when/how sorting actually happens)

• Fast 2D rendering on GPU

  • Goes into more detail on the sort-middle pipeline.
  • Mentions backdrop processing and curve flattening.
  • Mentions better performance numbers.

• piet-gpu progress: clipping

  • Describes the process of clipping in vector graphics; mentions the strategy of clipping as a Porter-Duff op.
  • Describes the fine rasterization stage of the pipeline.
  • Introduces the idea of using the stack monoid for bbox intersection.
  • Explains how operations can be encoded as multiple compact streams.

• Requiem for piet-gpu-hal

  • Goes over the pains of interoperability
  • Mentions the pain points and potential future improvements of WebGPU.

My recap of the posts on GPU compute

In chronological order:

• Prefix sum on Vulkan

  • Goes over the Vulkan memory model.
  • Talks about the performance model as it relates to subgroup size.
  • Lists portability considerations and abstraction layers

• The stack monoid

  • Describes a first version of The Stack Monoid where the payload is an actual stack

• The stack monoid revisited

  • Describes an in-place version of the stack monoid where the payload is a reduction of all the stack elements (eg for a clip stack, the intersection of all the clip rects).

• Prefix sum on portable compute shaders

  • Lays out a progress report of trying to implement decoupled lookback in Metal.
  • Explains how Metal lacks the necessary synchronization primitives to implement it, therefore WebGPU lacks them too.
  • Explains the associated outreach work (eg filing issues with the WebGPU spec so that it doesn't falsely advertise support for these synchronization primitives).

These blog posts aren't great documentation

A lot of these articles explore Vulkan, Metal, memory models, the limitations of such and such platform, etc, in a messy way. They overlap and cover designs we moved on from. They reflect Raph's thought process as he was trying different things in sequence and recording why they didn't work, and his understanding solidified.

In particular, there are three areas where these articles are not ideal as documentation:

• The Vello encoding has evolved a lot since the first articles were written. The "scene graph as a literal graph of paths and transforms" concept has been dropped. The single-stream approach has been replaced by packed multiplexed streams.
• The descriptions of individual pipeline steps are confusing for novices. I've read the sort-middle article at least five times and I'm still not clear what is being sorted. This isn't a dig at Raph: explaining this stuff is hard, and it requires a lot of visual aids. Making these visual-aids is time-consuming and we don't have great tools to produce them.
• These articles focus a lot on historical info: Vulkan, Metal, hardware abstraction layers, decoupled-lookback, whether we want to adopt WebGPU, why we don't want to adopt WebGPU yet, etc. Now that we've adopted WebGPU and decided we'll do without decoupled-lookback for now, we don't need to dump all these details on newcomers. We still want to say "here's all the things we considered and why we can't get them on WebGPU" just not as the primary focus.

My ideal documentation

Ideally, I'd like our documentation to include:

• An explainer on prefix sum. (Which doesn't mention decoupled lookback.)
• An explainer on prefix sum with the stack monoid.
• The pipeline diagram.
• A short article describing each stage of the pipeline, with visual aids.
• An overview article with a links to each article above and a summary of their content.

Obviously this is easier said than done.

Any incremental work that gets our documentation closer to that state would be very welcome.

[simbleau]

This is a great recap and summary of the architecture, but we should also talk about how we will make the documentation available to the users and help them accomplish their tasks.

"Who is our customer?" for this documentation.

I 100% agree the blogs are not good documentation. That being said, I also want to be candid and say probably 1% of people using Vello have a desire to contribute. Most people want to use vello, not learn it. Similarly, those needing to use wgpu/vulkan don't care about how the graphics calls translate to machine instructions.

These areas of focus seem more interesting:

• A tutorial series like https://sotrh.github.io/learn-wgpu/
  • A TLDR section on architecture (similarly, why Vello is probably your best choice right now)
  • Dedicated section to the cases like SVG/Bevy/Lottie
• A pull request template
• A contributing guide

I don't think it would be too hard to get off the ground with something like docusaurus.
I'm happy to contribute to the docs as well

[PoignardAzur]

I mean, this is very much a "I don't have time to do this, anyone else feel free to" post. I do hope it inspires someone (or even myself in the future) to go and do the thing. If you get enough momentum to get it off the ground, I'd be super grateful.

[PoignardAzur]

A contributing guide

Any thoughts in particular? Beyond the general "clone the project, run the tests, don't harass the maintainers" advice?

[nicoburns]

I feel like documenting the concepts behind Vello from a user perspective would be useful. As a user I don't need to know about what a sum prefix or a stack monoid is, but I do want to know what a Scene is and how it relates to a Surface, RenderSurface, Renderer, RenderContext, etc.

[simbleau]

A contributing guide

Any thoughts in particular? Beyond the general "clone the project, run the tests, don't harass the maintainers" advice?

Nothing in particular there.

Just that we should be focusing on the customer stories, like Nico said: "As a user..." e.g. "How can I get involved contributing? How can I draw a triangle or curve? How can I render an SVG?" and go from there, rather than trying to explain the universe.

So to be clear, I'm not against explaining what a prefix sum or a stack monoid is, but that should be tucked away in "extra reading", not part of core tutorial/documentation.

[armansito]

The diagram

I've written my feedback to the diagram in a zulip thread:

• The diagram is very arrow-rich, which makes it hard to read. Any other changes should move towards making arrows fewer and shorter.
• The bump-allocation buffer symbols could probably be put directly into their respective "yellow box". At glance, only two of them are shared between multiple passes. For these you probably want to duplicate the box but have a dashed line between the two to remind the buffer is the same.
• The diagram is fairly horizontal, but you can expect it to show up in pages which will have a vertical layout. As such, it would probably be better to have the difference "stages" to be in a strict vertical order (Inputs -> Input Processing -> Binning -> Coarse Rasterization -> Segment Sort -> Fine Rasterization -> PostScript tiger).
• All boxes should be order in such away that arrows always flow to the right and down. Right now, "Path", "Tile" and "SegmentCount" have left-flowing arrows, which makes the flow hard to follow.
• There should be a legend indicating what the different color boxes mean.

I think this feedback still stands.

Overall, a better architecture diagram could be a great launching point for a documentation effort; but of course, that's easier said than done.

I've seen your feedback and I'm going to try to make time soon to address it. I think your note around the density of arrows is valid. Raph and I have discussed making different level-of-detail versions of the diagram and I think that's worth doing (I'll start with this and adding a legend).

The current diagram is intentionally laid out to fully capture the data flow between the architecture stages and prioritizes serving as a detailed map, and I would like to keep it that way. I think your other comments about the visual direction of the flow can be addressed by adding different lower LOD versions, derived from the bigger diagram.

My ideal documentation

Ideally, I'd like our documentation to include:

• An explainer on prefix sum. (Which doesn't mention decoupled lookback.)
• An explainer on prefix sum with the stack monoid.
• The pipeline diagram.
• A short article describing each stage of the pipeline, with visual aids.
• An overview article with a links to each article above and a summary of their content.

Obviously this is easier said than done.

Any incremental work that gets our documentation closer to that state would be very welcome.

I agree, and I think many of these can be in Wiki pages under this repository. I wrote some of this documentation before while adding the vello engine definitions in Skia. I think I can adapt much of that into a wiki page (or some markdown) to accompany the diagram.

Having that documentation is important for those who may want to contribute to the guts of Vello or build a custom engine with the shaders and encoding crates. I'd be happy to contribute to this as much as possible.

I also agree with @nicoburns and @simbleau that the project really needs some sort of user guide / documentation.

[simbleau]

For the record, I'd like a dedicated site hosted on a static site generator like Docusaurus over GitHub Wiki.

[DJMcNab]

I think the Linebender wiki would be the obvious place for this to go