Automatic batching/instancing of draw commands

[superdump]

Objective

• Implement the foundations of automatic batching/instancing of draw commands as the next step from GPU Instancing #89
  • NOTE: More performance improvements will come when more data is managed and bound in ways that do not require rebinding such as mesh, material, and texture data.

Solution

• The core idea for batching of draw commands is to check whether any of the information that has to be passed when encoding a draw command changes between two things that are being drawn according to the sorted render phase order. These should be things like the pipeline, bind groups and their dynamic offsets, index/vertex buffers, and so on.
  • The following assumptions have been made:
    • Only entities with prepared assets (pipelines, materials, meshes) are queued to phases
    • View bindings are constant across a phase for a given draw function as phases are per-view
    • batch_and_prepare_render_phase is the only system that performs this batching and has sole responsibility for preparing the per-object data. As such the mesh binding and dynamic offsets are assumed to only vary as a result of the batch_and_prepare_render_phase system, e.g. due to having to split data across separate uniform bindings within the same buffer due to the maximum uniform buffer binding size.
• Implement GpuArrayBuffer for Mesh2dUniform to store Mesh2dUniform in arrays in GPU buffers rather than each one being at a dynamic offset in a uniform buffer. This is the same optimisation that was made for 3D not long ago.
• Change batch size for a range in PhaseItem, adding API for getting or mutating the range. This is more flexible than a size as the length of the range can be used in place of the size, but the start and end can be otherwise whatever is needed.
• Add an optional mesh bind group dynamic offset to PhaseItem. This avoids having to do a massive table move just to insert GpuArrayBufferIndex components.

Benchmarks

All tests have been run on an M1 Max on AC power. bevymark and many_cubes were modified to use 1920x1080 with a scale factor of 1. I run a script that runs a separate Tracy capture process, and then runs the bevy example with --features bevy_ci_testing,trace_tracy and CI_TESTING_CONFIG=../benchmark.ron with the contents of ../benchmark.ron:

(
    exit_after: Some(1500)
)

...in order to run each test for 1500 frames.

The recent changes to many_cubes and bevymark added reproducible random number generation so that with the same settings, the same rng will occur. They also added benchmark modes that use a fixed delta time for animations. Combined this means that the same frames should be rendered both on main and on the branch.

The graphs compare main (yellow) to this PR (red).

3D Mesh many_cubes --benchmark

The mesh and material are the same for all instances. This is basically the best case for the initial batching implementation as it results in 1 draw for the ~11.7k visible meshes. It gives a ~30% reduction in median frame time.

The 1000th frame is identical using the flip tool:

     Mean: 0.000000
     Weighted median: 0.000000
     1st weighted quartile: 0.000000
     3rd weighted quartile: 0.000000
     Min: 0.000000
     Max: 0.000000
     Evaluation time: 0.4615 seconds

3D Mesh many_cubes --benchmark --material-texture-count 10

This run uses 10 different materials by varying their textures. The materials are randomly selected, and there is no sorting by material bind group for opaque 3D so any batching is 'random'. The PR produces a ~5% reduction in median frame time. If we were to sort the opaque phase by the material bind group, then this should be a lot faster. This produces about 10.5k draws for the 11.7k visible entities. This makes sense as randomly selecting from 10 materials gives a chance that two adjacent entities randomly select the same material and can be batched.

The 1000th frame is identical in flip:

     Mean: 0.000000
     Weighted median: 0.000000
     1st weighted quartile: 0.000000
     3rd weighted quartile: 0.000000
     Min: 0.000000
     Max: 0.000000
     Evaluation time: 0.4537 seconds

3D Mesh many_cubes --benchmark --vary-per-instance

This run varies the material data per instance by randomly-generating its colour. This is the worst case for batching and that it performs about the same as `main` is a good thing as it demonstrates that the batching has minimal overhead when dealing with ~11k visible mesh entities.

The 1000th frame is identical according to flip:

     Mean: 0.000000
     Weighted median: 0.000000
     1st weighted quartile: 0.000000
     3rd weighted quartile: 0.000000
     Min: 0.000000
     Max: 0.000000
     Evaluation time: 0.4568 seconds

2D Mesh bevymark --benchmark --waves 160 --per-wave 1000 --mode mesh2d

This spawns 160 waves of 1000 quad meshes that are shaded with ColorMaterial. Each wave has a different material so 160 waves currently should result in 160 batches. This results in a 50% reduction in median frame time.

Capturing a screenshot of the 1000th frame main vs PR gives:

     Mean: 0.001222
     Weighted median: 0.750432
     1st weighted quartile: 0.453494
     3rd weighted quartile: 0.969758
     Min: 0.000000
     Max: 0.990296
     Evaluation time: 0.4255 seconds

So they seem to produce the same results. I also double-checked the number of draws. main does 160000 draws, and the PR does 160, as expected.

2D Mesh bevymark --benchmark --waves 160 --per-wave 1000 --mode mesh2d --material-texture-count 10

This generates 10 textures and generates materials for each of those and then selects one material per wave. The median frame time is reduced by 50%. Similar to the plain run above, this produces 160 draws on the PR and 160000 on `main` and the 1000th frame is identical (ignoring the fps counter text overlay).

     Mean: 0.002877
     Weighted median: 0.964980
     1st weighted quartile: 0.668871
     3rd weighted quartile: 0.982749
     Min: 0.000000
     Max: 0.992377
     Evaluation time: 0.4301 seconds

2D Mesh bevymark --benchmark --waves 160 --per-wave 1000 --mode mesh2d --vary-per-instance

This creates unique materials per instance by randomly-generating the material's colour. This is the worst case for 2D batching. Somehow, this PR manages a 7% reduction in median frame time. Both main and this PR issue 160000 draws.

The 1000th frame is the same:

     Mean: 0.001214
     Weighted median: 0.937499
     1st weighted quartile: 0.635467
     3rd weighted quartile: 0.979085
     Min: 0.000000
     Max: 0.988971
     Evaluation time: 0.4462 seconds

2D Sprite bevymark --benchmark --waves 160 --per-wave 1000 --mode sprite

This just spawns 160 waves of 1000 sprites. There should be and is no notable difference between main and the PR.

2D Sprite bevymark --benchmark --waves 160 --per-wave 1000 --mode sprite --material-texture-count 10

This spawns the sprites selecting a texture at random per instance from the 10 generated textures. This has no significant change vs main and shouldn't.

2D Sprite bevymark --benchmark --waves 160 --per-wave 1000 --mode sprite --vary-per-instance

This sets the sprite colour as being unique per instance. This can still all be drawn using one batch. There should be no difference but the PR produces median frame times that are 4% higher. Investigation showed no clear sources of cost, rather a mix of give and take that should not happen. It seems like noise in the results.

Summary

Benchmark	% change in median frame time
many_cubes	🟩 -30%
many_cubes 10 materials	🟩 -5%
many_cubes unique materials	🟩 ~0%
bevymark mesh2d	🟩 -50%
bevymark mesh2d 10 materials	🟩 -50%
bevymark mesh2d unique materials	🟩 -7%
bevymark sprite	🟥 2%
bevymark sprite 10 materials	🟥 0.6%
bevymark sprite unique materials	🟥 4.1%

---

Changelog

• Added: 2D and 3D mesh entities that share the same mesh and material (same textures, same data) are now batched into the same draw command for better performance.

[superdump]

I think this is ready for an initial review pass. If people are on-board with this, then I think the batching can probably be made more generic and reusable and then systems handling batching for both core and custom phases can be added more easily. Perhaps there is also a need to be able to opt-out of automatic batching if that's necessary for people to be able to do completely custom stuff. I think if the entity would just not match the object query, then the prepare systems would skip those entities/phase items.

[superdump]

I just noticed that this breaks many_foxes. I'll look into that.

[superdump]

So the reason many_foxes broke was because the batching assumes ownership of the mesh bind group, and that isn't quite true. Skinning and morph targets customise the mesh bind group to additional bind one or two other buffers at a dynamic offset per mesh entity, if a skin or morph target are present in the mesh. I've thrown in an initial hack as a separate commit that adds skinning and morph targets to MeshFlags which are per-entity mesh flags, and then if either of the flags are set on the current mesh entity, the batch gets split to be able to re-specify the dynamic offsets. This works, and it isn't necessarily a bad thing that those two properties are written to the MeshFlags and made available in the shader, but at the same time, I'm not sure I think it's the right solution. I just don't have a better idea right in this moment.

[superdump]

I'll probably add some batching tests when I can.

[rparrett]

These examples don't appear to be working properly. The rest seem OK. (29b8a29 was tested, all examples, comparing screenshots)

custom_gltf_vertex_attribute
many_foxes
mesh2d_manual

[superdump]

Right, many_foxes should be fixed now. I’ll check the other two. Thanks!

EDIT: Fixed.

[JMS55]

We may want to consider renaming MeshUniform to ObjectUniform or something.

[superdump]

Yeah, I want to call it PerObjectData or PerInstanceData. Or that without Per. ObjectUniform is also a decent suggestion. I’ve avoided ‘uniform’ because it feels like it implies the data is stored in a uniform buffer and it may not be.

[JMS55]

I like PerInstanceData imo.

[superdump]

Me too. I did have a concern about using the term instance and its clash with instanced rendering. But, I'm coming to think that it isn't a clash. We are using instance indices to look up the per-instance data. That they aren't in an instance-rate vertex buffer (note that being instance-rate is a qualifier to indicate that the data in that vertex buffer is per-instance and is stepped as instances are stepped) doesn't matter.

@cart any opinion on this renaming? Also, what about 2D vs 3D meshes? Per2dInstanceData? PerInstance2dData? PerInstanceData2d? I think I prefer Per2dInstanceData as an initial reaction.

[robtfm]

first pass - i haven't done any testing yet, just a read through. the perf benefits are looking awesome though.

[superdump]

I revisited my notes in #89 (comment) and I see that when writing that, I didn't consider significant batching for CPU-driven rendering.

I noted two things:

We need to also check the dynamic offsets used for the material, if any.

I haven't implemented this. Currently the only way for a Material to have a dynamic offset as part of its binding is if a custom material contains a dynamic offset uniform/storage buffer as one of the material members. I feel like we could live with this and fix it in a separate PR if/when someone notices it is missing. Otherwise there are more gains to be had that benefit more people from focusing on other efforts.

One more problem is that objects may be / are commonly observed from multiple views, for example as soon as shadow mapping is enabled. But even doing something like split-screen, how do you decide which view should define the order of data in the per-object binding that is being built in this system? As an initial solution I have punted the problem and I just duplicate the per-object data. A follow-up could write the per-object data to an array and then store another binding with indices into that array so that only the index is duplicated, not the entire per-object data.

This approach has carried over. The same per-object data would be written multiple times to the GpuArrayBuffer. I have noted how to improve it, but I think it can also be done in a separate PR and isn't critical for an initial implementation. It will hurt users of prepasses and shadows the most I guess.

---

I think the best we will be able to do for CPU-driven will practically be reducing bindings so much so that encoding opaque draws is occasionally swapping material bindings and mostly just drawing instances of index/vertex ranges and nothing else. The rest will have to be GPU-driven.

[robtfm]

Currently the only way for a Material to have a dynamic offset as part of its binding is if a custom material contains a dynamic offset uniform/storage buffer as one of the material members.

materials can't currently use dynamic offsets as SetMaterialBindGroup::render assumes there are none. So we are not losing any functionality.

The same per-object data would be written multiple times

it's still an improvement over the current status quo where a binding+draw is issued per mesh per view.

[cart]

Love how straightforward this is. Very uncontroversial and clear. Just a few comments.

[cart]

What is the story here for webgl? I thought webgl didn't support push constants (and im guessing wgpu is emulating them somehow with implicit uniforms or something?) What purpose does this serve?

[cart]

Could use a comment explaining this

[superdump]

This is a copy-pasted workaround for how the shader instance index built in is handled in the GL backend. WGSL defines the instance index as the base instance index + the instance being drawn in this sequence. So basically the range we pass to the draw command. But GL doesn't, it is just the instance in the sequence. As such, we work around it by passing a push constant, which is implemented as a glUniform with no buffer. This was the approach I found to work around the shortcoming of the GL backend and its deviation from the semantics of what instance index in WGSL should mean. This should really be fixed in wgpu.

[cart]

Seems like it would be safer to make "no automatic batching" the default and then opt in to automatic batching via a marker type. Is it done in reverse for perf reasons?

[superdump]

You could say that. I want people to get better performance by default. Why not batch if you can? And then if stuff breaks, opt-out or fix the things needed to get batching working.

[cart]

I think "automatic batching" should definitely apply to user facing types for things like PbrBundle, SpriteBundle, etc. But this is a RenderApp component. If someone is building a new rendered entity type with custom rendering logic, idk if we should batch that by default if there are "hidden" data access constraints that we don't check or enforce. Provided adding a AutomaticBatching component doesn't cost us anything significant for things in the hot path, I think it makes sense to opt-in things like sprites, 2d + 3d meshes, etc.

[rparrett]

Interestingly, now that bevymark is using random z values by default, I am no longer seeing a performance improvement in mesh2d mode with this PR. I still see ~2x fps with --ordered-z.

[superdump]

@rparrett yeah. That's because it practically has to re-bind the material between every draw as they're no longer in groups. This illustrates the cost of rebinding to update a dynamic offset between draws.