[Bug]: significant performance degradation in Nouveau indexing speed comparing 3.5.0 to 3.5.1

[mrjones-plip]

Version

CouchDB 3.5.1

Describe the problem you're encountering

From the great research that @dianabarsan did per the Slack conversation she started:

We're testing a database with ~1mil synthetic docs and bench-marks in CouchDB 3.5.0 indexing one Nouveau index took 184,828 ms. In CouchDb 3.5.1 the same index took 2,683,576 ms - a ~14x slow down!

Initially it was thought the Nouveau setting of maxIndexesOpen from 3000 to 100 might have caused this, but the times above have maxIndexesOpen=3000. Changing pool_size = 1000 did not help either.

For reference, benchmarks for 3.5.0 and for 3.5.1 and are the Nouveau configs used in 3.5.1, in 3.5.0 we made no changes to configs.

Expected Behaviour

There should be no regression in Nouveau indexing speeds between 3.5.0 and 3.5.1

Steps to Reproduce

1. create a 3.5.0 Nouveau database with 1million docs
2. rebuild index, note time
3. upgrade to 3.5.1
4. rebuild index, note time increase

Your Environment

Docker with CouchDB 3.5.1

Additional Context

@dianabarsan discovered this when testing CouchDB 3.5.1 for the CHT

[rnewson]

Working on a fix here main...nouveau-streaming-index-update