Fixed data race in all_type_info in free-threading mode

[vfdev-5]

Description

• fixed data race in all_type_info in free-threading mode
• added test

For example, we have 2 threads entering all_type_info. Both enter all_type_info_get_cache`` function and there is a first one which inserts a tuple (type, empty_vector) to the map and second is waiting. Inserting thread gets the (iter_to_key, True) and non-inserting thread after waiting gets (iter_to_key, False). Inserting thread than will add a weakref and will then call into all_type_info_populate. However, non-inserting thread is not entering if (ins.second) {clause and returnsins.first->second;`` which is just empty_vector. Finally, non-inserting thread is failing the check in allocate_layout:

if (n_types == 0) {
    pybind11_fail(
        "instance allocation failed: new instance has no pybind11-registered base types");
}

On master running this test gives:

terminate called after throwing an instance of 'std::runtime_error'
  what():  instance allocation failed: new instance has no pybind11-registered base types
Fatal Python error: Aborted

Suggested changelog entry:

A free-threading data race in ``all_type_info()`` was fixed.

cc @colesbury

[colesbury]

I think we may need a further rework of all_type_info(). It's used by get_type_info() and some of the callers to that function modify the returned detail::type_info*.

I'm not entirely sure what the locking strategy should be here. It's possible that all the callers to get_type_info() and all_type_info() should hold the internals lock.

[colesbury]

On further thought, I think this approach makes sense. I don't think we need to rework the locking strategy and can address the other data races on type_info fields separately.

I'll write up a summary of the issues.

[vfdev-5]

@rwgk can you please review this PR?

[rwgk]

Will try tomorrow.

@cryos could you please also take a look?

[rwgk]

@colesbury could you please formally approve?

Looks good to me, but I'm not a free-threading expert. I'll merge when I see approvals from @colesbury and @cryos.

[rwgk]

Looks good to me, but waiting for approvals from @colesbury and @cryos.

[colesbury]

Yes, this LGTM.

I'll try to put up a follow up PR later this week for the remaining issues described in #5421.

[cryos]

This looks good to me.