ZIO: Set minimum number of free issue threads to 32

[amotin]

Free issue threads might block waiting for synchronous DDT, BRT or GANG header reads. So unlike other taskqs using ZTI_SCALE to scale with number of CPUs, here we also need some amount of threads to potentially saturate pool reads. I am not sure we always want the 96 threads we had before ZTI_SCALE introduction at #11966 on small systems, but lets make it to at least 32.

While here, make free taskqs configurable, similar to read and write ones.

How Has This Been Tested?

With this patch on a FreeBSD system with 16 CPUs I see 30 (rounding effects) free issue threads instead of 12.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[behlendorf]

Looks very reasonable to me.

[amotin]

I've further modified spa_taskqs_init() to not create 32 taskqs of one thread each on a system with one CPU. The new code will create one taskq with 32 threads, that will be more balanced.

[robn]

Yeah, makes sense.

[mutters, shakes fist at sky]