DOCS: add add_prefix_space to processors.ByteLevel

[CloseChoice]

closes #1819

We already support add_prefix_space for processors.ByteLevel, it is just not documented since this class is defined via pyo3's pyclass, see here. See further down the signature for the new function, which mirrors python's __new__ method: https://github.com/huggingface/tokenizers/blob/main/bindings/python/src/processors.rs#L492.

[ArthurZucker]

LGTM but lets update to proper doc

[ArthurZucker]

the doc is wrong in the sense that this does not remove the space in the post processor nor does it add it. It just accounts for it in the offset computation if trim_offsets.

                    // If we are processing the first pair of offsets, with `add_prefix_space`,
                    // then we shouldn't remove anything we added. If there are more than one
                    // leading spaces though, it means we didn't add them, and they should be
                    // removed.

try to incorporate this please!

[CloseChoice]

Thanks for the review, I updated the docs. I somehow misunderstood the meaning of add_prefix_space for the postprocessor, but it simply refers to the preprocessor and has more of a technical meaning here. To actually understand the meaning I added some code to the tests and thought that might be helpful to keep. Let me know if you want further changes.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

just missing make style

[CloseChoice]

just missing make style

done

[ArthurZucker]

AssertionError: The content of py_src/tokenizers/processors/__init__.pyi seems outdated, please run `python stub.py`

😉

[CloseChoice]

That was weird, nothing happened when I ran python stub.py, just once I merged main it had an effect. So I hope I didn't miss something again, sorry for the inconvenience.

[ArthurZucker]

you need to cargo build or maturin dev then run python stub

[ArthurZucker]

Ty!