Consider adopting lookup4

[lemire]

The simdjson library has a new UTF8 validator called lookup 4 which is simpler and faster than most alternatives.

See

https://github.com/simdjson/simdjson/blob/master/src/generic/stage1/utf8_lookup4_algorithm.h

It is really not a lot of code!!!

[milkey-mouse]

first fastvalidate-utf-8, then lookup2, 3, and 4... you're writing new validators faster than I can port them!

[lemire]

I think that lookup4 is going to be hard to beat. It is really down to the metal. Have a look.

[milkey-mouse]

I just benchmarked lookup4 (as implemented in simdjson) vs. this implementation (vs. the standard library's). lookup4 comes out on top in all but a couple cases. It's also much more consistent (not relying on the branch predictor as much)

[lemire]

+1

[pickfire]

This is done in https://github.com/rusticstuff/simdutf8