Documentation Needed. How to use lexer?

Hello there, are there any kind of documentation on how to use the lexer? For example, what is the expected input and output, token structure, and what types of input are supported, requirements etc.
