Expand functionality to support tokenization

[libsteve]

You should be able to specify state transitions with the following data:

• the state to transition from
• either:
  • a string to transition over
  • an automata to transition over
• either:
  • the state to transition to
  • a function that takes the matched string transitioned over and returns:
    • what state to transition to
    • a 'token' value that can hold the data

Using this method, evaluating the automata may be able to return a list of the 'token' values generated during execution.

possible signature of adding a transition:

addTransition(
    start_state, 
    [ transition_string | transition_automata ], 
    [ token_data (*)(matched_string), ]
    destination_state);

possible signature of executing an automata:

struct { list_of_tokens, did_pass, passing_length } execute(automata);

The goal of these modifications is that the framework could in theory be used as a lexer.