Skip to content

Tokenisation v0.1.2

Latest
Compare
Choose a tag to compare
@Commander-lol Commander-lol released this 12 Apr 04:55
· 3 commits to master since this release

Version 0.1.2 now supports the second parameter of the use command, allowing an input file or directory to be supplied. The module will be loaded and all of the input files will now be tokenised as per the combined module rules (loaded the same as the 0.1.1 release).

A note on lexical rules; all rules will be prefaced with a ^ character and are matched against the start of the sub section of source code that either ended with the previous token or that is after the most recent section of whitespace if the option to ignore it has been enabled. Rules that already contain ^ will not have a second one added.