Experiments with the LIL lib

[quartzjer]

This was started by @matthijskooijman and is an ongoing experiment :)

LIL: The Little Interpreted Language

[quartzjer]

When combined w/ our bootstrap:

Sketch uses 180,826 bytes (71%) of program storage space. Maximum is 253,952 bytes.
Global variables use 18,181 bytes of dynamic memory.

The two serial readers don't play nice together though

[matthijskooijman]

I fixed up the example a bit more - it now disables the bitlash shell and supports arrow-up history recall :-)

[matthijskooijman]

Some observations and wild ideas regarding LIL:

• LIL currently uses a hashtable for variable lookups (so one for global and one for each function scope). In practice, most scopes will be small so allocating 1Kbyte of memory for a hashtable (256 buckets x 4 bytes) is a lot of overkill (even on regular architectures, I think). We should replace this with a simple flat list, possibly sorted or indexed by name lenght or something like that. For more advanced cases we could perhaps upgrade to a hashtable when the environment becomes bigger (and perhaps remember we upgraded the env for a function so we can start out bigger on the next invocation), but initially just a flat table should do fine.

• In LIL, everything is a string. This means that when storing stuff, the interpreter is continuously converting from and to strings. It can probably be changed to only convert when actually needed, to prevent int -> str -> int or list -> str -> list conversions.

• We might want to consider to actually introduce types into the language. I'm not quite sure about this yet, but it might be useful to have some basic typechecking, or if code could behave differently based on the type of arguments.

  • For example, now everything is a string, but also a command. You can of course explicitly eval a string to interpret it as code eval $foo, but there is also a lot of implicit eval'ing happening (e.g. [foreach v $list $code] evals the code passed in. It's a pro that it's so easy to switch between code and strings (think of code passed in through the mesh), but it's also a con that it's easy to accidentally use data as code. If you would add proper typing to the language, you could e.g say that string literals using { } become "code strings", add an appropriate string -> code string conversion function and refuse to evaluate normal strings.

  • Making a distinction between lists and strings might also be useful. Currently, every string can be interpreted as a list, which applies word-splitting on it. Applying proper quoting can make this unambigous, if you know you'll be passed a list or a string. You'll have to have conventions on what arguments / values are lists and which are strings. Allowing both becomes impossible. For example, consider you have a function that concatenates strings. You might want to call that using a single list, or multiple arguments:

    concat a b c
    set list = a b c
    concat $list
    

  Which is currently impossible.

  • Another useful type distinction would be "functions", see the next point.
  • As for the type distinction between strings and numbers, I'm not so sure that that is actually useful to apply (it's useful internally to prevent unneeded conversions, but I think that requiring explicit conversion in the script language itself just clutters things).
  • In general, typing allows some typechecking. Most notably, it could make sense to allow annotating function arguments with expected types, so the parser can automatically check and show errors (without having to re-implement this every time).

• LIL supports anonymous functions, but in the background these actually generate normal functoins with a randomized name, returning the name. There doesn't seem to be any garbage collection for these anonymous functions, so if you use them in a loop, or inside a function that is regularly called, you'll end up with a lot of unused "anonymous" functions.

  The reason for this is probably the (only) way to call a function is to evaluate a command, where the first word is the name of the function. It would seem easy to allow a function definition as this first word and call that, but you can't reliably tell the difference between a function name and definition without additional constraints. An obvious solution would be to use some special character at the start of a string to indicate a function definition / lambda, and forbid that character in (or at the start of) normal function names. Alternatively, if you add a typing system, you could have func return values of an "anonymous function" type, and use that to distinguish. A final option would be to drop the special casing of functions alltogether - a function is just a variable with a specific content. E.g., these would be identical: $func add {a b} {$expr $a + $b} and $set add {{a b} {$expr $a + $b}}. Calling a function is then different, it refers to a variable instead: $add 1 2 (which would expand to {{a b} {$expr $a $b}} 1 2 which is reduced by the parser to {$expr 1 + 2} etc. Essentially this is basic lambda calculus / lambda reduction. I don't really think I like using the $ everywhere though, and merging the variable and function namespaces seems elegant (and works well in languages like python), but I'm not sure if it's a great idea here.

• LIL supports varargs by defining a function with a single argument called "args". E.g. you can do func num {args} {print "I got " [count $args] "arguments"}. It might be useful to also support some fixed arguments and some varargs, like func addall {x args} { ...something appropriate here...}.

• In various spots we've seen the use for a callback pattern when something asynchronous is supposed to happen. Somewhat inspired by icedcoffeescript, which I stumbled upon yesterday, I'm thinking we could perhaps support asynchronicity properly in the interpreter. Essentially, we would allow a function that needs to wait for something, all the way at the bottom of the call stack, to wrap up the interpreter state, unwind the normal C call stack and then the main loop can continue. Once the thing that's waited on has happened, it can continue evalutating the rest of the LIL code.

  I have some ideas on how to implement this (essentially every time you eval a piece of LIL code, you also pass on a continuation (function pointer + data) to be called later when execution blocks. When execution actually blocks, we either jump up directly (using a longjmp), or return a special value that indicates to all intermediate functions that a block has happened.

  One of the cool things about this is that we could actually implement a semaphore, that would allow doing something like:

  wait_for foo
  print "Foo happened!"
  

  And later in some event handler, do:

  done foo
  

  To let the original code continue executing.

  Not sure if this is really feasible, but I think it could be made to work, and it would be awesome. It would also complicate C code that wants to evaluate LIL script, but I guess that's mostly our code, not user code (code that just does something and doesn't block, nor eval LIL script would be ok). OTOH, we propagate the use of Shell.eval a lot, which would be complicated as well. I should probably write some test code for this :-)