[NRY] Implements Unicode (UCS-2) strings into VM

[tcr]

Not ready yet.

[natevw]

I can't build this branch. I get:

Undefined symbols for architecture x86_64:
  "_tm_ucs2_str_charat", referenced from:
      _l_tm_ucs2_str_charat in liblibcolony.a(libcolony.lua_tm.o)
  "_tm_ucs2_str_length", referenced from:
      _l_tm_ucs2_str_length in liblibcolony.a(libcolony.lua_tm.o)
  "_tm_utf8_char_encode", referenced from:
      _l_tm_utf8_char_encode in liblibcolony.a(libcolony.lua_tm.o)
  "_tm_utf8_str_tolower", referenced from:
      _l_tm_utf8_str_tolower in liblibcolony.a(libcolony.lua_tm.o)
  "_tm_utf8_str_toupper", referenced from:
      _l_tm_utf8_str_toupper in liblibcolony.a(libcolony.lua_tm.o)
ld: symbol(s) not found for architecture x86_64

Any ideas? Both the l_tm and the tm definitions are in the same file, but somehow the latter are getting optimized out?

[natevw]

Build fixed by natevw@9189dc6

[natevw]

I think I have a plan. It is horrible and I like it.

Paragraph number background

Currently, colony stores strings as UTF-8, because Lua. We'd like to keep this, because Lua and default node.js encoding and memory usage. Memory usage is important. (So is Lua AFAICT.)

But JavaScript strings are pretty much UTF-16 arrays, and all our APIs have to act like that. This sucks, because unless we store a UTF-16 copy (which we can't even because all Lua strings share the same metatable) we ± have to convert from UTF-8 to UTF-16 every time any variety of substring/charCode is accessed. Which particularly sucks if the user is for (var i = 0; i < str.length; ++i) something(str[i]);ing in their code.

Paragraph number plan

We need an efficient (speed and storage) way to convert between UTF-16 indexes and UTF-8 indexes. Storing a size_t* buffer the length of the UTF-8 string is right out, it's more memory and missing a metatable to hang it on per-string anyway [maybe there's tricks to weak map instances to values but then it's still] more memory.

Here is my horrible idea:

1. colony-compiler munges strings during compilation
2. …it munges them to make sure they have CESU-80 split surrogate pairs
3. …it munges them to make sure the sequences are always either 1 or 3 bytes [depends on previous, required by next]
4. …it munges them to prefix a list of where all the multibyte sequences are (these offsets are themselves UTF-8 encoded ;-)

With that, we have to be careful how we use the raw Lua strings, but now e.g. str_mt.__index can do something like this:

function (str, idx16)
   -- *handwavy part* read prefix of str and convert idx to idx8+mbflag
   return (mbflag) ? str.sub(idx8, 3) : str.sub(idx8, 1)
end

…which looks suboptimal (and also Lua doesn't ternary so who does this guy think he's trying to fool) but I think will actually be waaaay more optimal than the current code, which involves all sorts of conversions which are all done by hops in/out of C and also require reading the whole string.

In the presumably common case of ASCII input, the prefix is just a single leading '\0' terminator and idx8 is just 1 (or 2? because Lua?)

Other considerations: a proper UTF-8 encoding must use the shortest representation (i.e. sometimes 2 bytes). If this matters (CESU isn't proper UTF-8 anyway, but still) so when making a Buffer from a string we might want to fix. But we already know where all the 3 bytes are. (This 3 byte thing is sort of the key to everything: otherwise our prefix would need to encode both the position and the length of multibyte sequences and who wants to do that? We don't have to, because we can pad 2-byte values to 3, and assume we won't get longer sequences because we're actually representing UTF-16 with our UTF-8…)

Paragraph number what next

If you've made it this far, and aren't scared, and don't tell me not too, I should get on with actually trying to fix colony UCS-2 support via this strategy. It will require a (hopefully smallish) change to colony-compiler, changing the string and regex logic to respect+leverage the prefix, making sure Buffer to/froms work too.)

[natevw]

Based on various accomplishments like such as in "Discussing with @tcr" and also "The Sleeping On It" here's an update on the horrible idea above:

• It is horrible. It would likely have a ton of implications that would need to be dealt with, e.g. how to ensure obj.thing is the same as obj['thing'] and whatnot.
• There's probably a way to track per-string metadata somewhere. There's certainly a will.

I'm still enamored with the general concept of tracking the offsets (and probably lengths, sans compiler munging) of only the multibyte sequence, or at least a tight/optimized-for-ASCII (table,index,lookup) thing is the way to go.

As a first pass, my plan now is to refactor the code to use a "helper function" for indexing — at first this helper function will be an inefficient brute force ordeal, then [assuming the approach proves useful] I hope to figure out how to do some sort of per-string-hash weakmap metadata association to make it fast.

[natevw]

(for later: http://www.lua.org/pil/17.html)

[raffecat]

My admittedly unrefined thoughts on the matter came down to this: most strings in js tend to be used in an opaque manner: as field names, concatenated, read or written as utf-8; in all these cases, using utf-8 internally is fast, compact, and avoids conversions - especially for web content.

There are only a few places that must behave as if the encoding is UTF-16:

• length property: this can walk the string on first access and cache UTF-16 length (including would-be surrogate pairs) in the string header, i.e. modify the VM to add a cached-length field.
• charAt, charCodeAt, slice, substring: cache the last offset used in the string header, and search from there next time (inline caching) - makes most typical loops efficient enough, even backwards iteration; utf-8 is self-synchronizing.
• search methods: the storage encoding doesn't have a great impact here, except for the start/end positions, which can make use of the cached offset.

Originally I thought of switching between 8-bit and UTF-16 internally when the string is created (Lua strings are immutable, and are actually pre-hashed at creation time for table lookups) - i.e. if the raw data contains any code points > 255, internally use an array of 16-bit code points instead of 8-bit data. This would impact every function that touches Lua strings, so I expect it would be a lot more work and would impact the LuaJIT runtime as well.

[natevw]

Made some good progress on this today. My current branch isn't much more "fixed" yet, but the .length and __index functions are now using an IMO better factored approach that should fingers crossed be nicely applicable everywhere else too.

Interesting things I've discovered along the way:

• something in colony-compiler is already sanitizing UTF-8 (didn't look what) — this is good and bad
• utf8proc is very strict about UTF-8 correctness, so we can't use it with CESU-8 or other non-sanitized forms

Neither of these is a huge showstopper but may affect the final approach to handling the "surrogate pairs" stuff. Before I worry about that though, I plan to do a first pass at everything assuming that we'll get CESU-8 right as the dust settles.

[natevw]

Closing this PR, going to start make a new one with clearer notes about what it accomplishes.