We received some great contributions from @dirk that improved error handling (#5) and refined how we build our N-API bindings (#7, #9). He also clarified our build process in the documentation and added an explicit Electron dependency now that the new beta supports N-API (#10). Thanks @dirk!
Our plan is to dedicate 12 full weeks to Xray and see how far we can get with the implementation. We originally planned to start this trial period 2 weeks ago, but decided to defer it in order to spend more time doing planning around our vision for real time collaboration. So last week will count as week 1 of 12. This week is week 2 of 12.
We're currently rendering text with a fairly naive strategy, where we just transform code points to glyphs and position them one after another with WebGL. The great thing about this strategy is it's really fast. It takes me ~1.2 ms to render a full screen's worth of text on a late 2016 MacBook Pro. The downside of this strategy is that we don't perform correct text shaping.
Last week, we explored running all of our lines through HarfBuzz compiled as a separate WebAssembly module, but in our tests, running HarfBuzz on 50 lines of 100 characters each was taking between 4.5ms and 20ms, depending on the font. Since our target for a frame is 8ms, this makes us pretty reluctant to pursue this path further. We're a code editor, not a word processor, so it's not clear that we need all the features that a full-on text shaping engine provides.
If we don't do some sort of text shaping (and HarfBuzz seems like the only game in town), here's what we'll be missing:
- No ligatures support: Text shapers combine code points with tables embedded in the font to decide when to render ligatures. We're a code editor, so this isn't a deal-breaker, but fonts like Fira Code rely on ligatures to render special characters for common programming sequences such as
<=. - No kerning support: For fixed width fonts, we weren't able to observe any noticeable difference for a lack of kerning. For variable-width fonts like Helvetica, rendering without kerning looks a bit odd. Again, we're a code editor, so not a deal-breaker.
- No support for bi-directional text. This isn't a deal-breaker in the short term, since all of the dominant programming languages are based on latin scripts. Again, it's not our ambition to become a general word processor. Long term, however, we need to support right-to-left text appearing in strings and comments in order to be usable by developers working with languages like Arabic and Hebrew. Interestingly, Sublime Text does not appear to support bidirectional text, but we'd like to do better.
- No support for context-sensitive substitutions. In Arabic and Indic scripts, the same characters can render different glyphs depending on their context. Sublime also does not support this.
Based on what we have learned and the above limitations, this our plan for text shaping going forward. In the near-term:
- Don't support full text shaping in the general case. We want to emphasize maximal speed for the common case, which shouldn't require full text shaping. If running text shaping on every line took less than 1ms, it would be worth it, but we'd prefer not to pay what it appears to cost.
At some point in the future, make the following enhancements:
- Add bi-directional text support. We've run into trouble building a library that combines both Rust and C/C++ in a single WebAssembly module, so the ideal path would be to find or write an implementation of the Unicode bi-directional text algorithm in Rust and embed it in
xray_core. One important detail is that we need to preserve the correspondence between column positions in the source and rendered text in order to render cursors and interpret mouse interactions, so just transforming the text alone will be insufficient. - Use presentational characters to render Arabic as described in this blog post, again porting an existing implementation of this transformation to Rust and incorporating it into
xray_core. Again, we'll need to maintain a mapping of how characters in the input and the output map for cursor positioning. Several of the existing implementations of this transformation are GPL-licensed, so we'll need to be careful to avoid deriving our work from one of them. - Add limited ligatures support at some point in the future to
xray_core. This would involve loading the font and consulting the lookup table for ligatures. The goal would be support for fonts like Fira Code, and the hope is that we will be able to efficiently perform just this subset of the generalized text shaping workload within our budget of 1ms. - Render sequences of Indic characters as atomic units via canvas rather than trying to render and composite individual glyphs like we do for other scripts. This would rely on the text-shaping built into the browser to render words in these scripts. We will pay a performance cost, but since we're anticipating these characters to appear rarely as part of comments and strings, it should be acceptable and better than adding the performance and complexity of full shaping for cases where it isn't needed.
Producing a lightning fast editor that runs on the web is going to involve trade-offs, and we'll need to make some tough decisions. Avoiding full text shaping is one of them. It would be great to be fast and perfectly correct in all cases, but we're not willing to sacrifice speed in the common case for perfect correctness at the corners.
We're going to post some help-wanted issues to see if anyone is interested in helping out with some of the compromise solutions in the above plan.
So in conclusion, we didn't end up merging any code related to text shaping, but we did learn a lot and came up with a clear plan for how to proceed.
The bulk of the week was spent adding support for selections to the editor. The first step was an introduction of a new abstraction called anchors. Anchors serve a similar role to markers in Atom today, but they have a much cleaner implementation due to the buffer being a CRDT.
An anchor is a value that tracks a logical position in a buffer. You create an anchor by calling one of the following methods on the buffer:
anchor_before_offsetanchor_after_offsetanchor_before_pointanchor_after_point
These return an opaque Anchor value, which can be converted back to a concrete offset or point in the future via the following methods:
offset_for_anchorpoint_for_anchor
Internally, an anchor is an enum that either represents either the Start or End of the file or some point in the Middle of the file via an insertion_id, offset, and bias. If you create an anchor at offset 10, its position will be updated by any edits that occur prior to offset 10, so that it always tracks the same logical position in the text. So if you create this anchor before offset 10, it will have a left bias and remain at that offset even if there is a subsequent insertion at its exact location. If the anchor is created after offset 10, it will have a right bias and be pushed rightward by insertions at its location.
Selections are built on top of anchors. Each anchor maintains a vector of selections ordered by their start anchor, maintaining the invariant that the selection ranges are always disjoint. We use anchors for selections rather than absolute positions so that the logical intention of the user is maintained even in the face of edits to the buffer by other users or by packages. We implemented basic cursor movements and selection expansions (up, down, left, and right) as well as methods to add a selection above and below the current.
We plan to render selections and cursors as additional WebGL shader passes that draw solid rectangles. We have the plumbing mostly in place to do this, but haven't finished actually populating the buffers on the GPU to tell the shaders where to draw. We're hoping to have that finished early this week, so we can move on to handling the input to actually move the selections and cursors around. That will raise the question of how we handle key bindings and commands in Xray, which could take some time to iron out.
Once we can render and manipulate selections, we'll move on to handling keystrokes to perform actual edits to the buffer. The splice method already exists to enable edits, so it should just be a matter of calling it in a loop in reverse order of the selection ranges. Once we add some caching related to translating anchors to positions, we can measure our performance and see how many cursors we can type with within our 8ms target window. Hopefully we do well.
We'll be a bit short-handed this week due to @as-cii being on reactive duty for Atom and @nathansobo heading to Denver on Wednesday to give a talk at Pivotal Labs. We hope to finish selection rendering and ideally also get an initial solution in place for key bindings and commands to move those selections around. If things go really well, we'll start on editing.