Skip to content

Commit

Permalink
Move todos to HTML for visibility; add gif
Browse files Browse the repository at this point in the history
  • Loading branch information
@jonathonherbert
jonathonherbert committed Jul 1, 2024
1 parent 15ea1d4 commit d15fe7e
Show file tree
Hide file tree
Showing 4 changed files with 210 additions and 72 deletions.
99 changes: 30 additions & 69 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

An experiment to see if a query DSL, coupled with first-class typeahead and syntax highlighting, might provide a more consistent and discoverable way to search Guardian content.

Sandbox available at https://cql-sandbox.gutools.co.uk/. A language server is currently available via API Gateway in the developer playground account – add the url to get started, or ping me and I'll send it!

## Why?

At the moment at the Guardian, there are a few ways to query CAPI:
- directly, via the API and a query string
- via many different GUIs across many different tools, each with variable support for the search functionality CAPI has to offer.
Expand Down Expand Up @@ -31,50 +35,7 @@ Concerns:
- where is the language server located? Do we embed it in the input, or make it a feature of CAPI?
- Typeahead feature might include section/tag lookup, interactions w/ API etc. – keeping this server side could reduce pace at which client would change, which if this component spreads across estate as a library would be helpful.

Todo:

- [x] Scanning
- [x] Parsing
- [x] Query string builder
- [x] Add group and binary syntax
- [x] String ranges in scanned tokens
- [x] Parse hints for typeahead
- [x] ~~ScalaJS to provide parser in web env~~ ScalaJS adds 180kb to your bundle as the price of entry? Yeah we're not doing that
- [x] Add a language server for funsies
- [x] Web component - environment and first pass at component infra
- [x] Web component - syntax highlighting
- [x] Ensure untokenised string components still display, we're getting 500s and invisible characters on trailing + chars
- [x] Web component - typeahead
- [x] First pass at implementation
- [x] Handle typing on the trailing edge (off by one)
- [ ] Web component - async lookup
- [x] Implement async lookup in language server
- [ ] Add loading state
- [x] Bug: open parentheses crashes the server 🙃
- [x] Fix tests
- [x] Fix input scrolling
- [x] Bug: fix crash on adding query meta within parentheses or after binary operators (with useful error state)
- [x] Add '@' syntax for content return format (e.g. show-fields)
- [x] Fix issue with incomplete binaries and output modifiers
- [ ] Fill out additional fields:
- [x] Dates!
- [x] Add `type` property to suggestion envelope to ensure correct interface is displayed (NB: Circe wraps sealed trait in object with single key as name of class)
- [x] Correct focus when date appears
- [x] What do we do when users want to navigate through the string without focus being stolen? Perhaps we don't need to autofocus? Solution for now: autofocus on first input, keydown or tab to focus input when value is already present (to allow user to scrub through dates unimpeded)
- [ ] Parse to correct format for query
- [ ] Other, less fancy fields
- [x] Fix crash on empty parens
- [x] Fix crash on leading colon when key is not valid query or output key
- [x] Move to contenteditable (issues with scrolling in Chrome, e.g. https://issues.chromium.org/issues/41081857, make syncing scroll state of overlay difficult)
- [ ] Add typeahead for binaries
- [ ] Ensure content is displayed when server does not respond
- [ ] Error handling for 4/5xx

Infra
- [x] Configure CI for lambda
- [x] Add handler for lambda
- [x] Add CI for static site
- [x] Add configuration for CAPI key
See index.html in the prosemirror client for todos.

### Notes

Expand Down Expand Up @@ -212,28 +173,28 @@ That's a lot of things. Unavoidable things to know:

Perhaps tokens are all we need: the parser can just understand those sorts of tokens that relate to chip keys and values by name.

## Prosemirror client todos

- [x] Handle keyboard navigation on typeahead menu
- [x] Handle selection on typeahead menu (including clicks)
- [x] Correctly display typeahead menu when there's no content in a chip key or value
- [x] Serialise to CQL string on copy
- [x] Deletion pattern for chips on backspace
- [x] Deletion handle for chips UI
- [ ] Add good testing story
- [ ] Date suggestions
- [ ] Tab through content
- [ ] Display labels, not values, in chips?
- [ ] Chip polarity
- [ ] Cursor movement:
- [ ] Ctrl-e to move to start and end of doc
- [ ] ...
- [ ] Telemetry:
- [ ] Create chip
- [ ] Autocomplete key, value selection
- [ ] Remove chip – mouse
- [ ] Remove chip – backspace
- [ ] Tab behaviour?
- [ ] Paste?
- [ ] ...
- [ ] ...
## Async vs. sync – why not both? 🤷

Async vs. sync language server trades off a few things:

### Async

- ✅ Update once, available everywhere
- ❌ Latency can be a problem for UX – worse in US/AU. How to limit this feeling?
- ❌ Maintenance cost, ownership cost – some team must own and maintain this service, with gladness in their heart

### Sync

- ✅ Instant UI for everything but resolver lookups
- ✅ Smaller surface area for sync bugs
- ❌ Must bundle into code, increasing weight of client (likely negligible) and possibly necessitating publishing (more dev friction)

Can we have both?
- Async better for widely used product (e.g. CAPI), where cost of updates across estate increases – async mitigates that cost vs. ownership/maintenance story
- Sync better for smaller products, where search is e.g. only used in that product, and benefit of central server negligible

Perhaps an API that accepts a language service that is optionally asynchronous?

This will necessitate a rethink in how we handle suggestions, as at the moment they're included in the LS response. Suggestions are necessarily async (value lookups may go across the wire).
- Factor them out into a separate, async call. This would mean a rethink of the suggestions API.
- Make suggestions event-driven, to ensure we make a single call.
179 changes: 178 additions & 1 deletion prosemirror-client/index.html
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<!doctype html>
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
Expand All @@ -9,5 +9,182 @@
<body>
<div id="app"></div>
<script type="module" src="/src/main.ts"></script>
<h2 id="todos">Todos</h2>
<ul>
<li><input checked type="checkbox" /> Scanning</li>
<li><input checked type="checkbox" /> Parsing</li>
<li><input checked type="checkbox" /> Query string builder</li>
<li><input checked type="checkbox" /> Add group and binary syntax</li>
<li><input checked type="checkbox" /> String ranges in scanned tokens</li>
<li><input checked type="checkbox" /> Parse hints for typeahead</li>
<li>
<input checked type="checkbox" />
<del>ScalaJS to provide parser in web env</del> ScalaJS adds 180kb to
your bundle as the price of entry? Yeah we&#39;re not doing that
</li>
<li>
<input checked type="checkbox" /> Add a language server for funsies
</li>
<li>
<input checked type="checkbox" /> Web component - environment and first
pass at component infra
</li>
<li>
<input checked type="checkbox" /> Web component - syntax highlighting
<ul>
<li>
<input checked type="checkbox" /> Ensure untokenised string
components still display, we&#39;re getting 500s and invisible
characters on trailing + chars
</li>
</ul>
</li>
<li>
<input checked type="checkbox" /> Web component - typeahead
<ul>
<li>
<input checked type="checkbox" /> First pass at implementation
</li>
<li>
<input checked type="checkbox" /> Handle typing on the trailing edge
(off by one)
</li>
</ul>
</li>
<li>
<input type="checkbox" /> Web component - async lookup
<ul>
<li>
<input checked type="checkbox" /> Implement async lookup in language
server
</li>
<li><input type="checkbox" /> Add loading state</li>
</ul>
</li>
<li>
<input checked type="checkbox" /> Bug: open parentheses crashes the
server 🙃
</li>
<li><input checked type="checkbox" /> Fix tests</li>
<li><input checked type="checkbox" /> Fix input scrolling</li>
<li>
<input checked type="checkbox" /> Bug: fix crash on adding query meta
within parentheses or after binary operators (with useful error state)
</li>
<li>
<input checked type="checkbox" /> Add &#39;@&#39; syntax for content
return format (e.g. show-fields)
<ul>
<li>
<input checked type="checkbox" /> Fix issue with incomplete binaries
and output modifiers
</li>
</ul>
</li>
<li>
<input type="checkbox" /> Fill out additional fields:
<ul>
<li>
<input checked type="checkbox" /> Dates!
<ul>
<li>
<input checked type="checkbox" /> Add <code>type</code> property
to suggestion envelope to ensure correct interface is displayed
(NB: Circe wraps sealed trait in object with single key as name
of class)
</li>
<li>
<input checked type="checkbox" /> Correct focus when date
appears
</li>
<li>
<input checked type="checkbox" /> What do we do when users want
to navigate through the string without focus being stolen?
Perhaps we don&#39;t need to autofocus? Solution for now:
autofocus on first input, keydown or tab to focus input when
value is already present (to allow user to scrub through dates
unimpeded)
</li>
<li>
<input type="checkbox" /> Parse to correct format for query
</li>
</ul>
</li>
<li><input type="checkbox" /> Other, less fancy fields</li>
</ul>
</li>
<li><input checked type="checkbox" /> Fix crash on empty parens</li>
<li>
<input checked type="checkbox" /> Fix crash on leading colon when key is
not valid query or output key
</li>
<li>
<input checked type="checkbox" /> Move to contenteditable (ProseMirror)
(issues with scrolling in Chrome, e.g.
<a href="https://issues.chromium.org/issues/41081857"
>https://issues.chromium.org/issues/41081857</a
>, make syncing scroll state of overlay difficult)
</li>
<li>
<input checked type="checkbox" /> Handle keyboard navigation on
typeahead menu
</li>
<li>
<input checked type="checkbox" /> Handle selection on typeahead menu
(including clicks)
</li>
<li>
<input checked type="checkbox" /> Correctly display typeahead menu when
there&#39;s no content in a chip key or value
</li>
<li><input checked type="checkbox" /> Serialise to CQL string on copy</li>
<li>
<input checked type="checkbox" /> Deletion pattern for chips on
backspace
</li>
<li><input checked type="checkbox" /> Deletion handle for chips UI</li>
<li><input type="checkbox" /> Add good testing story</li>
<li><input type="checkbox" /> 🐛 Suggestions are off by one for consecutive chips</li>
<li><input type="checkbox" /> Date suggestions</li>
<li><input type="checkbox" /> Tab through content</li>
<li><input type="checkbox" /> Chip polarity</li>
<li><input type="checkbox" /> Add typeahead for binaries</li>
<li>
<input type="checkbox" /> Error handling, with position, when the query
is malformed
</li>
<li><input type="checkbox" /> Error handling for &gt;400 codes</li>
<li>
<input type="checkbox" /> Cursor movement:
<ul>
<li>
<input type="checkbox" /> Ctrl-e to move to start and end of doc
</li>
<li><input type="checkbox" /> ...</li>
</ul>
</li>
<li><input type="checkbox" /> Display labels, not values, in chips?</li>
<li>
<input type="checkbox" /> Telemetry:
<ul>
<li><input type="checkbox" /> Create chip</li>
<li><input type="checkbox" /> Autocomplete key, value selection</li>
<li><input type="checkbox" /> Remove chip – mouse</li>
<li><input type="checkbox" /> Remove chip – backspace</li>
<li><input type="checkbox" /> Tab behaviour?</li>
<li><input type="checkbox" /> Paste?</li>
<li><input type="checkbox" /> ...</li>
</ul>
</li>
<li><input type="checkbox" /> ...</li>
</ul>
<h3 id="infra">Infra</h3>
<ul>
<li><input checked type="checkbox" /> Configure CI for lambda</li>
<li><input checked type="checkbox" /> Add handler for lambda</li>
<li><input checked type="checkbox" /> Add CI for static site</li>
<li><input checked type="checkbox" /> Add configuration for CAPI key</li>
<li><input type="checkbox" /> ...</li>
</ul>
</body>
</html>
2 changes: 1 addition & 1 deletion prosemirror-client/src/cqlInput/CqlInput.spec.ts
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { test, expect, mock } from "bun:test";
import { test, mock } from "bun:test";
import { createCqlInput } from "./CqlInput";
import { CqlService } from "../CqlService";

Expand Down
2 changes: 1 addition & 1 deletion prosemirror-client/src/style.css
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
-moz-osx-font-smoothing: grayscale;
}

#app {
body {
max-width: 1280px;
margin: 0 auto;
padding: 2rem;
Expand Down

0 comments on commit d15fe7e

Please sign in to comment.