Skip to content

Conversation

@amascolo
Copy link
Contributor

@amascolo amascolo commented Oct 27, 2024

Support for sorting and hybrid queries (i.e. using SortedDict and Range). Includes all queries ported over to SDQL.

@amascolo amascolo self-assigned this Oct 27, 2024
@amascolo
Copy link
Contributor Author

Details that came up during implementation:

  1. added a Timer expression in SDQL to mark where the timer should start – since we need to exclude the initial data sorting time in benchmarks (nice aside is that now we don't need to treat load expressions as a special case)

  2. added external function SortedIndices - to sort the initial data

  3. added external function SortedVec - to sort the tries

  4. special case to call emplace_back - when constructing @vec { <...> -> 1 } we interpret the relational form <...> -> 1 as i -> <...> and append elements to a std::vec

  5. special case to call at - since in SortedDict we can't use [ ] which has special semantics for insertion (this wasn't an issue previously for vecdict since after construction we only iterated on it – i.e. set semantics – whereas here we access elements)

  6. cache the last found element – to avoid calling SortedDict::find twice in situations where we check with contains then retrieve the element using at (in handwritten C++ code we could assign the result of find – i.e. an iterator - to some variable, but this isn't representable in SDQL)

@amascolo
Copy link
Contributor Author

Have ported query FJ 3a to SDQL and checked its performance is identical to handwritten C++.

Codegen and runtime should work everywhere, just need to generate all the other queries in SDQL.

@amascolo
Copy link
Contributor Author

@amirsh this PR is now ready to merge, subject to your approval:

  • Supports all sort-based and hybrid JOB queries in SDQL ✅
  • Performance within 2% of C++ (no regression on hash-based queries)

@amascolo amascolo requested a review from amirsh November 28, 2024 16:19
@amascolo amascolo marked this pull request as draft November 29, 2024 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants