Deforest All the Things

[countvajhula]

Summary of Changes

Further deforestation of list operations within Qi (integration branch).

This introduces a qi/list collection that includes functional list operations resembling the ones in racket/list, but which exhibit Qi syntax even in higher order positions, and are deforested (i.e. they avoid building intermediate collections on the way to constructing the result).

There is already basic deforestation on the main branch as of Qi 4. This PR improves the architecture of deforestation so that it is:

• … more general, covering a large number of common list operations (taking racket/list as a representative goal)
• … extensible. Once each interface in racket/list is deforested, that will give us an idea of what the core operations really are which can express all of the others, and what a good interface to these core operations would look like that could be exposed to users for deforesting custom list operations.
• … more sound. All deforestation will be done within the Qi core language, and we will discontinue optimizing host language expressions to preserve a clear separation of languages and invariants

As we continue on this work, it will hopefully also give us some idea of whether and how deforestation could be extended to datatypes more general than lists (for instance, vectors and even Qi's core values).

Remaining Work

• update the developer documentation [Sid/Dominik]
• write release notes [Sid]
• make a release announcement (notify people in advance) [Sid/Stephen]
• final review [all]
• Follow the release steps [all - during the meeting on Jan 17]

WIP

• document Qi's theory of effects (needs to be updated) (Theory of effects #165 ) [Sid/all]
• values-oriented zip (nice-to-have) (zip core form #190 ) [Sid/Ben]

Won't Do

• we have take -- should we also have drop? [this is not trivial]
• benchmarks for the new qi/list additions [benchmarks take time to run, and these would not show anything the existing benchmarks don't already show]
  • list-ref
  • length
  • empty?
  • filter-map
• restore test coverage [created Investigate shortfall in test coverage #189 to investigate later]
• better error reporting in report-syntax-error (datum->syntax in report-syntax-error should pass a srcloc for better error reporting #172 ) [not trivial]

Done

(wasn't tracked here from the beginning)

• benchmark for take, taking fractions of the input size instead of a constant number [Dominik]
• Test Frosthaven Manager and other codebases using Qi with the compiler branch
• remove unused dependencies (Unused dependencies #174 )

See commits and meeting notes since Jan 2024.

After release:

• schedule a round of DIA to update attributions on the repo -- it's quite out of date!

Public Domain Dedication

• In contributing, we relinquish any copyright claims on our contributions and freely release them into the public domain in the simple hope that they will provide value.

(Why: The freely released, copyright-free work in this repository represents an investment in a better way of doing things called attribution-based economics. Attribution-based economics is based on the simple idea that we gain more by giving more, not by holding on to things that, truly, we could only create because we, in our turn, received from others. As it turns out, an economic system based on attribution -- where those who give more are more empowered -- is significantly more efficient than capitalism while also being stable and fair (unlike capitalism, on both counts), giving it transformative power to elevate the human condition and address the problems that face us today along with a host of others that have been intractable since the beginning. You can help make this a reality by releasing your work in the same way -- freely into the public domain in the simple hope of providing value. Learn more about attribution-based economics at drym.org, tell your friends, do your part.)

[countvajhula]

FYI: I'm rebasing this branch to main to get the syntax-spec-v2 changes.

[countvajhula]

Hey all,
I've edited the PR description to include outstanding items I see and any work in progress as well as nice-to-have's (please add any I've missed! edit: Are there any open issues we could tackle as part of this release?).

@spdegabrielle suggested having a release party on Jan 17 to make an event out of it and give people time to plan to attend. We could discuss things like future plans and projects (continuations? generalized deforestation? category theory? experimenting with different backends? language experiments???), the possible (super fun!) optimization-themed community event (in summer '25?), and anything else. That would also give us more time to do some of these "polish" items (i.e. mostly benchmarks and charts!) and add the finishing touches to the amazing work we've done over the last year. What do we think about this possibility?

If that sounds good to you, let me know if you'd like to be there but Jan 17 does not work for you, and we can pick another day.

If we're comfortable moving forward with the merge without the above items, we could still do that tomorrow and then continue on the main branch as we prepare for the release event. Lmk, or, if you'll be joining tomorrow, then talk to you then :)

[countvajhula]

Dominik and I addressed a few outstanding issues today and we also reviewed others and decided they are nontrivial - best to do after the release. I've updated the PR description and created/commented on relevant issues.

And sounds like we're on for the release party on Jan 17! I will send out an email soon.

[benknoble]

Excellent! For the todos, anything that gets deferred will hopefully still get tracked ;)

[benknoble]

RE: dropping symbolic aliases in 3d661fb (add back symbolic aliases that got dropped at some point, 2025-01-15), I think they disappeared in c609625 (extract more non-core forms as macros, 2022-08-25), which is contained in v4.0!

[benknoble]

At least, git log -S <alias> upstream/deforest-all-the-things finds that commit for both AND and OR; for NOT, I haven't managed to find the removal yet.

[countvajhula]

Whoops, guess they've been gone quite a while 😬 Actually, they might have slipped out as early as the addition of Syntax Spec (was that 3.0?), as it's quite possible we just neglected to add the aliases to the provide form in the expander at the time. Prior to that, just having the datum matching of the symbolic variants in the original flow macro was sufficient for them to be part of the "core"/flat language. While the rules of that original macro carried over to qi0->racket, the ground truth for the core language switched to being the syntax spec in the expander, and it just might have slipped through the cracks then.

[benknoble]

That explains why the "pair with MB" commit came up a few times in my search. I definitely wasted too much time looking though: I probably should have tried git bisect.