Documenting the happy path for implementation via Rails engines

[tylerwillingham]

From the README (bolding the relevant bit)

https://github.com/rubyatscale/packs-rails sets up Rails autoloading, as well as rspec and FactoryBot integration, for packages arranged in a flat list. packs-rails is quite convenient, but for autoloading we recommend to use Rails::Engines instead.

I liked this approach when I worked at Shopify so much so that I want to recreate it but where I worked our packages were well established and I never saw a new one introduced during my tenure. I think lightly documenting the "how" here to illuminate the "Why" that someone might ask after reading that line in the README would be valuable for myself and the rest of the community. I'm happy to own the documentation contribution but I'd love some input.

---

I'll kickstart this with some initial questions I have:

• Would a new package be initialized via bin/rails plugin new ...?
• By leveraging Rails engines are you implicitly signing yourself up for per-package dependency (gem) management?
• What do folks see as a tripwire for when you should reach for an engine/when you should reconsider an engine?

[tylerwillingham]

I'm still happy to contribute to this but I wanted to circle back after finding a conference talk on YouTube that helps make this more clear:

All you need is Rails (Engines): Compartmentalising your Monolith) by @pinzonjulian

Which also points to @rafaelfranca's helpful repo rafaelfranca/lego

[exterm]

A note - when I was working on this at Shopify, we didn't make all packages engines. We had lots of packages that just followed the existing file path structure, often because namespacing was already in place and we just dropped in a package.yml file.

We had a special kind of package that we called component; these would generally be pretty large, top level, and represent bounded contexts (from Domain Driven Design). Only these were structured like Rails applications and thus engines.

I do find the notion of nested engines as explored by @pinzonjulian in the linked talk interesting, however we didn't do that back then (might be different now, I am no longer at Shopify). IMO in the interest of simplicity it doesn't really make sense to force rails application structure for all packages, especially nested ones.

[tylerwillingham]

This makes sense, in my corner our monolith was all "components" and thus all engines so that's all I saw

[pinzonjulian]

Hey @tylerwillingham

I'm happy you found my talk! I've been meaning to create content around this topic but haven't gotten around to it but I'm all for helping get these ideas out there.

However, I'm not sure if Packwerk is the place to document this because it's a very Rails-specific thing. I'd like to hear Packwerk's maintainers take first before committing to writing about this.

As a counter argument, any place is better than no place at all. At least having the discussion is a good place to start.

Now, on the specifics on how to create an engine for this purpose:

Would a new package be initialized via bin/rails plugin new ...?

In my opinion, Rails' plugin generator is not the right tool for the job because it was built for a different use case in mind. As I mention in my talk, it generates more than 80 files that are not necessary to get you started. With a couple of folders and a single class inheriting from Rails::Engine you should be good to go.

By leveraging Rails engines are you implicitly signing yourself up for per-package dependency (gem) management?

This also assumes the package is a Ruby Gem which as I explored in the talk, it's not necessary. An engine can start as a simple code organisation tool and progress into more complex areas as needed like becoming a gem and managing its own dependencies. Again, not necessary for a simple split.

What do folks see as a tripwire for when you should reach for an engine/when you should reconsider an engine?

I'm not sure I understand this question. Is this about when to split your monolith into an engine or is it about when should you consider an engine vs a package (i.e. folder with a package.yml in it)?

[tylerwillingham]

@pinzonjulian it was pretty impressive to see you reverse engineer some of what I experienced while I was at Shopify from the outside.

I've been sure to share your talk several times in my new company. As folks work on modularizing their applications, there's a lot they can learn from and lean into, but I think the guidance your talk sets out sets the framework for a delightful experience building sustainable, large Rails application, once you're organically feeling those pain points.

In my opinion, Rails' plugin generator is not the right tool for the job because it was built for a different use case in mind. As I mention in my talk, it generates more than 80 files that are not necessary to get you started. With a couple of folders and a single class inheriting from Rails::Engine you should be good to go.

I go back and forth on this. Because it's documented by Rails I find it easy to opt into this and then delete what you know you need to get rid of. That feels like a more sane path for the majority of folks and it that can then be automated if you're in the process of defining several packages in relatively short succession.

This also assumes the package is a Ruby Gem which as I explored in the talk, it's not necessary. An engine can start as a simple code organisation tool and progress into more complex areas as needed like becoming a gem and managing its own dependencies. Again, not necessary for a simple split.

Your talk does a good job at demonstrating what can be optional here 👍 Even for more complex splits, I can see the benefit more clearly after going through that exercise.

I'm not sure I understand this question. Is this about when to split your monolith into an engine or is it about when should you consider an engine vs a package (i.e. folder with a package.yml in it)?

The latter.

This is objectively probably a tangential question. I am personally pretty weary of some of the advocacy for deconstructing into tiny packages. My only experience in delightful modular Rails apps comes from seeing the majority (if not all) packages as components/.

In many cases the lightweight engine approach you've demoed feels lean enough, while leaning into Rails conventions. So it's hard for me to fully wrap my head around the goldilocks package that fits in between being small enough to define something in a module with a couple of classes vs a lean Rails engine.

[rafaelfranca]

One day we will document this, but not in Packwerk. This isn't really a Packwerk feature, it is a Rails feature that already exists today, just not documented.

I have an existing documentation inside our internal library already but I'm not happy with it yet. Mostly because I don't want this documentation to become a reason for people write applications in the way Shopify does as soon as they start the application. I'm a strong believer that people should not start applications in a modular way. This is just like starting applications writing microservices.

But, when I find the right and balanced way to write this documentation, it will be written.

[exterm]

I'm interested in contributing to this, if that seems helpful.

[tylerwillingham]

Very fair.

My initial inclination to document something here comes from the recommendation living here, but providing documentation for different implementation details isn't inherently the responsibility of this gem. 👍

I have an existing documentation inside our internal library already but I'm not happy with it yet.

I think I saw this briefly but never circled back to reading more closely. :old_man_yells_at_past_self:

I'm a strong believer that people should not start applications in a modular way. This is just like starting applications writing microservices.

I could not possibly agree more.

[...] it is a Rails feature that already exists today, just not documented.

Can you point me to some code I could read to learn more?

[tylerwillingham]

I'm happy to check back here and keep the discussion going but this feels objectively like the answer so I'm marking it as such.

[rafaelfranca]

Can you point me to some code I could read to learn more?

Yes. It is mostly the implementation of Rails::Engine. @pinzonjulian got it right, that lego repository is a show case of that and to make this approach work you need Rails to generate less code than it does now in the plugin generator. There is a part of it that is new code to teach Rails how to automatically find engines that isn't in Rails yet, but some day will be.

[tylerwillingham]

Stopping back in to share another resource for future readers interested.

I've been trying to pursue the simplest path without relying on layers of other libraries and experimenting with that in tylerwillingham/overengineered (named so that people don't think this is a recommendation on how to build new apps).

Like Julian's example repo and Rafael's, it's using engines with some files removed and a few lines of code in application.rb to load those engines.