-
-
Notifications
You must be signed in to change notification settings - Fork 188
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mockup and resource registry redone #3211
Comments
Having to do a Javascript bugfix with the existing resource registry is a complete nightmare... Huge +1 from me here |
I am generally +1 on the intention of this PLIP. Though, we have to take into account that this completely breaks the add-on architecture of Plone as we know it. We were always reluctant to do this in the past because it is a huge step and a complete paradigm change. How do you plan to keep the add-ons control panel with this new approach? You can not go to the add-ons control panel in Plone and install an add-on that has JS/CSS. You will always have to do a separate compile step and you need all the JS tooling properly installed for this. How do you plan to handle this? The add-ons control panel would give the user a false promise of being able to install an add-on product TTW, which won't be possible any longer. Are we going to remove TTW add-ons? Do we build a new JS package infrastructure? How do you plan to make JS packages depend on each other? How do you plan to do optimize the bundle when you have lots of add-ons installed? I am playing devil's advocate here (mainly because the questions above are the challenges we had to solve in Volto). Though, this loss of one of Plone's core features was one of the main reasons we started with Plone-Angular/Plone-React/etc. We could not imagine doing such a fundamental break in Plone itself. And if the user has to install the complete JS toolchain and run webpack for each new add-on anyways, you can as well go all the way If we talk about going in what I believe is the right direction, we should also talk about the long-term implications of this PLIP. Say we do this painful first step, the next step that makes sense is to make patternslib rely on Plone REST API, then make patternslib use a new modern JS framework (Rok wanted to rewrite mockup in React even before Plone 5 was released) and replace to old cruft. React is already in core and it basically won the JS framework war. Using something else at this point and not re-using Volto/React components would be just nuts (sorry for being blunt here). At this point we will have a wild mix of backend and frontend templates which is hellish (I saw many companies and people trying this and they failed 100% of the time). Then you want a clean separation between frontend and backend (which Plone REST API provides) at some point... We went through this entire learning and decisions making process 3-5 years ago. We did all the mistakes, we tried way too many frameworks to end up with Volto. If this PLIP gets into core and we continue to work successfully on this, we either end up with a second Volto or with Plone classic becoming more and more what Volto already is. In any case, we will split the limited resources that we have working towards the same goal, to have a well maintained JS stack for Plone. As said, I am not opposed to this PLIP. I am strongly against bringing yet another JS framework into core. Though, I hate to see that we put effort into solving a problem that we had not too long time ago and that we already successfully solved with Volto. One other thing to consider: we had to come up with a complete new add-ons architecture for Volto and we still have to figure out how to make this work in Plone 6. If we will have two JS add-on architectures and JS build systems for Plone 6 this will confuse the hell out of people. |
As I understand @thet this is still perfectly possible, and much easier than before. |
Do you plan to use @datakurre's approach to querying the Plone resources and bundle them externally? |
@tisto thanks for raising these questions! I think most of add on JavaScript might just be installable as simple, separately packaged bundles. That should not make much problems as long as they do not have dependencies to Mockup's dependencies (except jQuery which is still available in the global namespace). If so, we might end up delivering multiple versions of these dependencies. To solve that, integrators would need to create a custom JavaScript bundle with all the necessary dependencies - something I see a neccessity for any mid-sized project anyways, also with our current infrastructure in Plone 5.x. Regarding moving to plone.restapi: a big +1 for all of Mockup's API calls. For Patternslib this is not so relevant - currently we only use the Patternslib core to register patterns. At a recent talk with @MrTango we agreed that we might not move to plone.restapi right away to keep up the backwards compatibility - there might be some customized endpoints used in projects. However, using plone.restapi instead of custom endpoints is on my wishlist too. Regarding a rewrite to use a modern JS framework: Also +1 but not right away. The first goal would be to modernize the whole codebase. Then we can explore possibilities to integrate other frameworks - even much easier when we switched to ES6+ and removed RequireJS. Reusing Volto components: definetly a good idea, especially since Volto has a lot of components which Mockup also offers but in a much better quality. I'd love to have Volto more modularized and be able to just import individual components without having to rewrite Mockup in React. The other way around would also be a very interesting option: moving towards Web Components and reusing thosre in Mockup and Volto at the same time (Mockup then might even be obsolete). And no, I don't see the resource registry involved in compiling bundles, as @datakurre did for his Webpack projects. Actually all resources should be cleared from the resource registry - IMO we do and should not need them anymore. If one wants to create a optimized bundle it would be necessary to set up a JavaScript project, add all dependencies to package.json and create a bundle there. We need a template to make it easy to bootstrap such a project. |
I have good news for (all of) you. This PLIP is already implemented, it's called Volto: There is a whole track (Track 1) at the Plone conference about it, so you all can get updated with all the new things the Volto community has implemented there, and the plans for the future. https://2020.ploneconf.org/schedule Oh, one more thing, as far as I know Volto is Plone 6. |
@sneridagh that does not help, even if you put many blood and tears into Volto. We need Plone 6 Core still, we can not all switch immediately to Plone 6 Volto. Do not get me wrong, I really like Volto. But Plone 6 Core with server side template based rendering will exist for a while.
So no, Plone 6 is both: Core and Volto. On install there must be a choice what to install. I know the marketing team hates that, the Volto team either. But we can not ignore the reality of Plone users, companies, organisations, developler, ... |
That's not the issue, there can't be two modern JS stories in Plone. Full stop. Svelte? Add yet another different web technology to the stack? Web components what? Really? No, thanks. We worked our asses off in Volto to improve the approachability to Plone for now do this. When you start to work on it, you'll get into the same pitfalls and will solve the same issues we already solved. You'll end up doing exactly the same we did for Volto. By doing this, you will confuse people, and valuable resources will be spent in re-writing what it's already done. That also hurts. Sorry, overlook them and ignore (and continue doing so) them hurts me, and all the Volto community (and yes, there is one, a healthy one). |
@sneridagh Plone 6 will still ship with the page templates rendered on the server. There is also the barceloneta-lts project for modernizing the UI. And for that we need (IMO) an updated JavaScript stack. This PLIP is just about modernizing the existing JavaScript stack and is not meant to compete with Volto. Regarding Svelte, Web Components etc: Just ignore that. Whatever developers use to add JS functionality to "classic" Plone is up to them. Still, I'd really love to see Volto modularized in a way that individual components like folder content browser can be reused outside of Volto. I did not have time to explore that, but this is on my list. |
@sneridagh I understand your point, but I don't think it's all black or white. The goal is, what ever get's in there has to be really reusable. Svelte components and especially web components (custom elements) are ideal for this job. They work pretty much everywhere (Plone, static HTML, Patternslib, JS Frameworks) and also be able to use Redux, like Volto. The way I see it, this will bring the core Plone and Volto closer together and make it easier for people to switch at some point. It might not be obvious right away, but Volto can benefit from sharing components with existing Plone UI. But back to the main points of this PLIP:
I know, nothing in this list is concerning, when one is using the Volto frontend, but this is not an option for everybody and not for every project at this time. I feel comfortable with backend and frontend development, but many existing Plone developer are not and for them the existing Plone core does a better job. Let's please not forget them and force people to abandon the existing Stack. They both can exist and hopefully start sharing more together than the backend. Volto will grow and so will the developer base on the frontend side, but this is not going over night. |
LTS is one of the suggested names for the old-style Plone stack with server-side rendered templates, aka Plone 6 Classic / Plone 6 Core / Plone 6 LTS. Anyway, in the spirit of LTS I think it would make little sense to force people to rewrite any add-on code whatsoever. Any backwards incompatibility means we are forcing someone to decide whether to spend their dev money on an upgrade vs a switch to a different platform. We simply cannot afford to lose a single Plone installation. So don't give them a reason to leave. I appreciate all of the ideas in this PLIP, and I know how painful the resource registry and the JS experience is. So I welcome any and all improvements. But, again, in the spirit of LTS, I suggest making only the improvements that don't break backwards compatibility. The resulting PLIP might be much uglier and possibly even harder when done with this goal in mind, but a bird in the hand is worth two in the bush. |
|
+1 for not breaking with add-on's. There is no intention of that, we will do what we can to make it as seamless as possible for add-on developers. |
And to make it clear, there will be no other JS framework comming into core. custom-elements are not a framework but an existing web standard which is supported by all modern browsers. How you create such custom elements is up to you. One way is from cratch, one is to use libraries like LIT-Element and another is to use compiler like Stencil and Svelte, which produces JS at the end. So there is no framework at the end and also there is no runtime of a framework. |
Removing RequireJS will break backwards compatibility for add-ons which depend on RequireJS. |
@thet and add-ons with JavaScript not implemented as pattern are not affected. Except if they are using RequireJS? |
@jensens yes, exactly. "legacy" bundles as we called them (no compile flag set, no requirejs) should still work as before. |
@sneridagh you can always be Guillotina 7 Super power UI ;) |
Added a note about using Patterns from Patternslib instead of Mockup where possible as an optional task. First step would of course to be to bring Mockup to ES6 and the test suite green. |
@thet @MrTango You are such heros for working on this! Just a few thoughts from my experience on building all of our Plone 5 themes with webpack to possibly keep in mind:
|
Since we had some confusions in todays FrameWorkTeam meeting, about what the change with the Resource Registry and the removal of requireJS means for Add-on's. Add-on's will be installable as before, no change there. There will be no big Webpack to rule them all. We will use webpack to build the plone-core bundle which includes things like Mockup/Patterns and some core resources. This bundle will be registered in the Resource Registry as it is now. Resouces for add-on's will be manged by the individual add-on. There is no dependency injection happening. Add-on's should build decent bundle sizes and thing about if they schould be loaded all the time or just in some situations. If an add-on has a bigger library it depends on, this should be dynamically (lazy) loaded when the code is used. This keeps the bundle sizes down and resourcs are only loaded when they are needed. |
Hi all, I hope I'm not speaking out of turn here. This seems to me like a very good approach, but I'd like to stress the importance of high quality comprehensive documentation for major infrastructural changes like this. In my experience, one of the biggest issues with the Plone 5 Resource Registry was a lack of documentation of how to use it either as an add-on developer or a site integrator. The initial documentation was largely inadequate until a while after release, and it's still far from comprehensive. For example, there's still no documentation indicating best practices for writing/updating an add-on to support multiple versions of Plone with different RR schemes. I believe that sort of thing is very important if we want people to e.g. update add-ons to properly support newer versions of Plone. |
From the discussion at the open space at the Plone conference 2020: Adding JavaScript code via addons.Adding JavaScript code via addons will be easier in a way that we allow to add bundles by providing them as static resources and adding a registry entry for that. The simplification to the current situation will be:
As far as the optimization of extra bundles added via addons goes, we have these options:
Notheworthy mention: CalmJS (https://pypi.org/project/calmjs/) allows for dependencies managed in Python code (setup.py/cfg). As this is much like Funstatic a solution fun off the standard we probably do not follow it. Whatever we do, we need to make it simpler. We want to reduce complexity and probide a well documented, painless, easy, sane and reproducable way of managing JavaScript in Plone. |
💯 IMO, an after all the discussions, considering the community response to sophisticated Plone 5 resource management, returning back to "just bundles" is the only right way to do. Hopefully there is sane way add-ons to support this in backwards compatible way, when required. Conditional records should help https://pypi.org/project/plone.app.registry/#conditional-records I'm looking forward for Webpack module federation to solve most of the issues with add-on families similar to eea-packages and allow them to avoid duplicating dependencies in individual bundles. The complete Webpack approach presented in plonetheme.webpacktemplate will become a bit harder now that the source files are now longer in the registry (or packed with the add-on packages), but must most probably be fetched from their original sources or npmjs. That was my first approach years ago, but back then it was a nightmare to maintain compared to fetching everything from Plone registry. But I'm sure that this will become much easier now that Mockup itself is modernized. So, once I'd updated plonetheme.webpacktemplate for Plone 6, also this use case is covered. |
PLIP looks good. Only thing I would love to see is documentation for addon writers to port their addons to the new tech. |
Resource RegistryWe have a new much simpler resource registry (RG) which is basically a form not a JS based UI as before, so it will not break when you change things in the RG. You only register compiled bundles, this is very close to as it was possible before. There will be no more individual resources, as we don't support bundling in the browser. What you use to create your resource bundles is up to you. It can be plain CSS/JS or compiled by Rollup or Webpack & co. Mockup / Patterslib PatternsAll resources are standard ES-modules and can be use in you own project Webpack & co and be further optimized. Better Add-on's supportBesides that the registration of resources will be simpler in general, you will have some advanced options to use JS-libraries, which you can use if you want. Things like Mockup, Patternslib and potentially others can be shared by add-on's via Webpack Module Federation. For an add-on this means, with a bit extra config, you can share a library with other participants in Plone and avoid double loading and conflicts. But all this is opt-in, so if you don't care, nobody will bother you. The migration of add-on's will be fairly simple. If it uses so called legacy (externally compiled) bundles, there is nothing to do. If you where so brave and wrapped the resources with requireJS, this can be removed. All you need is a simple bundle registration of whatever CSS/JS you have. For add-on's which where using Mockup/Patternslib via RequireJS, this needs to be changed to a Webpack Module Federation style, which will be documented soon. StatusWe only have some small bugs in pat-relateditems and some unit tests to fix, the rest in already in a good shape. |
This was discussed in the @plone/framework-team mailing list and nobody spoke against. Anyway it is still good to have a PLIP, I think. |
I have tested it, from the point of view of a site admin or editor, and a bit as add-on writer. I did not look at how things are implemented in the code. Tested on Mac in Firefox. Some review notes:
I have created an issue for the first point. I can create separate issues for the others if wanted. Where? In the The migration problem strikes me as the main one that would need to be tackled soon, otherwise a migrated site basically does not work. Being able to move items in folder contents would be welcome too. The other items seem minor and could be fixed later. I did not notice problems in the Site Setup or TinyMCE. Everything works the same or better. I did not do in depth testing though. I did not try adding an add-on with some css and javascript. But it looks really easy. You point to one javascript and one css file of the add-on. Maybe specify one or more other bundles like Actually, since it sounds easy, let me create an add-on with
I activated the add-on, and both my javascript and css worked. Without
I added On the network tab I see some javascript chunks getting loaded. I don't know if that is the module federation or something else. But it seems to work. Summary: it is looking pretty good! |
It happened to me too, If you re-run the last staticresources import step and cmfplone one, jquery bundle pops up. Seems like some upgrade steps add/erase things, and something runs not in the right order. |
This is by intend.
|
Also I would like to see some documentation on this one and if possible a demo add on, something similar to https://github.com/collective/example.p4p5. It does not need top work on both p5 and p6 (though it would be a nice plus) but at least it should be really clear how to wire up custom css + javascript in Plone 6 add on. |
This can be closed, right? |
PLIP (Plone Improvement Proposal)
Responsible Persons
Proposer: Johannes Raggam
Seconder: Maik Derstappen
Abstract
This PLIP is about reworking Mockup to be based on up-to-date JavaScript standards and to simplify the resource registry.
Mockup will be based on Patternslib version 4 and integrate many concepts from there:
The simplification of the resource registry is largely based on the removal of RequireJS:
Mockup and the resource registry should get much easier to understand and to work with, more up-to-date and match developer expectations better.
Additionally we want to advertise the development of new JavaScript functionality as web components but any JavaScript framework should be compatible with Plone.
Motivation
Mockup was once and avant garde answer to the growing problem of managing JavaScript in Plone. Based on Patternslib it offered clear and defined dependencies via RequireJS, a JavaScript testing framework and a flexible build system incorporating the resource registry with through the web compilation and the possiblity to overwrite specific resources with a custom implementation. Other technologies like web components or reactive JavaScript frameworks were not available or not an option back then.
Now, Mockup lags years behind current JavaScript development. It's no big fun to develop for Mockup (although there are nice parts, like live reloading in Plone without having to build bundles). Mockup depends on some long outdated and inactive libraries - the most important here is RequireJS. RequireJS was the corner stone to solve the dependency problem with JavaScript in Plone 4. But meanwhile it prevents us from using modern JavaScript features. There are also other outdated canditates: the complex integration of the Grunt build framework, the use of React 0.10 for our documentation framework, the integration of the unmaintained jquery.drag/drop which holds us on jQuery 1.x (there is a seperate PR for fixing this) - and so on.
The resource registry is also complex and hard to understand. The through the web compilation (actully client side compilation on the web browser) works like magic but creates a hard dependency on RequireJS and a specific version of the LESS compiler. This bounds us to a complex development workflow and prevents developers to make their own choices regarding JavaScript libraries, ECMAScript versions and build frameworks.
For good reasons we Plone community failed get developers on board helping out in the JavaScript area of Plone.
This PLIP aims to fix this situation by a number of actions outlined below.
Assumptions
Proposal & Implementation
Mockup will be completely restructured and based on the concepts of Patternslib 4:
We base the whole Mockup code on modern JavaScript ES6+.
We completly remove the dependency on RequireJS and use the ES6 module exports/import exclusively (this also removes the through-the-web compilation feature in CMFPlone).
We use dynamic imports (lazy loading) for 3rd party dependencies like TinyMCE or Select2. This dramatically reduces the size of generated bundles as no external dependencies other than the required minimum (Mockup, Patternslib core, jQuery, underscore) are loaded. All dynamically imported dependencies are compiled by Webpack to a
chunks
folder.We put all patterns (except some, like the resource registry or model editor) into one bundle. Due to the lazy loading approach of 3rd party dependencies we can include all useful patterns into the bundle without significantly increasing the bundle size.
We remove the customized Grunt build system.
For building JavaScript bundles we use Webpack.
We use BabelJS for transpiling ES6+ code to target older browsers.
We use a seperate polyfill package for IE11, which is loaded optionally via browser detection (and remove this as soons as IE11 is not relevant any more).
For documentation we use Markdown files and compile them with the 11ty static site generator.
We remove our self written documentation framework which is based on the outdated React 0.10 library.
For testing we use Jest, which itself uses jsdom instead of a browser backend. jsdom is a JavaScript implementation of the document object model and allows for much faster test runs than testing against PhantomJS.
PhantomJS, Karma, Mocha, Chai will be removed.
We apply consistent code formating rules throughout the whole Mockup code base, based on prettier and eslint, using a indentation of 4 spaces.
Optional: Where possible we want to use Patterns from Patternslib instead of Mockup.
Patternslib is better maintained and each pattern follows a strict specification defined in a design phase.
Patternslib patterns do not use a JSON configuration syntax but a simpler CSS like syntax.
Wether we subclass Patternslib patterns, adapt the configuration and change the parser or we start using also the Patternslib configuration syntax in Plone.
However, this would probably also include some markup changes (an example would be the switch from the date/time picker
pat-pickadate
topat-datetime-picker
).This is an optional task and would only be done after all existing patterns from Mockup are migrated and the test suite is green.
This would also raise the version number from 4 (version with ES6 migration) to 5 (optimized Patterns).
For the resource registry the changes will be:
We keep the bundle registration mechanism. JavaScript and CSS bundles (or resources) are registered as
bundles
similar to Plone 5.x.A bundle can contain one JS and/or one CSS file but not multiple JS files like it's currently the case.
We remove the concept of
resources
which was only necessary for RequireJS to find it's named dependencies and to compile TTW or viaplone-compile-resources
.All resource definitions can be removed from the Plone registry.
add_bundle_on_request
andremove_bundle_on_request
will still be available.add_resource_on_request
will be a deprecated alias foradd_bundle_on_request
.Removals:
TTW bundle compilation functionality.
plone-compile-resources
script. The Mockup bundle will be created with Webpack, other bundles as developers like to.Removal of the concept of stub modules.
Removal of the conditional comment setting (not supported since IE11)
Removal of unnecessary Plone core bundles.
A upgrade path should be provided to ensure a migration where it is possible. Bundles with
compile
set to on depend on RequireJS and need to be rewritten, though.jQuery will still be globally available by default as a separate bundle.
Caching URL improvements:
last_compiled
).chunks
folder.For integrators we encourage to write new JavaScript functionality wether as Patternslib Pattern or as a "web component" or "custom element". Bobtemplates.plone shows an example how to create a web component based on "Svelte", which has a low footprint and uses a standards-compliant web component approach. However, the framework is not important as anything can be used.
Deliverables
Risks
compile
set to on) need to be slightly rewritten.Participants
Resources
Mockup ES6 rewrite PR (WIP): plone/mockup#1025
Patternslib ES6 (production ready): https://github.com/patternslib/Patterns
Simple example pattern using Webpack, ES6, dynamic imports, Jest: https://github.com/Patternslib/pat-sortable-table/blob/master/src/pat-sortable-table.js
/cc @plone/framework-team
The text was updated successfully, but these errors were encountered: