Management for custom node collections

[joepavitt]

Also covers areas: Version Control & Centralized Management

Background

We've heard from many customers that it is common to build custom nodes for their specific use cases. These custom nodes generally come in two forms:

• Subflows: User can select a flow within the Node-RED editor and create a "subflow", which becomes a re-usable asset in that particular editor. The user can then save this to the Team Library in FlowFuse, and import it to other Instances within FlowFuse too.
• Custom nodes: A developer has coded a custom Node-RED node and published it either to the public NPM registry, or to a private NPM registry made available only to the respective team/business.

Current Offerings

Team Library

FlowFuse offers the Team Library as a centralised repository for common flow.json samples or .js function nodes that can be imported to any Instances in the team.

Pain Points:

• Updating a flow.json or .js file is a manual process, users must manually re-import and replace each instance of any imported contents from the Team Library.
• There is no concept of version control here, once a file is overriden, the previous iterations are lost.

Private NPM Registry

In FlowFuse it is possible to configure external, private NPM registries. This makes any present Node-RED nodes in that registry available in the Palette Manager

Pain Points

• User needs to setup, host and configure their own third-party npm registry
• Requires users to develop/code their own nodes. This is not a solution that covers subflows.

Proposal

• We have version control built in via the palette manager, where used can update to the latest version for a given node.
• Subflows are essentially a single node, but just configured via a flow.json, rather than a repository with an html, js file, etc.
• Can we build in tooling that permits users to publish subflows to a private NPM repository?
• Can we host a private NPM repository (per team, or per platform?) to make the onboarding easier and prevent the need for customers to configure and host their own.

Customers

Which customers would this be available to

Team/Enterprise

Which customers have requested this?

• https://app-eu1.hubspot.com/contacts/26586079/record/0-2/18782320332

[joepavitt]

Relevant issue: #3666
Relevant issue: #3734

[joepavitt]

Notes from brainstorming session:

Can we host a private NPM repository (per team, or per platform?) to make the onboarding easier and prevent the need for customers to configure and host their own.

@hardillb - there are open source containers for the registry, but not sure we can do this with a single registry to provide to multiple teams. - https://verdaccio.org/

• Key here is that we need compartmentalising the registries/collections.
• Ideally, we just run one, but scope the projects to teams - far less maintenance.
• Needs technical exploration on the "what is possible?"
• @hardillb - I don't like the idea of running registry per team.
• It's going t cost us a lot to run this, as it's new containers
  • @joepavitt - this will be scoped to Enterprise
• How do users publish to this?
  • Joe: I'd be expecting this was an npm publish
  • Scoping @teamId/package-name
• registry.flowfuse.com vs. registry.team.flowfuse.com complexity/overhead of management too
• @knolleary can this be an opt-in feature? Especially in case whereby the registries needs to be per team to save costs

Actions

• PoC on authentication/access control over NPM registry, verify if a single registry is possible, and what constraints exist