Proposal for dev scripts integration

[noirbizarre]

Hi 👋🏼

I've been spending a lot of time integrating some multiple python dev tools into a common consistent stack but I reach the conclusion that Python packaging (which might not be the proper term here) is still lacking some PEPs.

PEP 621 is in my opinion the main benefitable change for everybody (Thanks poetry for the inspiration, I believe most of the spec comes from this project if I'm not mistaken) and I can't wait for #3332 to be implemented.

But, if there is everything needed in poetry and/or the PEP 621 to describe a project and its exposed executable, the development process is not standardised at all. This is why I tried draft something for the development scripts here

Given I had poetry in mind while writing this because I'n using it, I would love to have your feedback on this.
Do you see value in this proposition ?
Is there a chance to have this kind of use-cases integrated (I would love to be able to poetry test or poetry lint and use multiple tools and backend behind to implement those tasks) ? (Note: the question is not if there is someone to implement it, I'd be glad to do it but more if there is a POC submitted as a PR, it there any chance that other would be interested by this, contribute on the POC and hopefully reach a mergeable state acceptable by the poetry team)

[abn]

I need to read your proposal in a bit more detail. However, this has come up before

• Add tasks #591 (closed with Add tasks #591 (comment))
• Saving custom scripts in pyproject.toml (alias npm run) #2496

While I do understand the need/motivation for this, I do not think having these exposed as root commands is ideal. This can cause all sorts of problems for tool developers, one of which being name collisions. I think npm avoids this using npm run task. I suspect the variations in what a task needs to do can make things tricky to say the least. My initial thought is that this might be better off as a plugin, but if it becomes a PEP the conversation might be different. Certain things could be abstracted out reliably, but this might only be a limited subset of what every project needs (this might end up being very similar to what tox does already).

I suspect the best way to make progress on a proposal like this would be develop a plugin that does something like this as a PoC. I believe @nat-n created https://github.com/nat-n/poethepoet to address this, but do not think that has been ported to being a template. I do have a few (personal) reservations on the approach being used in that project, but it is probably a good starting point and worth reaching out.

[noirbizarre]

Thanks for the links, I'll read them carefully.

For the proposal, yes, the end goal is PEP, exactly for the reason you cited: a PEP would allow each tool to only manage the complexity againt the PEP itself instead of having to handle each and every tool specifically. In the case of poetry + invoke (example), this would bring a clear separation of concerns:

• invoke exposes all available commands
• poetry can call them withtout having to know how dependencies are managed... it just know the invoke exposed them and it will call invoke with a known API, not an invokespecific one (bsaic API is a basically a subprocess run while the rich API can bring more but need to be defined)
• invoke is in charge of running the asked command, dependency resolution included. Poetry doesn't have the knowledge of it (and shouldn't ever)
• poetry can hook some specific command in the product lifecycle using the same mecanism (ie. run test and/or lint when performing a build or publish, as example again)

Poetry should never have any other knowledge than the PEP content (script discovery, special meanings if any, entrypoint specification if any) and invoke too. Tomorrow I want to switch from invoke to nox on a project, I just change the discovery provider by the nox one and that's all, no changes at all on poetry side.

Where it's becoming way more interesting when you start integrating CI tools or IDE (because replacing a inv my-command by poetry run my-command doesn't bring any value by itself: your IDE know with this PEP where to look for scripts, and how to run them (even more if we have special meaning as well as entrypoints), your CI can have dedicated actions to run some tasks, whatever project amanger, script runner... you use.

For the root commands, I'm against too except for the same cases as npm: the test script is a special one, you can call it with both npm run test and npm test. I believe this is not the only case. In my example, I just took testand lint which are 2 recurrent and well known cases (and also 2 cases on which there was already some aborted work).

I'm gonna continue the work on the Python's Discuss thread (dig the Brett Canon insights and especially what has been done in PEP517) and I'll probably publish an updated post by the end of the week with details.

I'll work on a POC (I'll take a closer look at poethepoet), either as a branch or as a plugin if it fits wells