Render a single pyproject.toml file from multiple templates

[drewoldag]

Hello! First let me say, I'm loving Copier! Our team adopted it about a year ago and now all of our projects share the same base template that we update regularly.

Over the year, our template has now grown, and it's becoming clear that soon we'll need to break it apart into smaller templates. I apologize in advance if this question has been asked (and answered) before, but I've had trouble tracking down a concrete answer for this.

Instead of writing in the abstract, let me be clear in our specific use case. Our template currently can, among other things, can create placeholder tests and automatically render Jupyter notebooks for use on ReadTheDocs. To support that we create a pyproject.toml file, that has a list of dependencies including pytest and nbsphinx (among several others).

If we separate the portion of the template that renders the Jupyter notebooks into a separate "documentation template", what is the best practice for defining the requirements in the pyproject.toml file? Experimenting locally leads me to believe that if we apply multiple templates that all affect the same file, that file will either be overwritten or unmodified by each new template applied.

I've seen a few discussions that seem close to addressing this: https://github.com/orgs/copier-org/discussions/1465, #934, but they feel a bit abstract for me. I'm hoping to find some advice for this particular issue that might unblock the path forward.

[drewoldag]

From what I've seen, it seems the best advice is to simply separate templates such that there is no overlap of files that they would affect in a rendered project. That seems mostly feasible, with the exception of pyproject.toml file. Almost any "add-on" template that provides new functionality or stub files would almost certainly include some new dependencies, so it seems that there must be a path forward that doesn't include manually updating the pyproject.toml file.

[yajo]

Most tools support configuration in custom files. I know it's nice to keep it all together in a ˋpyproject.tomlˋ file, but if it is more practical, you can just generate many small dot-files, each one by its own tool, and be happy with that.

[drewoldag]

Another place where this seems to be a problem is with pre-commit hooks in the .pre-commit-config.yaml file.

[yajo]

Regarding this, indeed we don't have the same flexibility.

Maybe you could have a specific linters/formatters template that you add to your projects as a separate one. This template asks the users which ones to enable, and configures the pre-commit config file. It can also drop the config files, although many of those linters can be also configured through CLI flags, which would let you have all config inside the pre-commit config file. In practice, you'd be having all configs in a file, which is mostly what you want.

Just some ideas.

[woile]

I think the solution to your particular problem is to use tasks

1. Run base template
2. Run scoped template
   1. Add a task

tasks:
  - poetry add mkdocs=1.0.0

Technically this works for any "configuration" that you can generate through cli's, but it stops being useful if you need to update a pipeline or a Dockerfile.

A possible solution to this is if copier supported "snippet injection". The way I see it, you could be able to insert in a rendered file a "keyword" like: COPIER SNIPPET: <ID>, and then in your template configuration you should be able to map jinja snippets to a snippet id, something like:

Example

In a Dockerfile:

# COPIER SNIPPET: DOCKER_LAYER

FROM python:3.11 as builder
# COPIER SNIPPET: DOCKER_RUN

In copier.yaml

snippets:
  - id: DOCKER_LAYER
    snippet_path: snippets/node-layer.jinja
    file_path: Dockerfile 
  - id: DOCKER_LAYER
    snippet_path: snippets/rust-layer.jinja
    file_path: Dockerfile
  - id: DOCKER_RUN
    snippet_path: snippets/python-build.jinja
    file_path: Dockerfile

[OverkillGuy]

I've been hitting exactly this issue of "how does a layered template MODIFY files", and have come up with a hacky but creative solution that may help you.

The idea is to store patch(1) files that can be applied via a copier task that runs via git apply!

See the below patch file example for pyproject.toml both adding a dependency (indeed easy via dedicated poetry add X command) but also a python entrypoint (aka poetry "plugin"), which stands in for "change that is harder to automate".

template/patch/mass-driver.patch:

diff --git a/pyproject.toml b/pyproject.toml
index 7ae5802..ea08686 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -17,6 +17,8 @@ my-testing-project = "my_testing_project.cli:cli"

 [tool.poetry.dependencies]
 python = "^3.12"
+# Library for making patches
+mass-driver = "*"


 [tool.poetry.group.test.dependencies]
@@ -81,3 +83,6 @@ pydocstyle = {convention = "google"}
 [tool.mypy]
 python_version = "3.12"
 check_untyped_defs = true
+
+[tool.poetry.plugins.'massdriver.drivers']
+my-testing-project = 'my_testing_project.patchdriver:SimplePatcher'

Complications

You may notice that some of the patch context includes legitimate base-template answer material like the python version and python package name.

So we turn the patch/mass-driver.patch into patch/mass-driver.patch.j2, make it a template, and change the content a bit!

BUT the patch won't apply properly if the template "above" doesn't know the base template's answers.

So off we go making the "overlay" template get the answers from the base template!

Note this solution breaks the "templates are horizontally independent" assumption that permeates in copier's documentation, replacing it with a notion of vertical "stack", or "overlay" of templates where the above "overlay" knows everything about the below template (answers-file filename, template questions/answers and their meaning...), and is assumed to have to target such a base specifically.

Adding base template answers to jinja context

Using a copier Context Hook (arbitrary Python func that can change the jinja context = extra metadata) I can read what the destination path looks like on filesystem, read the .copier-answers.yml file, yaml.load it, and add that to a baseconf dict in my context.

context.py:

import yaml
from copier_templates_extensions import ContextHook

# Expand the context to add all the old data from the sub-project we're loading from
class ContextUpdater(ContextHook):
    def hook(self, context):
        copier_conf = context["_copier_conf"]
        answers_path = copier_conf["dst_path"] / ".copier-answers.yml"  # Hardcoded!
        new_context = yaml.load(answers_path.read_text(), Loader=yaml.Loader)
        return {"baseconf": new_context}

template/patch/mass-driver.patch.j2:

diff --git a/pyproject.toml b/pyproject.toml
index 7ae5802..ea08686 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -17,6 +17,8 @@ {{ baseconf.project_slug }} = "{{ baseconf.package_name }}.cli:cli"

 [tool.poetry.dependencies]
 python = "^{{ baseconf.python_version }}"
+# Library for making patches
+mass-driver = "*"


 [tool.poetry.group.test.dependencies]
@@ -81,3 +83,6 @@ pydocstyle = {convention = "google"}
 [tool.mypy]
 python_version = "{{ baseconf.python_version }}"
 check_untyped_defs = true
+
+[tool.poetry.plugins.'massdriver.drivers']
+{{ baseconf.project_slug }} = '{{ baseconf.package_name }}.patchdriver:SimplePatcher'

And my copier's apply-patches task script, hooks/patches:

#!/usr/bin/env bash

set -euo pipefail

# Print, apply, and remove patches

# PWD is dst_path in templates
PATCHDIR=$PWD/patches/

echo "Checking for patches:"
find $PATCHDIR -type f -iname "*.patch"
echo "Applying patches..."
find $PATCHDIR -type f -iname "*.patch" -exec git apply {} \;
echo "Patch applied!"
find $PATCHDIR  -type f -iname "*.patch" -delete
echo "Patches removed!"

and finally all this tied in my copier.yml:

_subdirectory: template
_templates_suffix: .j2
_tasks:
  - {{_copier_conf.src_path}}/hooks/patches
 _jinja_extensions:
- copier_templates_extensions.TemplateExtensionLoader
- context.py:ContextUpdater

# [...] template questions

overall folder structure:

massdriver-template-overlay
├── context.py
├── copier.yml
├── hooks
│   └── variant_patches
└── template
    ├── {{_copier_conf.answers_file}}.j2
    ├── patches
    │   └── mass-driver.patch.j2
    └── src
        └── {{baseconf.package_name}}    # baseconf is dict from base template!
            └── patchdriver.py.j2

I'd love for copier to make it easier to do this "base template information" for me, but I don't see how this can be done in a way that can recurse (I fully expect 2 or more layers of templates that each know both layers below them, in the future, and cannot imagine how copier can do this in a generic way that is not brittle).

Hopefully this helps, sparks ideas on how to do template layering, or at least we all learn something from it =)

[woile]

Really nice solution, I like it even more than mine, as there's no "template leak", thanks for sharing!

A few questions:

1. How do you "produce" the template patches, do you modify a repo and then export the patch?
2. When you run the template, won't you have a patches folder? how do you avoid it from appearing in the output repo?

[OverkillGuy]

Ey, happy you like it.

For the template patch generation, I use a throwaway template-expanded repo, change the thing I want to change by hand, and save the git diff into a file.
Then (because that patch is not using jinja template variables), I start to replace manually the templated values with the template equivalent (looking back at the source template for reference), and test the templated patch by using a second repo from base template, on top of which I apply the overlay template.

In practice this looks like this:

A Makefile entry for make try with a few args to allow override:

.PHONY: try
# Expands the template in a local folder, for experimenting
PYTHON_VERSION=3.10
MAKE_TGT=all
# MAKE_TGT=docker-build docker-build-release
RANDOMIZED_PROJECT_NAME=$(shell poetry run python -c 'import faker_microservice;from faker import Faker;fake = Faker();fake.add_provider(faker_microservice.Provider);print(fake.microservice().replace("-", " ").replace("_", " ").capitalize())')
DYN_VERSION=True
ARGS?=
try:
# Wipe previous such templating if any
	-rm -rf template_expanded
# Re-expand with randomized vars
	poetry run copier copy \
		. \
		'template_expanded/new_project/' \
		--vcs-ref HEAD \
		--defaults \
		--UNSAFE \
		-d "description=A cool project" \
		-d "python_version=${PYTHON_VERSION}" \
		-d "dynamic_version=${DYN_VERSION}" \
		-d "project_name=${RANDOMIZED_PROJECT_NAME}" ${ARGS}
# Get in there and run make
	cd template_expanded/ \
		&& cd * \
		&& make ${MAKE_TGT}

This make try does template expansion then runs make in the new repo (which is my biggest target command).

This template_expanded repo is where I generate patches.

Then I create a second, clean repo from original template:

copier copy --trust gh:OverkillGuy/python-template sample_repo/
# gives me a clean repo including git commit and tag
# Apply the attempted overlay:
copier copy --trust --vcs-ref HEAD overlay-template/ sample_repo/ -a .copier-answers.overlay.yml
# check the changes are good:
cd sample_repo/
git status
git diff

As for the patches folder, yes I generate them initially, but erase it as a task, during the patch script above, using the -delete flag of find, so that I:

• first find (print) the patches
• find -exec = apply git patches
• find -delete = wipe the patch folder's content

[OverkillGuy]

CC @yajo, I'd like to hear what the maintainers think of this solution.

In particular, the breaking some of the (perceived from docs) assumptions around template-independence for multi-templates, to enable template "overlays" (stacks?) where the overlay knows and uses the innards of the template below.

But it opens some hard questions around how to template-update, as there's now
two versions to keep track of, and update command may get weird?

I believe this solution is cool as it enables something I've not seen elsewhere: overlay templates with meaningful file-modifications potential. And the overlay templates can thus be created/maintained in separate repo = third-parties (like private github orgs), on top of (public) base template.

But maybe there's a cooler solution, or maybe some of these ideas I'm proposing are worth porting in to copier itself, I dunno, hence pinging copier people.

(Small aside of "thank you for building copier": I thought I was going crazy, hoping to get cookiecutter to have template updates!)

[yajo]

Hi there! I love how you guys get creative about this!

No, I have no intention to support inheritable templates. There's a real need to document valid template reusability strategies. This could be one good contribution to the list.

However, the most powerful and maintainable strategy (for hard cases) I've found is the infamous copierception PoC from #934 (comment) (a.k.a. meta-templates; templates made to render templates).

That ticks all the boxes, but needs you to twist a bit your brain to understand. This may help: #934 (comment)

[adam-moss]

I arrived at this thread as a result of similar requirements, i.e. how to patch an existing file since it's a tricky challenge to avoid, especially in an upgrade scenario.

Reading the above as other tickets I wonder if there are some simpler use-cases that could be catered for that may hit the 80/20 rule. For example looking at what has been mentioned could the tasks (and migrations?) be enhanced?

E.g., if a method attribute (for want of a better word) was included you could have things like:

method == subprocess (or anything else, key thing is it is the default to maintain what it currently does)

method == patch_git, apply a git patch from an attribute patch_file

methods == patch_json or patch_yaml, apply a patch to json or yaml, would need some refinement around is it adding to (.e.g. pre-commit hooks), replacing, removing

method == patch_block, apply a managed block of text to a file, e.g. .gitignore. Analogous to ansible's blockinfile module

I think the above would be possible without introducing any new dependencies too.

[yajo]

I think we're overcomplicating it a bit.

Copier handles updates. Merges histories from your subproject and from the template.

When you apply 2 templates to 1 subproject and you update from one of those templates, from that template's point of view, anything you updated from the other template is just subproject history that it has to apply.

I guess it's easier to see it in action, so I recorded a video: https://youtu.be/EwzJA-v7DCQ

[woile]

I see what you mean, but for the users of the template, understanding this mechanic is too much. I think if instead it worked like git, it would be much more bearable. Like showing a git conflict that the user can take action right away. Maybe something like:

Something like a merge mechanism:

copier copy --merge ... . 

[yajo]

Yes, that'd be nice. You can open an issue about that enhancement if you want. However, for now, you can instruct your users on how to do that on your templates, or use tasks for the matter.