The usefulness of macros in Jinja2

[joshuataylor]

Hi!

I thought it would be good to start a discussion around macros, and discuss their current uses in Python (Jinja2), and how they could (or should!) be implemented in Minijinja.

I'm going to be focusing on presenting this from the perspective of someone who has extensively used dbt, and is currently writing a dbt-compatible engine in Rust. I'm going to be using dbt's macros as a reference, and whether or not I believe they are a good idea to implement in Minijinja. So please take my opinions and give me feedback, as I would love to hear your thoughts!

To give a quick primer on dbt (as it will make the rest of the ramble/discussion make more sense):
dbt is a tool that allows you to build SQL models, which are then compiled and executed against your data warehouse. It's a great idea, and allows you to build a lot of complex SQL models in a very simple way, and have them version controlled etc. I really like the concept, I'm not affiliated with the project, and I'm not trying to advertise it - I just want to give context :-).

With dbt, you define a macro on a project level, which can be used by any model in your project. Most macros in projects are fairly simple (at least ours are, we have ~30?). There are also packages (what dbt calls plugins) such as dbt-utils, which aims to provide a set of useful macros that can be used in any dbt project. These macros are more complex, and try to be cross-database compatible.

As the COMPATIBILITY.md states:

The macro tag works very similar to Jinja2 but with some differences. Most importantly the special caller, varargs and kwargs arguments are not supported. The external introspectable attributes catch_kwargs, catch_varargs and caller are not supported.

The caller tag is used extensively in dbt-utils and dbt, as it's extremely useful. This is because they have a macro like this:

source

{%- macro statement(name=None, fetch_result=False, auto_begin=True, language='sql') -%}
  {%- if execute: -%}
    {%- set compiled_code = caller() -%}
  ...

Setting the compiled_code allows plugins (dbt connectors, whatever) to wrap this later:
source

{% call statement('check_schema_exists', fetch_result=True) -%}
      select count(*)
      from {{ information_schema }}.schemata
      where upper(schema_name) = upper('{{ schema }}')
          and upper(catalog_name) = upper('{{ information_schema.database }}')
{%- endcall %}
{{ return(load_result('check_schema_exists').table) }}

This is super useful, because you can just have the SQL inside the call statement, which is then later loaded.

However... I'm not 100% convinced that this is a good idea. You start having to think about if you also pass in the context, and how you compile that.... it's becomes incredibly complex, and I'm not sure it's worth it.

So what would the Minijinja equivalent be? I'm not sure, honestly.

IMHO, passing in what you actually want to the function is a much clearer, especially as it removes some of the "magic".

[mitsuhiko]

When I implemented macros here I did consider implementing {% call %} as well. In many ways it's just a hidden macro so the extra work required to do that is pretty trivial. However I opted against it mostly because I personally did not use caller much and I always found the syntax quite confusing. That caller is actually used in the real world gives some indication that it's worth supporting I suppose.

[mitsuhiko]

I made a PR here that adds support: #164