diff --git a/search/search_index.json b/search/search_index.json
index 06013f9..e741fa7 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Tapestry","text":"Tapestry is a framework for writing (postgres)SQL queries and (pgTAP) tests using Jinja templates.
Tapestry is written in Rust but it can be used with applications written in any programming language. It's purely a command line tool that renders Jinja templates into SQL files. How to load the resulting SQL code into memory and use it at runtime is entirely up to the application.
This approach of loading SQL from files is not new. There are existing libraries such as yesql, hugsql (Clojure), aiosql (Python) etc. that provide excellent abstractions for it. In absence of such a lib for the language of your choice, it shouldn't take more than a few lines of code to implement a simple file loader. In Rust apps, I simply use the include_str! macro.
One limitation is that tapestry can only be used with PostgreSQL, because of the tight coupling with pgTAP.
You may find this tool useful if,
you prefer direct SQL queries over ORMs or query builders to interact with RDBMS from application code
you are not averse to the idea of having (reasonable amount of) business logic inside SQL queries
In fact, if you have had concerns about point 2 i.e. having business logic in SQL queries, perhaps tapestry addresses some of those concerns. Learn more about the rationale behind this tool.
If you prefer a hands-on introduction, check the Getting started page.
"},{"location":"rationale/","title":"Rationale","text":""},{"location":"rationale/#problems-with-using-raw-sql-in-application-code","title":"Problems with using raw SQL in application code","text":"For many years, I've believed that,
it's a good idea to write raw SQL queries (safely) for interacting with an RDBMS from application code using libs such as yesql, aiosql etc.
it's ok to add reasonable amount of business logic in the SQL queries, rather than using SQL merely for data access.
Still, I've had concerns about using these ideas in practice, specially in serious projects.
"},{"location":"rationale/#unit-testing-sql-queries","title":"Unit testing SQL queries","text":"Typically, unit tests are written against application code. As more and more business logic gets moved out of the application and into SQL queries, the queries become longer and more complex. In contrast, the application code is reduced to just making db calls using the driver/client library. At this point, it makes more sense to test the queries than the application code.
Fortunately for PostgreSQL, we have the excellent pgTAP extension that makes it easy to write unit tests for raw queries. Just like the raw queries themselves, pgTAP tests are typically defined in SQL files. But since the query and the tests are in separate files, it's possible that one modifies the SQL query, but forgets to update the tests, and the tests could still pass!
How to ensure that the tests actually run the exact same query that's being run by the application?
"},{"location":"rationale/#maintenance-overhead-of-multiple-slightly-differing-queries","title":"Maintenance overhead of multiple, slightly differing queries","text":"An application often needs to issue similar queries but returning different set of columns or with different WHERE clauses based on user input. In such cases, a unique query needs to be written and maintained for every combination of the input parameters. This could result in multiple queries that differ only slightly. If some core part of the query needs a change, one needs to remember to update multiple SQL files.
Moreover, higher level abstractions (e.g. yesql etc.) usually cache queries in memory, so they require the queries to be given a name or an identifier. Since the queries differ only slightly, trying to give them unique names can be tricky.
"},{"location":"rationale/#how-tapestry-solves-it","title":"How tapestry solves it?","text":"Tapestry was built to specifically address the above problems and concerns. It does so by generating actual queries as well as pgTAP test files from Jinja templates, instead of having the user write raw SQL.
"},{"location":"rationale/#query-templates","title":"Query templates","text":" You write query templates instead of raw queries Multiple queries can be mapped to the same query template. Mapping is defined in the tapestry.toml manifest file. User defined Jinja variables can be used for conditionally adding or omitting parts of the query e.g. a WHERE condition or column to return. These Jinja vars are also defined in the manifest file. Thus, it's easy to generate and maintain multiple queries that are similar enough to be defined using a single query template. "},{"location":"rationale/#test-templates","title":"Test templates","text":" pgTAP tests are also written as Jinja templatesTest templates are mapped to queries, again in the manifest file. One query can be mapped to multiple test templates. When tapestry renders the final test file from a test template, a special Jinja variable {{ prepared_statement }} gets expanded to the actual query that the test template is mapped to. This way, the generated test SQL file is guaranteed to have the exact same query which is used by the application code. "},{"location":"rationale/#naming-conventions","title":"Naming conventions","text":"Tapestry suggests some conventions for naming queries consistently but they are not mandatory.
"},{"location":"user-guide/","title":"Overview","text":"Tapestry is built to address the peculiar concerns that I've had about using libraries such as yesql, aiosql and the likes. While I agree with the philosophy behind such libs\u2014that SQL code is better written as SQL directly rather than building it through ORMs, query builders or worse, by string interpolation or concatenation\u2014I've had some concerns about using the approach in practice.
To understand more about the problems and how tapestry addresses them, please check the Rationale page.
The general idea behind this tool is, instead of users writing raw SQL queries, have them write Jinja templates from which SQL queries as well as pgTAP tests can be generated.
Here is a high level overview of how you'd use tapestry in your project:
Create a directory inside your project where the templates will be located. The tapestry init command does this for you.
Add some information in the tapestry.toml manifest file:
Lists of query templates, queries and test templates along with the mappings between them Location of query templates and test templates (input files) Location of where the output files are to be created etc... Run tapestry render command to generate the SQL files, both for queries as well as tests.
Use a lib such as yesql, aiosql etc. to load the queries rendered by the previous step into the application runtime.
Use pg_prove to run the pgTAP tests, preferably as part of CD/CI. You'd need to implement some kind of automation for this. The github repo also includes a docker image that may help with this.
"},{"location":"user-guide/commands/","title":"Commands","text":"This page documents all commands in the tapestry CLI.
Note that for all commands except init, this tool will try to read the tapestry.toml manifest file in the current directory and will fail if it's not found. This implies that all tapestry commands except init must be executed from within the \"tapestry project\" root dir.
"},{"location":"user-guide/commands/#init","title":"init","text":"The init command can be used for scaffolding a new tapestry \"project\". It will create the directory structure and also write a bare minimum manifest file for us. In a real project, you'd run this command from within the main project directory, so that the files can be committed to the same repo. Example:
Running the following command,
tapestry init myproj\n.. will create the following directory structure
$ cd myproj\n$ tree -a --charset=ascii .\n.\n|-- .pg_format\n| `-- config\n|-- tapestry.toml\n`-- templates\n |-- queries\n `-- tests\nThe validate command checks and ensures that the manifest file is valid. Additionally it also verifies that the paths referenced in the manifest actually exist and are readable.
"},{"location":"user-guide/commands/#render","title":"render","text":"The render command renders all the template files into SQL files.
"},{"location":"user-guide/commands/#status","title":"status","text":"The status command can be used to preview the effect of running tapestry render command. It will list which output files will be added, modified or remain unchanged if the render command is run. This command will not actually write the output files.
Output of running tapestry status inside the examples/chinook directory:
$ tapestry status\nQuery: unchanged: output/queries/artists_long_songs.sql\n Test: unchanged: output/tests/all_artists_long_songs_count_test.sql\nQuery: unchanged: output/queries/artists_long_songs-limit.sql\nQuery: unchanged: output/queries/artists_long_songs-genre-limit.sql\n Test: unchanged: output/tests/artists_long_songs-genre-limit_test.sql\nQuery: unchanged: output/queries/songs_formats-artist-album.sql\nQuery: unchanged: output/queries/songs_formats-artist-file_format-album.sql\n Test: unchanged: output/tests/songs_formats-afa_test.sql\nIn a way, it's sort of a dry run for the render command.
"},{"location":"user-guide/commands/#-assert-no-changes","title":"--assert-no-changes","text":"A more effective use of this command though is with the --assert-no-changes flag which will cause it to exit with non-zero code if it finds any output files that would get added or modified upon rendering. It's recommended to be run as part of CD/CI, to prevent the user from mistakenly releasing code without rendering the templates.
"},{"location":"user-guide/commands/#summary","title":"summary","text":"The summary command prints a tabular summary of all queries along with their associated (query) templates and tests.
"},{"location":"user-guide/commands/#-all","title":"--all","text":"When --all option is specified with this command, the summary will include query and test files inside queries_output_dir and tests_output_dir respectively that are not added to the manifest.
Note
There are legit use cases for having files in the query and test output directories that are not added to the manifest Examples:
queries that don't need any tests but need to be stored in the same directory as other queries, so that yesql, aiosql libs can load all of them together.
Existing queries which are not yet migrated to tapestry (gradual migration strategy).
pgTAP tests written for stored procedures, views, schema etc. that need to be stored in the same directory as other tests, so that all tests can be run together.
"},{"location":"user-guide/commands/#coverage","title":"coverage","text":"The coverage command prints a list of queries along with the no. of tests (i.e. pgTAP test files) for them. It also prints a coverage score which is calculated as the percentage of queries that have at least 1 test.
Example: Following is the output of running tapestry coverage inside the examples/chinook dir.
$ tapestry coverage\n+----------------------------------------+------------------------------------+\n| Query | Has tests? |\n+=============================================================================+\n| artists_long_songs | Yes (1) |\n|----------------------------------------+------------------------------------|\n| artists_long_songs*limit | No |\n|----------------------------------------+------------------------------------|\n| artists_long_songs@genre*limit | Yes (1) |\n|----------------------------------------+------------------------------------|\n| songs_formats@artist+album | No |\n|----------------------------------------+------------------------------------|\n| songs_formats@artist&file_format+album | Yes (1) |\n|----------------------------------------+------------------------------------|\n| Total | 60.00% |\n| | (3/5 queries have at least 1 test) |\n+----------------------------------------+------------------------------------+\n--fail-under","text":"By specifying the --fail-under option, the coverage command can be made to exit with non-zero return code if the percentage coverage is below a threshold.
$ tapestry coverage --fail-under=90 > /dev/null\n$ echo $?\n1\nThe value of --fail-under option must be an integer between 0 and 100.
The above command can be run as part of CD/CI to ensure that the test coverage doesn't fall below a certain threshold.
"},{"location":"user-guide/docker/","title":"Docker","text":""},{"location":"user-guide/docker/#docker-based-workflow-for-running-pgtap-tests","title":"Docker based workflow for running pgTAP tests","text":"tapestry only generates SQL files for queries and pgTAP tests. To be able to run the tests you need to install and setup:
PostgreSQL server pgTAP, which is a postgres extensionpg_prove, which is a command line test runner/harness for pgTAP tests While these can be setup manually, the tapestry github repo provides a docker based workflow for easily running tests generated by tapestry against a temporary pg database.
The relevant files can be found inside the docker directory under project root.
Note
I use podman instead of docker for managing containers. Hence all the docker commands in this doc have been actually tested using podman only. As podman claims CLI compatibility with docker, I am assuming that replacing podman with docker in the below mentioned commands should just work. If that's not the case, please create an issue on github.
"},{"location":"user-guide/docker/#build-the-docker-image","title":"Build the docker image","text":"cd docker\npodman build -t tapestry-testbed -f ./Dockerfile\npodman run --name taptestbed \\\n --env POSTGRES_PASSWORD=secret \\\n -d \\\n -p 5432:5432 \\\n tapestry-testbed:latest\nVerify that the 5432 port is reachable from the host machine.
nc -vz localhost 5432\nThe above podman run command will create a container and start it. After that you can manage the container using the podman container commands
podman container stop taptestbed\npodman container start taptestbed\nThe pg_prove executable is part of the image that we have built. But to be able to run tests inside the container, we need to make the database schema and the test SQL files accessible to it. For this we bind mount a volume into the container when running it, using the --volume option.
The container image has a bash script run-tests installed into it which picks up the schema and the test SQL files from the mounted dir.
The run-tests scripts makes certain assumptions about organization of files inside the mounted dir. Inside the container, the dir must be mounted at /tmp/tapestry-data/ and there must be be two sub directories under it:
schema: All SQL files inside this dir will be executed against the database server in lexicographical order to setup a temporary test database.
tests: All SQL files inside this dir will be considered as tests and specified as arguments to the pg_prove command.
Once such a local directory is created, you can run the tests as follows,
podman run -it \\\n --rm \\\n --network podman \\\n -v ~/tapestry-data/:/tmp/tapestry-data/ \\\n --env PGPASSWORD=secret \\\n --env PGHOST=$(podman container inspect -f '{{.NetworkSettings.IPAddress}}' taptestbed) \\\n tapestry-testbed:latest \\\n run-tests -c -d temptestdb\nIn the above command, temptestdb is the name of the db that will be created by the run-tests script. If your schema files themselves take care of creating the db, then you can specify that as the name and omit the -c flag.
To know more about the usage of run-tests script, run,
podman run -it --rm tapestry-testbed:latest run-tests --help\nThis tutorial is to help you get started with tapestry. It's assumed that the following software is installed on your system:
tapestrypg_formata working installation of PostgreSQL (official docs) pgTAP and pg_prove "},{"location":"user-guide/getting-started/#sample-database","title":"Sample database","text":"For this tutorial, we'll use the chinook sample database. Download and import it as follows,
wget -P /tmp/ https://github.com/lerocha/chinook-database/releases/download/v1.4.5/Chinook_PostgreSql_SerialPKs.sql\ncreatedb chinook\npsql -d chinook -f /tmp/Chinook_PostgreSql_SerialPKs.sql\nWe'll start by running the tapestry init command, which will create the directory structure and also write a bare minimum manifest file for us. In a real project, you'd run this command from within the main project directory, so that the files can be committed to the same repo. But for this tutorial, you can run it from any suitable location e.g. the home dir ~/
cd ~/\ntapestry init chinook\nThis will create a directory named chinook with following structure,
$ cd chinook\n$ tree -a --charset=ascii .\n.\n|-- .pg_format\n| `-- config\n|-- tapestry.toml\n`-- templates\n |-- queries\n `-- tests\nLet's look at the tapestry.toml manifest file that has been created (I've stripped out some comments for conciseness)
$ cat tapestry.toml\nplaceholder = \"posargs\"\n\nquery_templates_dir = \"templates/queries\"\ntest_templates_dir = \"templates/tests\"\n\nqueries_output_dir = \"output/queries\"\ntests_output_dir = \"output/tests\"\n\n[formatter.pgFormatter]\nexec_path = \"pg_format\"\nconf_path = \"./.pg_format/config\"\n\n[name_tagger]\nstyle = \"kebab-case\"\n\n# [[query_templates]]\n\n# [[queries]]\n\n# [[test_templates]]\nplaceholder defines the style of generated queries. Default is posargs (positional arguments) which will generate queries with $1, $2 etc as the placeholders. These are suitable for defining prepared statements.
Then there are four toml keys for defining directories,
query_templates_dir is where the query templates will be located
test_templates_dir is where the test templates will be located
queries_output_dir is where the SQL files for queries will be generated
tests_output_dir is where the SQL files for pgTAP tests will be generated.
All directory paths are relative to the manifest file.
You may have noticed that the init command created only the templates dirs. output dirs will be created when tapestry render is called for the first time.
The init command has also created a pg_format config file for us. This is because it found the pg_format executable on PATH. Refer to the pg_format section for more details.
Finally, name_tagger has been configured with kebab-case as the style.
"},{"location":"user-guide/getting-started/#adding-a-query_template-to-generate-queries","title":"Adding a query_template to generate queries","text":"Now we'll define a query template. But before that, you might want to get yourself familiar with the chinook database's schema.
Suppose we have an imaginary application built on top of the chinook database in which the following queries need to be run,
list all artists with their longest songs
list top 10 artists having longest songs
list top 5 artists having longest songs, and of a specific genre
As you can see, we'd need different queries for each of the 3 requirements, but all have a common logic of finding longest songs per artist. Using Jinja syntax, we can write a query template that covers all 3 cases as follows,
SELECT\n ar.artist_id,\n ar.name,\n max(milliseconds) * interval '1 ms' AS duration\nFROM\n track t\n INNER JOIN album al USING (album_id)\n INNER JOIN artist ar USING (artist_id)\n{% if cond__genre %}\n INNER JOIN genre g USING (genre_id)\n WHERE\n g.name = {{ placeholder('genre') }}\n{% endif %}\nGROUP BY\n ar.artist_id\nORDER BY\n-- Descending order because we want the top artists\n duration DESC\n{% if cond__limit %}\n LIMIT {{ placeholder('limit') }}\n{% endif %}\n;\nWe've used some custom Jinja variables for selectively including parts of SQL in the query. These need to be prefixed with cond__ and have to be defined in the manifest file (we'll come to that a bit later).
We have also used the custom Jinja function placeholder which takes one arg and expands to a placeholder in the actual query. This will be clear once we render the queries.
Let's save the above query template to the file templates/queries/artists_long_songs.sql.j2.
And now we'll proceed to defining the query_template and the queries that it can generate in the manifest file. Edit the tapestry.toml file by appending the following lines to it.
[[query_templates]]\npath = \"artists_long_songs.sql.j2\"\nall_conds = [ \"genre\", \"limit\" ]\nTo define a query_template we need to specify 2 keys:
path i.e. where the template file is located relative to the query_templates_dir defined earlier in the manifest. path itself is considered as the unique identifier for the query template.
all_conds is a set of values that will be converted to cond__ Jinja variables. In this case it means there are two cond__ Jinja templates supported by the template - cond__genre and cond__limit. Note that they are defined in the manifest without the cond__ suffix.
We can now define three different queries that map to the same query_template
[[queries]]\nid = \"artists_long_songs\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = []\n\n[[queries]]\nid = \"artists_long_songs*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"limit\" ]\n\n[[queries]]\nid = \"artists_long_songs@genre*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"genre\", \"limit\" ]\nTo define a query, we need to specify 3 keys,
id is an identifier for the query. Notice that we're following a naming convention by using special chars @ and *. Read more about Query naming conventions.
template is reference to the query template that we defined earlier.
conds is a subset of the all_conds key that's defined for the linked query template. In the context of this query, only the corresponding cond__ Jinja variables will have the value true, and the rest of them will be false.
We've defined three queries that use the same template. In the first query, both the conds that the template supports i.e. \"genre\" and \"limit\" are false. In the second query, \"limit\" is true but \"genre\" is false. In the third query, both \"genre\" and \"limit\" are true. Queries will be rendered based on these variables and the {% if cond__.. %} expressions in the template.
Don't worry if all this doesn't make much sense at this point. Things will be clear when we'll run tapestry render shortly.
"},{"location":"user-guide/getting-started/#rendering","title":"Rendering","text":"Now let's run the tapestry render command.
tapestry render\nAnd you'll notice some files created in our directory.
$ tree -a --charset=ascii .\n.\n|-- .pg_format\n| `-- config\n|-- output\n| |-- queries\n| | |-- artists_long_songs-genre-limit.sql\n| | |-- artists_long_songs-limit.sql\n| | `-- artists_long_songs.sql\n| `-- tests\n|-- tapestry.toml\n`-- templates\n |-- queries\n | `-- artists_long_songs.sql.j2\n `-- tests\nHere is what the generated output files look like:
artists_long_songs.sqlartists_long_songs-limit.sqlartists_long_songs-genre-limit.sql -- name: artists-long-songs\nSELECT\n ar.artist_id,\n ar.name,\n max(milliseconds) * interval '1 ms' AS duration\nFROM\n track t\n INNER JOIN album al USING (album_id)\n INNER JOIN artist ar USING (artist_id)\nGROUP BY\n ar.artist_id\nORDER BY\n -- Descending order because we want the top artists\n duration DESC;\n-- name: artists-long-songs-limit\nSELECT\n ar.artist_id,\n ar.name,\n max(milliseconds) * interval '1 ms' AS duration\nFROM\n track t\n INNER JOIN album al USING (album_id)\n INNER JOIN artist ar USING (artist_id)\nGROUP BY\n ar.artist_id\nORDER BY\n -- Descending order because we want the top artists\n duration DESC\nLIMIT $1;\n-- name: artists-long-songs-genre-limit\nSELECT\n ar.artist_id,\n ar.name,\n max(milliseconds) * interval '1 ms' AS duration\nFROM\n track t\n INNER JOIN album al USING (album_id)\n INNER JOIN artist ar USING (artist_id)\n INNER JOIN genre g USING (genre_id)\nWHERE\n g.name = $1\nGROUP BY\n ar.artist_id\nORDER BY\n -- Descending order because we want the top artists\n duration DESC\nLIMIT $2;\nThe SQL comments before the SQL with name of the query are generated by name_tagger added to the manifest. Learn more about Name tagging.
Also notice that the output SQL is formatted by pg_format.
"},{"location":"user-guide/getting-started/#adding-a-test_template","title":"Adding a test_template","text":"Now that we've defined and rendered queries, let's add test_template. Again there are two changes required - an entry in the manifest file and the Jinja template itself.
Add the following lines to the manifest file.
[[test_templates]]\nquery = \"artists_long_songs@genre*limit\"\npath = \"artists_long_songs-genre-limit_test.sql.j2\"\nHere we're referencing the query artists_long_songs@genre*limit hence this test is meant for that query. The path key points to a test template file that we need to create. So let's create the file templates/tests/artists_long_songs-genre-limit_test.sql.j2 with the following contents:
PREPARE artists_long_songs(varchar, int) AS\n{{ prepared_statement }};\n\nBEGIN;\nSELECT\n plan (1);\n\n-- start(noformat)\n-- Run the tests.\nSELECT results_eq(\n 'EXECUTE artists_long_songs(''Rock'', 10)',\n $$VALUES\n (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n (58, 'Deep Purple'::varchar, '00:19:56.094'::interval),\n (59, 'Santana'::varchar, '00:17:50.027'::interval),\n (136, 'Terry Bozzio, Tony Levin & Steve Stevens'::varchar, '00:14:40.64'::interval),\n (140, 'The Doors'::varchar, '00:11:41.831'::interval),\n (90, 'Iron Maiden'::varchar, '00:11:18.008'::interval),\n (23, 'Frank Zappa & Captain Beefheart'::varchar, '00:11:17.694'::interval),\n (128, 'Rush'::varchar, '00:11:07.428'::interval),\n (76, 'Creedence Clearwater Revival'::varchar, '00:11:04.894'::interval),\n (92, 'Jamiroquai'::varchar, '00:10:16.829'::interval)\n $$,\n 'Verify return value'\n);\n-- Finish the tests and clean up.\n-- end(noformat)\n\nSELECT\n *\nFROM\n finish ();\nROLLBACK;\nThe test syntax is SQL only but with some additional functions installed by pgTAP. If you are not familiar with pgTAP you can go through it's documentation. But for this tutorial, it's sufficient to understand that the {{ prepared_statement }} Jinja variable is made available to this template, and when it's rendered it will expand to the actual query.
Let's run the render command again.
tapestry render\nAnd now you should see the pgTAP test file created at output/tests/artists_long_songs-genre-limit_test.sql.
Note
Here the file stem of the test template path itself was used as the output file name. But it's also possible to explicitly specify it in the manifest file (see output in test_templates docs).
This is how the rendered test file looks like,
PREPARE artists_long_songs (varchar, int) AS\nSELECT\n ar.artist_id,\n ar.name,\n max(milliseconds) * interval '1 ms' AS duration\nFROM\n track t\n INNER JOIN album al USING (album_id)\n INNER JOIN artist ar USING (artist_id)\n INNER JOIN genre g USING (genre_id)\nWHERE\n g.name = $1\nGROUP BY\n ar.artist_id\nORDER BY\n -- Descending order because we want the top artists\n duration DESC\nLIMIT $2;\n\nBEGIN;\nSELECT\n plan (1);\n-- start(noformat)\n-- Run the tests.\nSELECT results_eq(\n 'EXECUTE artists_long_songs(''Rock'', 10)',\n $$VALUES\n (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n (58, 'Deep Purple'::varchar, '00:19:56.094'::interval),\n (59, 'Santana'::varchar, '00:17:50.027'::interval),\n (136, 'Terry Bozzio, Tony Levin & Steve Stevens'::varchar, '00:14:40.64'::interval),\n (140, 'The Doors'::varchar, '00:11:41.831'::interval),\n (90, 'Iron Maiden'::varchar, '00:11:18.008'::interval),\n (23, 'Frank Zappa & Captain Beefheart'::varchar, '00:11:17.694'::interval),\n (128, 'Rush'::varchar, '00:11:07.428'::interval),\n (76, 'Creedence Clearwater Revival'::varchar, '00:11:04.894'::interval),\n (92, 'Jamiroquai'::varchar, '00:10:16.829'::interval)\n $$,\n 'Verify return value'\n);\n-- Finish the tests and clean up.\n-- end(noformat)\nSELECT\n *\nFROM\n finish ();\nROLLBACK;\nAssuming that all the above mentioned prerequisites are installed, you can run the tests as follows,
sudo -u postgres pg_prove -d chinook --verbose output/tests/*.sql\nIf all goes well, the tests should pass and you should see output similar to,
1..1\nok 1 - Verify return value\nok\nAll tests successful.\nFiles=1, Tests=1, 0 wallclock secs ( 0.03 usr 0.01 sys + 0.01 cusr 0.00 csys = 0.05 CPU)\nResult: PASS\nIf you've reached this far, you should now have a basic understanding of what tapestry is and how to use it. Next, it'd be a good idea to understand the manifest file in more detail.
Note
The chinook example discussed in this tutorial can also be found in the github repo under the examples/chinook directory (there are a few more tests included for reference).
"},{"location":"user-guide/install/","title":"Installation","text":"Until tapestry is published to crates.io, you can install it directly from github,
cargo install --git https://github.com/naiquevin/tapestry.git\nTapestry depends on pg_format for formatting the generated SQL files. It's not a hard requirement but recommended.
On MacOS, it can be installed using homebrew,
brew install pgformatter\nNote that you need to install pg_format on the machine where you'd be rendering the SQL files using tapestry e.g. on your workstation and/or the build server.
"},{"location":"user-guide/install/#dependencies-for-running-tests","title":"Dependencies for running tests","text":"If you are using tapestry to render tests (which you should, because that's what the tool is meant for!) then you'd also need the pgTAP extension and the pg_prove command line tool.
pgTAP can be easily built from source. Refer to the instructions here.
You can install pg_prove from a CPAN distribution as follows:
sudo cpan TAP::Parser::SourceHandler::pgTAP\nRefer to the pgTAP installation guide for more details.
As tapestry is a postgres specific tool, it goes without saying that you'd need a working installation of postgres to be able to run the tests. Please refer to the official documentation for that.
"},{"location":"user-guide/layouts/","title":"Layouts","text":"Tapestry lets you control the layout of the query files i.e. how the generated SQL is organized in files. It supports two ways at present:
one-file-one-query: Each SQL query will be written to a separate fileone-file-all-queries: All SQL queries will be written to a single file To configure this, you need to specify the query_output_layout key in the manifest. The default option if not specified is one-file-one-query.
"},{"location":"user-guide/layouts/#layout-and-queriesoutput-field","title":"Layout and queries[].output field","text":"Users may specify output field for every query, which is the path where the generated SQL output will be written. If output is not specified, it's value is derived from the query id. This works well for the one-file-one-query layout.
When the layout is one-file-all-queries, it's expected that the output field of all queries must be the same. Otherwise the manifest fails to validate. To avoid duplication, a related setting query_output_file is provided.
If layout = one-file-all-queries, it's recommended to set query_output_file and omit the output field for individual queries.
If layout = one-file-one-query, then you must not set query_output_file. Whether or not to set the output field for individual queries is up to you.
"},{"location":"user-guide/manifest/","title":"Manifest","text":"Every tapestry \"project\" has a tapestry.toml file which is called the manifest. It is in TOML format and serves the dual purpose of configuration as well as a registry of the following entities:
query_templatesqueriestest_templates The various sections or top level TOML keys are described in detail below. When going through this doc, you may find it helpful to refer to the chinook example in the github repo. If you haven't checked the Getting started section, it's recommended to read it first.
"},{"location":"user-guide/manifest/#placeholder","title":"placeholder","text":"The placeholder key is for configuring the style of the placeholder syntax for parameters i.e. the values values that are substituted into the statement when it is executed.
Two options are supported:
"},{"location":"user-guide/manifest/#posargs","title":"posargs","text":"posargs is short for positional arguments. The placeholders refer to the parameters by positions e.g. $1, $2 etc. This is the same syntax that's used for defining prepared statements or SQL functions in postgres.
This option is suitable when your db driver or SQL library accepts queries in prepared statements syntax. E.g. sqlx (Rust).
Default: The manifest file auto-generated upon running the tapestry init command will have,
placeholder = posargs\nWhen placeholder=variables placeholders are added in the rendered query using the variable substitution syntax of postgres. The variable name in the query is preceded with colon e.g. :email, :department
This option is suitable when your db driver or SQL library accepts queries with variables. E.g. yesql, hugsql (Clojure), aiosql (Python)
Examples
Templateplaceholder = posargsplaceholder = variables SELECT\n *\nFROM\n employees\nWHERE\n email = {{ placeholder('email') }}\n AND department = {{ placeholder('department') }};\nSELECT\n *\nFROM\n employees\nWHERE\n email = $1\n AND department = $2;\nSELECT\n *\nFROM\n employees\nWHERE\n email = :email\n AND department = :department;\nNote
Note that the prepared_statement Jinja variable available in test templates will always have posargs based placeholders even if the placeholder config in manifest file is set to variables. That's the reason the Jinja var is named prepared_statement.
"},{"location":"user-guide/manifest/#query_templates_dir","title":"query_templates_dir","text":"Path where the query templates are located. The path is always relative to the manifest file.
Default: The manifest file auto-generated upon running the tapestry init command will have,
query_templates_dir = \"templates/queries\"\nPath where the query templates are located. The path is always relative to the manifest file.
Default: The manifest file auto-generated upon running the tapestry init command will have,
test_templates_dir = \"templates/tests\"\nPath to the output dir for the rendered queries. This path also needs to be defined relative to the manifest file.
Default: The manifest file auto-generated upon running the tapestry init command will have,
queries_output_dir = \"output/queries\"\nA common use case to modify this config would be to store SQL files in a directory outside of the tapestry \"project\" dir, so that only the SQL files in that directory can be packaged into the build artifact. There's no need to include the query/test template and the pgTAP test files in the build artifact. E.g.
queries_output_dir = \"../sql_queries\"\nPath to the output dir for rendered pgTAP tests. The path is always relative to the manifest file.
Default: The manifest file auto-generated upon running the tapestry init command will have,
tests_output_dir = \"output/tests\"\nLayout to be used for the generated query files. The two options are:
one-file-one-query: Each SQL query will be written to a separate file
one-file-all-queries: All SQL queries will be written to a single file
It's optional. The default value is one-file-one-query.
"},{"location":"user-guide/manifest/#query_output_file","title":"query_output_file","text":"query_output_file is optional but it's use is valid only when the layout is one-file-all-queries. It basically saves the user from having to define the same output for all queries.
Refer to the Layouts section of the user guide for more info.
"},{"location":"user-guide/manifest/#formatterpgformatter","title":"formatter.pgFormatter","text":"This section is for configuring the pg_format tool that tapestry uses for formatting the rendered SQL files.
There two config params under this section:
"},{"location":"user-guide/manifest/#exec_path","title":"exec_path","text":"Location of the pg_format executable.
"},{"location":"user-guide/manifest/#conf_path","title":"conf_path","text":"Path to the pg_format config file. It can be used for configuring the behavior of pg_format when it gets executed on rendered SQL. As with all paths that we've seen so far, this one is also relative to the manifest file.
Example
[formatter.pgFormatter]\nexec_path = \"pg_format\"\nconf_path = \"./.pg_format/config\"\nAs mentioned in the installation guide, pg_format is not a mandatory requirement but it's recommended.
Upon running the tapestry init command, this section will be included in the auto-generated manifest file only if the executable pg_format is found on PATH. In that case, a default pg_format config file will also be created.
To read more about configuring pg_format in the context of tapestry, refer to the pg_format section of the docs.
"},{"location":"user-guide/manifest/#name_tagger","title":"name_tagger","text":"name_tagger is a TOML table, which if present in the manifest will cause the generated SQL queries to be name tagged.
"},{"location":"user-guide/manifest/#style","title":"style","text":"name_tagger.style can be used to control how name tags will be derived from query id. The two options are:
kebab-casesnake_caseexact Any special characters in the query id will be replaced with an appropriate character based on the above option \u2014 hyphen in case of kebab-case and underscore in case of snake_case. The third option exact is different in the sense that the query id will be used as it is as the name tag.
Example:
[name_tagger]\nstyle = \"kebab-case\"\nNote
Note the autological naming of options kebab-case (with a hyphen) v/s snake_case (with an underscore).
"},{"location":"user-guide/manifest/#query_templates","title":"query_templates","text":"query_templates is an array of tables in TOML parlance. So it needs to defined with double square brackets and can be specified multiple times in the manifest file.
For every query template, there are two keys to be defined:
"},{"location":"user-guide/manifest/#path","title":"path","text":"It's where the Jinja template file is located relative to the query_templates_dir defined earlier in the manifest. path itself is considered as the unique identifier for the query template.
Use .j2 extension as the convention for the query template file.
"},{"location":"user-guide/manifest/#all_conds","title":"all_conds","text":"It's a set of values that will be converted to cond__ Jinja variables that can be referenced inside the template. Note that they are defined in the manifest without the cond__ suffix.
This field is optional. If not specified, an empty set is considered as the default.
For documentation on how to write a query_template, refer to Writing query templates
Example:
[[query_templates]]\npath = \"artists_long_songs.sql.j2\"\nall_conds = [ \"genre\", \"limit\" ]\n\n[[query_templates]]\npath = \"songs_formats.sql.j2\"\nall_conds = [ \"artist\", \"file_format\", \"album_name\" ]\nNote
When all_conds is not specified, it essentially means that the query is a valid SQL statement and not a Jinja template. Then why define it as a template? The answer to that is \u2014 so that it can be embedded in tests.
"},{"location":"user-guide/manifest/#queries","title":"queries","text":"queries is an array of tables in TOML parlance. So it needs to defined with double square brackets and can be specified multiple times in the manifest file.
A query can be defined using the following keys,
"},{"location":"user-guide/manifest/#id","title":"id","text":"id is an identifier for the query.
"},{"location":"user-guide/manifest/#template","title":"template","text":"template is a reference to a query_template defined previously in the manifest.
"},{"location":"user-guide/manifest/#conds","title":"conds","text":"conds is a subset of the all_conds key that's defined for the linked query template. It's an optional and if not specified, an empty set will be considered by default.
"},{"location":"user-guide/manifest/#output","title":"output","text":"output is the path to the output file where the SQL query will be rendered. It must be relative to the queries_output_dir config.
It's optional to specify the output. If not specified, the filename of the output file will be derived by slugifying the id. This property allows us to use certain Naming conventions for giving suitable and consistent names to the queries.
Example:
[[queries]]\nid = \"artists_long_songs@genre*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"genre\", \"limit\" ]\nThe derived value of output for the above will be artists_long_songs-genre-limit.sql.
"},{"location":"user-guide/manifest/#name_tag","title":"name_tag","text":"name_tag can be optionally set to specify a custom name tag for the query. Name tags are prefixed to the SQL queries as comments and they are used by SQL loading libraries such as yesql, aiosql etc. Read more about in Name tagging queries.
Note
A query will be tagged with the specified name_tag only if name_tagger is set.
"},{"location":"user-guide/manifest/#test_templates","title":"test_templates","text":"test_templates is an array of tables in TOML parlance. So it needs to defined with double square brackets and can be specified multiple times in the manifest file.
A test_template can be defined using the following keys,
"},{"location":"user-guide/manifest/#query","title":"query","text":"query is a reference to query defined in the manifest.
"},{"location":"user-guide/manifest/#path_1","title":"path","text":"path is the path to the jinja template for the pgTAP test. It must be relative to the test_templates_dir.
Use .j2 extension as the convention for the test template file.
"},{"location":"user-guide/manifest/#output_1","title":"output","text":"output is the path where the pgTAP test file will be rendered. It must be relative to the tests_output_dir.
Specifying output for test_templates is optional. If not specified, it will be derived from the file stem of path i.e. by removing the .j2 extension.
For detailed documentation on how to write a test_template, refer to Writing test templates
"},{"location":"user-guide/naming-conventions/","title":"Query naming conventions","text":"One of the problems that I have encountered when using SQL loading libraries such as yesql and aiosql is that the queries defined in SQL files need to be given unique names. Often, one ends up writing a group of queries that are mostly similar to each other and differ only slightly. Giving unique and consistent names to each query can become tricky.
A tool like tapestry cannot automatically give a name to a query. However, since the queries are listed in the manifest file, we can partly address the problem with the use of naming conventions.
These naming conventions involve clever use of special characters such as @, +, & and *. Let's look at some examples from examples/chinook dir.
[[queries]]\nid = \"artists_long_songs@genre*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"genre\", \"limit\" ]\n\n[[queries]]\nid = \"songs_formats@artist&file_format+album\"\ntemplate = \"songs_formats.sql.j2\"\nconds = [ \"artist\", \"album_name\", \"file_format\" ]\nIn the above queries, the id is defined using an alphanumeric prefix (artists_long_songs and songs_formats) followed by suffix that's an encoding of the conditional Jinja variables relevant to the query.
The convention is as follows,
@ precedes \"cond\" vars used for conditionally including a filter i.e. a WHERE clause. + precedes \"cond\" vars used for conditionally returning a column * precedes \"cond\" vars used for conditionally including any other part of the query e.g. LIMIT, ORDER BY etc. & is used as a delimiter between two \"cond\" vars of same type e.g. @artist&file_format. The name of the output file for the SQL query will be generated by slugifying the id i.e. by replacing the above special characters with hyphen (-). In case of the above two queries, the output file names will be artists_long_songs-genre-limit.sql and songs_formats-artist-file_format-album.sql respectively.
Note that these naming conventions are only recommended by tapestry and are not mandatory.
"},{"location":"user-guide/pg-format/","title":"Configuring pg_format","text":"tapestry relies on pg_format for formatting the rendered SQL files. This makes sure that,
the rendered SQL files have consistent indentation you don't need to worry about SQL indentation when writing Jinja templates However, pg_format is not a hard requirement for tapestry. If pg_format is not installed on your system at the time of running tapestry init command, the formatter.pgFormatter section will not be added in the auto-generated manifest file.
The behavior of pg_format tool in the context of tapestry can be configured by adding a config file. The sample config file in the pg_format github repo can be used for reference.
The default config file generated by tapestry init command is located at .pg_format/config (relative to the manifest file) and looks like this,
# Lines between markers 'start(noformat)' and 'end(noformat)' will not\n# be formatted. If you want to customize the markers, you may do so by\n# modifying this parameter.\nplaceholder=start\\(noformat\\).+end\\(noformat\\)\n\n# Add a list of function to be formatted as PG internal\n# functions. Paths relative to the 'tapestry.toml' file will also work\n#extra-function=./.pg_format/functions.lst\n\n# Add a list of keywords to be formatted as PG internal keywords.\n# Paths relative to the 'tapestry.toml' file will also work\n#extra-keyword=./.pg_format/keywords.lst\n\n# -- DANGER ZONE --\n#\n# Please donot change the following config parameters. Tapestry may\n# not work otherwise.\nmultiline=1\nformat=text\noutput=\nAs you can see, the generated file itself is well documented.
"},{"location":"user-guide/pg-format/#disallowed-configuration","title":"Disallowed configuration","text":"In the context of tapestry, some pg_format config params are disallowed (or they need to configured only in a certain way) for proper functioning of tapestry. These are explicitly defined with the intended value in the config file and annotated with DANGER ZONE warning in the comments. These must not be changed.
"},{"location":"user-guide/pg-format/#selectively-opting-out-of-sql-formatting","title":"Selectively opting out of SQL formatting","text":"A commonly faced problem with formatting pgTAP tests using pg_format is that hard coded expected values get formatted in a way that could make the test case unreadable for humans.
Example: Consider the following pgTAP test case written in a test template file,
SELECT results_eq(\n 'EXECUTE artists_long_songs(''Rock'', 2)',\n $$VALUES\n (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n (58, 'Deep Purple'::varchar, '00:19:56.094'::interval)\n $$,\n 'Verify return value'\n);\nBy default pg_format would format the above SQL snippet as follows,
SELECT\n results_eq ('EXECUTE artists_long_songs(''Rock'', 2)', $$\n VALUES (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval), (58, 'Deep Purple'::varchar, '00:19:56.094'::interval) $$, 'Verify return value');\nTo retain the readability, we need to preserve the user's custom indentation. This is where the placeholder config param of pg_format is useful
Note
pg_format's placeholder config is not to be confused with placeholder config key in tapestry's manifest.
This can be done by adding noformat markers before and after the snippet.
-- start(noformat)\nSELECT results_eq(\n 'EXECUTE artists_long_songs(''Rock'', 2)',\n $$VALUES\n (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n (58, 'Deep Purple'::varchar, '00:19:56.094'::interval)\n $$,\n 'Verify return value'\n);\n-- end(noformat)\nIf you want to customize the markers for whatever reason, you can modify the placeholder param in the pg_format config file.
"},{"location":"user-guide/query-tags/","title":"Query tags","text":""},{"location":"user-guide/query-tags/#name-tagging-queries","title":"Name tagging queries","text":"Typically, the output query files rendered by tapestry are intended to be used by libraries such as yesql, aiosql etc. These libraries require the queries to be \"name-tagged\". Tagging is done by simply adding a comment before the query as follows,
-- name: my-query\n-- A simple query\nSELECT 1;\nThis way, these libraries can map the queries with the functions that it generates in code. These functions wraps around the database client/driver code and provides an easy interface for the user.
The following example is taken from yesql's README:
-- name: users-by-country\nSELECT *\nFROM users\nWHERE country_code = :country_code\n...and then read that file to turn it into a regular Clojure function:
(defqueries \"some/where/users_by_country.sql\"\n {:connection db-spec})\n\n;;; A function with the name `users-by-country` has been created.\n;;; Let's use it:\n(users-by-country {:country_code \"GB\"})\n;=> ({:name \"Kris\" :country_code \"GB\" ...} ...)\nTapestry does support name tagging of queries, but it's disabled by default. To enable it, just add the following lines in the manifest,
[name_tagger]\nstyle = \"kebab-case\"\nThis will result in name tags added to queries. The name tags are derived from the query ids. The style setting allows us to control how the id should be slugified to derive the name tag. For e.g. kebab-case will cause all non-alphanumeric characters in the id to be replaced by hyphens.
The other options for style are snake_case and exact.
"},{"location":"user-guide/query-tags/#custom-name-tags","title":"Custom name tags","text":"The above method derives name tags from query ids. But yesql and aiosql sometimes require the query names to be suffixed with specific characters to indicate specific operations. Example: In yesql, the name tags for INSERT/UPDATE/DELETE statements need to be suffixed with !.
-- name: save-person!\nUPDATE person\n SET name = :name\n WHERE id = :id\nThere are two ways to achieve this:
Specify exact as the name_tagger.style. Then the query id itself to be used as the name tag (as it is).
Specify the optional queries[].name_tag field when defining the queries.
While it may seem like the first approach involves less effort, the downside is that we'd be giving up on the Naming conventions that tapestry recommends.
Libraries such as yesql and aiosql usually don't allow special characters in the name tags as they use them to generate functions in code. So yesql recommends the name tags to be in kebab-case as Clojure functions follow that convention, whereas aiosql needs the name tags to be in snake_case as that's the requirement and also the convention in Python.
"},{"location":"user-guide/query-templates/","title":"Writing query templates","text":"Query templates are Jinja template files. One query template can be used for generating multiple SQL queries.
Often, an application needs to issue mostly similiar (or slightly different) queries to the db based on user input. Some examples:
two queries that are exactly similar, except that one returns all columns i.e. * whereas the other returns only selected rows two queries that are exactly the same, except that one has a limit multiple similar queries but different combination of WHERE clauses Using Jinja templates, it's possible to write a single query template that can render multiple SQL queries. This is possible with a combination of Jinja variables and {% if .. %}...{% endif %} blocks. This is pretty much the main idea behind query templates.
"},{"location":"user-guide/query-templates/#cond-variables","title":"\"cond\" variables","text":"Query templates need to be defined in the manifest where we specify all_conds which is a set of \"cond\" vars that the template supports.
Let's look at a query template from the chinook example distributed with the github repo.
SELECT\n track.name as title,\n artist.name as artist_name,\n {% if cond__album_name %}\n album.title as album_name,\n {% endif %}\n media_type.name as file_format\nFROM\n album\n JOIN artist USING (artist_id)\n LEFT JOIN track USING (album_id)\n JOIN media_type USING (media_type_id)\n\n{% if cond__artist or cond__file_format %}\n WHERE\n {% set num_conds = 0 %}\n {% if cond__artist %}\n artist.name = {{ placeholder('artist') }}\n {% set num_conds = num_conds + 1 %}\n {% endif %}\n\n {% if cond__file_format %}\n {% if num_conds > 0 %}\n AND\n {% endif %}\n media_type.name = {{ placeholder('file_format') }}\n {% set num_conds = num_conds + 1 %}\n {% endif %}\n{% endif %}\n;\nThe entry in the manifest file for the above query_template is,
[[query_templates]]\npath = \"songs_formats.sql.j2\"\nall_conds = [ \"artist\", \"file_format\", \"album_name\" ]\nBecause of the 3 all_conds defined in the manifest file, we have the following Jinja variables available inside the Jinja template.
cond__artistcond__file_formatcond__album_name The cond__artist and cond__file_format vars are used for conditionally including WHERE clauses. Because we want to add the WHERE clause only if either of the two vars are true, and because we want to add the AND operator only if both are true, nested if blocks are used and a temp \"counter\" variable num_conds is defined i.e. it's assigned to 0 and then incremented by 1 if the cond__artist var is true.
The third variable cond__album_name is used for conditionally including a column in the returned result.
"},{"location":"user-guide/query-templates/#query","title":"Query","text":"Now let's look at how a query associated with this template is defined in the manifest.
[[queries]]\nid = \"songs_formats@artist+album\"\ntemplate = \"songs_formats.sql.j2\"\nconds = [ \"artist\", \"album_name\" ]\noutput = \"songs_formats__artist__album.sql\"\nIn this query, only 2 of the 3 \"cond\" variables will be true.
As a total of 3 all_conds values are supported by the query template, 8 different queries can be generated from it using different subsets of all_conds.
[ ]\n[ \"artists\" ]\n[ \"artists\", \"file_format\" ]\n[ \"artists\", \"album_name\" ]\n[ \"file_format\" ]\n[ \"file_format\", \"album_name\" ]\n[ \"album_name\" ]\n[ \"artist\", \"file_format\", \"album_name\" ]\nJust like query_templates, test_templates are also Jinja template files. But while one query template could be used to generate several queries, one test template can be used to generate only one pgTAP test file.
However, many test templates can be associated with a single query. In other words, if multiple pgTAP test suites are to be written for the same query, that's possible.
The test syntax is SQL only but with some additional functions installed by pgTAP. If you are not familiar with pgTAP you can go through it's documentation. Important thing to note is that the Jinja variable {{ prepared_statement }} is made available to every test template, and at the time of rendering, it will expand to the actual query.
Let's look at a templates from the chinook example.
Refer to the test template songs_formats-afa_test.sql.j2. The first few lines are:
PREPARE song_formats (varchar, varchar) AS\n{{ prepared_statement }};\nHere we're using the prepared_statement Jinja variable to create a prepared statement for the user session. The name of the prepared statement is song_formats and it takes two positional args, both of type varchar.
Later in the same file, the prepared statement is executed as part of a pgTAP test case,
SELECT results_eq(\n 'EXECUTE song_formats(''Iron Maiden'', ''Protected AAC audio file'')',\n $$VALUES\n ...\n ...\n $$,\n 'Verify return value'\n);\nCheck the songs_formats-afa_test.sql output file to see how the actual test file looks like.
Note
Note that the SQL query that prepared_statement Jinja var expands to will always have posargs based placeholders, even if the placeholder config in manifest file is set to variables. That's the reason why the Jinja var is named prepared_statement
"},{"location":"user-guide/test-templates/#function-instead-of-ps","title":"Function instead of PS","text":"Sometimes it's tedious to test for result sets returned by the query. In such cases, it helps to manipulate the result returned by the query and compare a derived property. E.g. If a query results too many rows, it's easier to compare the count than the actual values in the rows.
One limitation of prepared statements and the EXECUTE syntax for executing them is that it's not sub-query friendly i.e. it's not possible to execute a prepared statement as part of another query.
The following is NOT valid SQL
SELECT\n count(*)\nFROM (EXECUTE song_formats ('Iron Maiden', 'Protected AAC audio file'));\nIn such cases, we can define a SQL function using the same prepared_statement Jinja variable.
An example of this can be found in the chinook example - all_artists_long_songs_test.sql.j2
CREATE OR REPLACE FUNCTION all_artists_long_songs ()\nRETURNS SETOF record\nAS $$\n{{ prepared_statement }}\n$$ LANGUAGE sql;\n\nBEGIN;\nSELECT\n plan (1);\n\n-- start(noformat)\n-- Run the tests.\nSELECT is(count(*), 204::bigint) from all_artists_long_songs() AS (artist_id int, name text, duration interval);\n-- Finish the tests and clean up.\n-- end(noformat)\n\nSELECT\n *\nFROM\n finish ();\nROLLBACK;\nWhen it comes to automated tests, It's a very common requirement to setup some test data to be able to write test cases. pgTAP tests are not any different. In case of pgTAP one needs to create test data in the database.
Since pgTAP tests are just SQL files, test data creation can be done using SQL itself in the same file. Reusable setup code can also be extracted into SQL functions that can be created as part of importing the database schema.
The chinook directory doesn't include an example of this. But here's an example from one of my real projects that uses tapestry.
In my project, there are two entities categories and items (having tables of the same names) with one-to-many relationship i.e. one category can have multiple items.
In several pgTAP tests, a few categories and items need to be created. To do this, a function is defined as follows,
CREATE OR REPLACE FUNCTION tapestry.setup_category_n_items (cat_id varchar, item_idx_start integer, item_idx_end integer)\n RETURNS void\n AS $$\n INSERT INTO categories (id, name)\n VALUES (cat_id, initcap(replace(cat_id, '-', ' ')));\n INSERT INTO items (id, name, category_id)\n SELECT\n 'item-' || t AS id,\n 'Item ' || t AS name,\n cat_id AS category_id\n FROM\n generate_series(item_idx_start, item_idx_end) t;\n$$\nLANGUAGE sql;\nAnd then it's used in pgTAP tests like this,
...\n\nBEGIN;\nSELECT plan(1);\n\n-- Fixtures\n-- create 2 categories, 'cat-a' and 'cat-b' each having 5 items\nSELECT\n tapestry.setup_category_n_items ('cat-a', 1, 5);\nSELECT\n tapestry.setup_category_n_items ('cat-b', 6, 10);\n\n-- Test cases\n\n...\nTapestry is a framework for writing (postgres)SQL queries and (pgTAP) tests using Jinja templates.
Tapestry is written in Rust but it can be used with applications written in any programming language. It's purely a command line tool that renders Jinja templates into SQL files. How to load the resulting SQL code into memory and use it at runtime is entirely up to the application.
This approach of loading SQL from files is not new. There are existing libraries such as yesql, hugsql (Clojure), aiosql (Python) etc. that provide excellent abstractions for it. In absence of such a lib for the language of your choice, it shouldn't take more than a few lines of code to implement a simple file loader. In Rust apps, I simply use the include_str! macro.
One limitation is that tapestry can only be used with PostgreSQL, because of the tight coupling with pgTAP.
You may find this tool useful if,
you prefer direct SQL queries over ORMs or query builders to interact with RDBMS from application code
you are not averse to the idea of having (reasonable amount of) business logic inside SQL queries
In fact, if you have had concerns about point 2 i.e. having business logic in SQL queries, perhaps tapestry addresses some of those concerns. Learn more about the rationale behind this tool.
If you prefer a hands-on introduction, check the Getting started page.
"},{"location":"rationale/","title":"Rationale","text":""},{"location":"rationale/#problems-with-using-raw-sql-in-application-code","title":"Problems with using raw SQL in application code","text":"For many years, I've believed that,
it's a good idea to write raw SQL queries (safely) for interacting with an RDBMS from application code using libs such as yesql, aiosql etc.
it's ok to add reasonable amount of business logic in the SQL queries, rather than using SQL merely for data access.
Still, I've had concerns about using these ideas in practice, specially in serious projects.
"},{"location":"rationale/#unit-testing-sql-queries","title":"Unit testing SQL queries","text":"Typically, unit tests are written against application code. As more and more business logic gets moved out of the application and into SQL queries, the queries become longer and more complex. In contrast, the application code is reduced to just making db calls using the driver/client library. At this point, it makes more sense to test the queries than the application code.
Fortunately for PostgreSQL, we have the excellent pgTAP extension that makes it easy to write unit tests for raw queries. Just like the raw queries themselves, pgTAP tests are typically defined in SQL files. But since the query and the tests are in separate files, it's possible that one modifies the SQL query, but forgets to update the tests, and the tests could still pass!
How to ensure that the tests actually run the exact same query that's being run by the application?
"},{"location":"rationale/#maintenance-overhead-of-multiple-slightly-differing-queries","title":"Maintenance overhead of multiple, slightly differing queries","text":"An application often needs to issue similar queries but returning different set of columns or with different WHERE clauses based on user input. In such cases, a unique query needs to be written and maintained for every combination of the input parameters. This could result in multiple queries that differ only slightly. If some core part of the query needs a change, one needs to remember to update multiple SQL files.
Moreover, higher level abstractions (e.g. yesql etc.) usually cache queries in memory, so they require the queries to be given a name or an identifier. Since the queries differ only slightly, trying to give them unique names can be tricky.
"},{"location":"rationale/#how-tapestry-solves-it","title":"How tapestry solves it?","text":"Tapestry was built to specifically address the above problems and concerns. It does so by generating actual queries as well as pgTAP test files from Jinja templates, instead of having the user write raw SQL.
"},{"location":"rationale/#query-templates","title":"Query templates","text":" You write query templates instead of raw queries Multiple queries can be mapped to the same query template. Mapping is defined in the tapestry.toml manifest file. User defined Jinja variables can be used for conditionally adding or omitting parts of the query e.g. a WHERE condition or column to return. These Jinja vars are also defined in the manifest file. Thus, it's easy to generate and maintain multiple queries that are similar enough to be defined using a single query template. "},{"location":"rationale/#test-templates","title":"Test templates","text":" pgTAP tests are also written as Jinja templatesTest templates are mapped to queries, again in the manifest file. One query can be mapped to multiple test templates. When tapestry renders the final test file from a test template, a special Jinja variable {{ prepared_statement }} gets expanded to the actual query that the test template is mapped to. This way, the generated test SQL file is guaranteed to have the exact same query which is used by the application code. "},{"location":"rationale/#naming-conventions","title":"Naming conventions","text":"Tapestry suggests some conventions for naming queries consistently but they are not mandatory.
"},{"location":"user-guide/","title":"Overview","text":"Tapestry is built to address the peculiar concerns that I've had about using libraries such as yesql, aiosql and the likes. While I agree with the philosophy behind such libs\u2014that SQL code is better written as SQL directly rather than building it through ORMs, query builders or worse, by string interpolation or concatenation\u2014I've had some concerns about using the approach in practice.
To understand more about the problems and how tapestry addresses them, please check the Rationale page.
The general idea behind this tool is, instead of users writing raw SQL queries, have them write Jinja templates from which SQL queries as well as pgTAP tests can be generated.
Here is a high level overview of how you'd use tapestry in your project:
Create a directory inside your project where the templates will be located. The tapestry init command does this for you.
Add some information in the tapestry.toml manifest file:
Lists of query templates, queries and test templates along with the mappings between them Location of query templates and test templates (input files) Location of where the output files are to be created etc... Run tapestry render command to generate the SQL files, both for queries as well as tests.
Use a lib such as yesql, aiosql etc. to load the queries rendered by the previous step into the application runtime.
Use pg_prove to run the pgTAP tests, preferably as part of CD/CI. You'd need to implement some kind of automation for this. The github repo also includes a docker image that may help with this.
"},{"location":"user-guide/commands/","title":"Commands","text":"This page documents all commands in the tapestry CLI.
Note that for all commands except init, this tool will try to read the tapestry.toml manifest file in the current directory and will fail if it's not found. This implies that all tapestry commands except init must be executed from within the \"tapestry project\" root dir.
"},{"location":"user-guide/commands/#init","title":"init","text":"The init command can be used for scaffolding a new tapestry \"project\". It will create the directory structure and also write a bare minimum manifest file for us. In a real project, you'd run this command from within the main project directory, so that the files can be committed to the same repo. Example:
Running the following command,
tapestry init myproj\n.. will create the following directory structure
$ cd myproj\n$ tree -a --charset=ascii .\n.\n|-- .pg_format\n| `-- config\n|-- tapestry.toml\n`-- templates\n |-- queries\n `-- tests\nThe validate command checks and ensures that the manifest file is valid. Additionally it also verifies that the paths referenced in the manifest actually exist and are readable.
"},{"location":"user-guide/commands/#render","title":"render","text":"The render command renders all the template files into SQL files.
"},{"location":"user-guide/commands/#status","title":"status","text":"The status command can be used to preview the effect of running tapestry render command. It will list which output files will be added, modified or remain unchanged if the render command is run. This command will not actually write the output files.
Output of running tapestry status inside the examples/chinook directory:
$ tapestry status\nQuery: unchanged: output/queries/artists_long_songs.sql\n Test: unchanged: output/tests/all_artists_long_songs_count_test.sql\nQuery: unchanged: output/queries/artists_long_songs-limit.sql\nQuery: unchanged: output/queries/artists_long_songs-genre-limit.sql\n Test: unchanged: output/tests/artists_long_songs-genre-limit_test.sql\nQuery: unchanged: output/queries/songs_formats-artist-album.sql\nQuery: unchanged: output/queries/songs_formats-artist-file_format-album.sql\n Test: unchanged: output/tests/songs_formats-afa_test.sql\nIn a way, it's sort of a dry run for the render command.
"},{"location":"user-guide/commands/#-assert-no-changes","title":"--assert-no-changes","text":"A more effective use of this command though is with the --assert-no-changes flag which will cause it to exit with non-zero code if it finds any output files that would get added or modified upon rendering. It's recommended to be run as part of CD/CI, to prevent the user from mistakenly releasing code without rendering the templates.
"},{"location":"user-guide/commands/#summary","title":"summary","text":"The summary command prints a tabular summary of all queries along with their associated (query) templates and tests.
"},{"location":"user-guide/commands/#-all","title":"--all","text":"When --all option is specified with this command, the summary will include query and test files inside queries_output_dir and tests_output_dir respectively that are not added to the manifest.
Note
There are legit use cases for having files in the query and test output directories that are not added to the manifest Examples:
queries that don't need any tests but need to be stored in the same directory as other queries, so that yesql, aiosql libs can load all of them together.
Existing queries which are not yet migrated to tapestry (gradual migration strategy).
pgTAP tests written for stored procedures, views, schema etc. that need to be stored in the same directory as other tests, so that all tests can be run together.
"},{"location":"user-guide/commands/#coverage","title":"coverage","text":"The coverage command prints a list of queries along with the no. of tests (i.e. pgTAP test files) for them. It also prints a coverage score which is calculated as the percentage of queries that have at least 1 test.
Example: Following is the output of running tapestry coverage inside the examples/chinook dir.
$ tapestry coverage\n+----------------------------------------+------------------------------------+\n| Query | Has tests? |\n+=============================================================================+\n| artists_long_songs | Yes (1) |\n|----------------------------------------+------------------------------------|\n| artists_long_songs*limit | No |\n|----------------------------------------+------------------------------------|\n| artists_long_songs@genre*limit | Yes (1) |\n|----------------------------------------+------------------------------------|\n| songs_formats@artist+album | No |\n|----------------------------------------+------------------------------------|\n| songs_formats@artist&file_format+album | Yes (1) |\n|----------------------------------------+------------------------------------|\n| Total | 60.00% |\n| | (3/5 queries have at least 1 test) |\n+----------------------------------------+------------------------------------+\n--fail-under","text":"By specifying the --fail-under option, the coverage command can be made to exit with non-zero return code if the percentage coverage is below a threshold.
$ tapestry coverage --fail-under=90 > /dev/null\n$ echo $?\n1\nThe value of --fail-under option must be an integer between 0 and 100.
The above command can be run as part of CD/CI to ensure that the test coverage doesn't fall below a certain threshold.
"},{"location":"user-guide/docker/","title":"Docker","text":""},{"location":"user-guide/docker/#docker-based-workflow-for-running-pgtap-tests","title":"Docker based workflow for running pgTAP tests","text":"tapestry only generates SQL files for queries and pgTAP tests. To be able to run the tests you need to install and setup:
PostgreSQL server pgTAP, which is a postgres extensionpg_prove, which is a command line test runner/harness for pgTAP tests While these can be setup manually, the tapestry github repo provides a docker based workflow for easily running tests generated by tapestry against a temporary pg database.
The relevant files can be found inside the docker directory under project root.
Note
I use podman instead of docker for managing containers. Hence all the docker commands in this doc have been actually tested using podman only. As podman claims CLI compatibility with docker, I am assuming that replacing podman with docker in the below mentioned commands should just work. If that's not the case, please create an issue on github.
"},{"location":"user-guide/docker/#build-the-docker-image","title":"Build the docker image","text":"cd docker\npodman build -t tapestry-testbed -f ./Dockerfile\npodman run --name taptestbed \\\n --env POSTGRES_PASSWORD=secret \\\n -d \\\n -p 5432:5432 \\\n tapestry-testbed:latest\nVerify that the 5432 port is reachable from the host machine.
nc -vz localhost 5432\nThe above podman run command will create a container and start it. After that you can manage the container using the podman container commands
podman container stop taptestbed\npodman container start taptestbed\nThe pg_prove executable is part of the image that we have built. But to be able to run tests inside the container, we need to make the database schema and the test SQL files accessible to it. For this we bind mount a volume into the container when running it, using the --volume option.
The container image has a bash script run-tests installed into it which picks up the schema and the test SQL files from the mounted dir.
The run-tests scripts makes certain assumptions about organization of files inside the mounted dir. Inside the container, the dir must be mounted at /tmp/tapestry-data/ and there must be be two sub directories under it:
schema: All SQL files inside this dir will be executed against the database server in lexicographical order to setup a temporary test database.
tests: All SQL files inside this dir will be considered as tests and specified as arguments to the pg_prove command.
Once such a local directory is created, you can run the tests as follows,
podman run -it \\\n --rm \\\n --network podman \\\n -v ~/tapestry-data/:/tmp/tapestry-data/ \\\n --env PGPASSWORD=secret \\\n --env PGHOST=$(podman container inspect -f '{{.NetworkSettings.IPAddress}}' taptestbed) \\\n tapestry-testbed:latest \\\n run-tests -c -d temptestdb\nIn the above command, temptestdb is the name of the db that will be created by the run-tests script. If your schema files themselves take care of creating the db, then you can specify that as the name and omit the -c flag.
To know more about the usage of run-tests script, run,
podman run -it --rm tapestry-testbed:latest run-tests --help\nIf your SQL queries are even moderately complex, you'd want them to be formatted, mainly for readability. But with tapestry, you don't write the actual SQL by hand. Instead you write SQL code in bits and pieces within Jinja2 templates. Trying to get the SQL formatted the way you like using jinja2's whitespace control doesn't work well.
For that reason, tapestry takes care of formatting SQL at the time of rendering. For that, it supports a bunch of popular SQL formatting tools that the user may already have installed on their system. The currently supported formatting tools are:
pg_format sqlfluff sqlformat (inbuilt) The first two are external tools that tapestry \"shells-out\" to. Hence they are expected to be installed on your system.
Tapestry also comes with it's own inbuilt formatter that can be used in case none of the above tools are installed. It's powered by the sqlformat-rs crate. You may choose this if you don't prefer to install an additional system level dependency. Although the level of config supported by sqlformat is quite rudimentary.
"},{"location":"user-guide/formatting/#selecting-a-formatter","title":"Selecting a formatter","text":"When you initialize a new tapestry project by running tapestry init, it will try to find if any supported formatting tools are installed on your system. Based on that, it will show a prompt for selecting the tool of your choice.
$ tapestry init myproject\n? Choose an SQL formatter\n None (no formatting)\n> sqlformat (built-in)\n pg_format\n sql-formatter\n sqlfluff\n[The above SQL formatters were found on your system and available for use. Choose one or None to opt out of formatting]\nThe option selected by default is sqlformat which is the aforementioned inbuilt formatter.
None is also an option in case you'd like to opt out of SQL formatting. In that case, tapestry will skip the formatting step altogether.
"},{"location":"user-guide/formatting/#configuring-the-formatter","title":"Configuring the formatter","text":" "},{"location":"user-guide/formatting/#support-for-more-formatters","title":"Support for more formatters","text":"The underlying formatting component of tapestry is designed to be extensible, so that support for more tools can be added without much effort. If you want a particular SQL formatting tool to be supported, feel free to open an issue or a PR on github.
"},{"location":"user-guide/getting-started/","title":"Getting started","text":"This tutorial is to help you get started with tapestry. It's assumed that the following software is installed on your system:
tapestrypg_formata working installation of PostgreSQL (official docs) pgTAP and pg_prove "},{"location":"user-guide/getting-started/#sample-database","title":"Sample database","text":"For this tutorial, we'll use the chinook sample database. Download and import it as follows,
wget -P /tmp/ https://github.com/lerocha/chinook-database/releases/download/v1.4.5/Chinook_PostgreSql_SerialPKs.sql\ncreatedb chinook\npsql -d chinook -f /tmp/Chinook_PostgreSql_SerialPKs.sql\nWe'll start by running the tapestry init command, which will create the directory structure and also write a bare minimum manifest file for us. In a real project, you'd run this command from within the main project directory, so that the files can be committed to the same repo. But for this tutorial, you can run it from any suitable location e.g. the home dir ~/
cd ~/\ntapestry init chinook\nNew in version 0.2.0
Running init will prompt you to choose an sql formatter.
? Choose an SQL formatter\n None (no formatting)\n sqlformat (built-in)\n> pg_format\n sqlfluff\n[The above SQL formatters were found on your system and available for use. Choose one or None to opt out of formatting]\nThis example assumes that pg_format is chosen as the preferred formatter. For more details about SQL formatting support, see SQL formatting.
This will create a directory named chinook with following structure,
$ cd chinook\n$ tree -a --charset=ascii .\n.\n|-- .pg_format\n| `-- config\n|-- tapestry.toml\n`-- templates\n |-- queries\n `-- tests\nLet's look at the tapestry.toml manifest file that has been created (I've stripped out some comments for conciseness)
$ cat tapestry.toml\nplaceholder = \"posargs\"\n\nquery_templates_dir = \"templates/queries\"\ntest_templates_dir = \"templates/tests\"\n\nqueries_output_dir = \"output/queries\"\ntests_output_dir = \"output/tests\"\n\n[formatter.pgFormatter]\nexec_path = \"pg_format\"\nconf_path = \"./.pg_format/config\"\n\n[name_tagger]\nstyle = \"kebab-case\"\n\n# [[query_templates]]\n\n# [[queries]]\n\n# [[test_templates]]\nplaceholder defines the style of generated queries. Default is posargs (positional arguments) which will generate queries with $1, $2 etc as the placeholders. These are suitable for defining prepared statements.
Then there are four toml keys for defining directories,
query_templates_dir is where the query templates will be located
test_templates_dir is where the test templates will be located
queries_output_dir is where the SQL files for queries will be generated
tests_output_dir is where the SQL files for pgTAP tests will be generated.
All directory paths are relative to the manifest file.
You may have noticed that the init command created only the templates dirs. output dirs will be created when tapestry render is called for the first time.
The init command has also created a pg_format config file for us. This is because it found the pg_format executable on PATH. Refer to the pg_format section for more details.
Finally, name_tagger has been configured with kebab-case as the style.
"},{"location":"user-guide/getting-started/#adding-a-query_template-to-generate-queries","title":"Adding a query_template to generate queries","text":"Now we'll define a query template. But before that, you might want to get yourself familiar with the chinook database's schema.
Suppose we have an imaginary application built on top of the chinook database in which the following queries need to be run,
list all artists with their longest songs
list top 10 artists having longest songs
list top 5 artists having longest songs, and of a specific genre
As you can see, we'd need different queries for each of the 3 requirements, but all have a common logic of finding longest songs per artist. Using Jinja syntax, we can write a query template that covers all 3 cases as follows,
SELECT\n ar.artist_id,\n ar.name,\n max(milliseconds) * interval '1 ms' AS duration\nFROM\n track t\n INNER JOIN album al USING (album_id)\n INNER JOIN artist ar USING (artist_id)\n{% if cond__genre %}\n INNER JOIN genre g USING (genre_id)\n WHERE\n g.name = {{ placeholder('genre') }}\n{% endif %}\nGROUP BY\n ar.artist_id\nORDER BY\n-- Descending order because we want the top artists\n duration DESC\n{% if cond__limit %}\n LIMIT {{ placeholder('limit') }}\n{% endif %}\n;\nWe've used some custom Jinja variables for selectively including parts of SQL in the query. These need to be prefixed with cond__ and have to be defined in the manifest file (we'll come to that a bit later).
We have also used the custom Jinja function placeholder which takes one arg and expands to a placeholder in the actual query. This will be clear once we render the queries.
Let's save the above query template to the file templates/queries/artists_long_songs.sql.j2.
And now we'll proceed to defining the query_template and the queries that it can generate in the manifest file. Edit the tapestry.toml file by appending the following lines to it.
[[query_templates]]\npath = \"artists_long_songs.sql.j2\"\nall_conds = [ \"genre\", \"limit\" ]\nTo define a query_template we need to specify 2 keys:
path i.e. where the template file is located relative to the query_templates_dir defined earlier in the manifest. path itself is considered as the unique identifier for the query template.
all_conds is a set of values that will be converted to cond__ Jinja variables. In this case it means there are two cond__ Jinja templates supported by the template - cond__genre and cond__limit. Note that they are defined in the manifest without the cond__ suffix.
We can now define three different queries that map to the same query_template
[[queries]]\nid = \"artists_long_songs\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = []\n\n[[queries]]\nid = \"artists_long_songs*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"limit\" ]\n\n[[queries]]\nid = \"artists_long_songs@genre*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"genre\", \"limit\" ]\nTo define a query, we need to specify 3 keys,
id is an identifier for the query. Notice that we're following a naming convention by using special chars @ and *. Read more about Query naming conventions.
template is reference to the query template that we defined earlier.
conds is a subset of the all_conds key that's defined for the linked query template. In the context of this query, only the corresponding cond__ Jinja variables will have the value true, and the rest of them will be false.
We've defined three queries that use the same template. In the first query, both the conds that the template supports i.e. \"genre\" and \"limit\" are false. In the second query, \"limit\" is true but \"genre\" is false. In the third query, both \"genre\" and \"limit\" are true. Queries will be rendered based on these variables and the {% if cond__.. %} expressions in the template.
Don't worry if all this doesn't make much sense at this point. Things will be clear when we'll run tapestry render shortly.
"},{"location":"user-guide/getting-started/#rendering","title":"Rendering","text":"Now let's run the tapestry render command.
tapestry render\nAnd you'll notice some files created in our directory.
$ tree -a --charset=ascii .\n.\n|-- .pg_format\n| `-- config\n|-- output\n| |-- queries\n| | |-- artists_long_songs-genre-limit.sql\n| | |-- artists_long_songs-limit.sql\n| | `-- artists_long_songs.sql\n| `-- tests\n|-- tapestry.toml\n`-- templates\n |-- queries\n | `-- artists_long_songs.sql.j2\n `-- tests\nHere is what the generated output files look like:
artists_long_songs.sqlartists_long_songs-limit.sqlartists_long_songs-genre-limit.sql -- name: artists-long-songs\nSELECT\n ar.artist_id,\n ar.name,\n max(milliseconds) * interval '1 ms' AS duration\nFROM\n track t\n INNER JOIN album al USING (album_id)\n INNER JOIN artist ar USING (artist_id)\nGROUP BY\n ar.artist_id\nORDER BY\n -- Descending order because we want the top artists\n duration DESC;\n-- name: artists-long-songs-limit\nSELECT\n ar.artist_id,\n ar.name,\n max(milliseconds) * interval '1 ms' AS duration\nFROM\n track t\n INNER JOIN album al USING (album_id)\n INNER JOIN artist ar USING (artist_id)\nGROUP BY\n ar.artist_id\nORDER BY\n -- Descending order because we want the top artists\n duration DESC\nLIMIT $1;\n-- name: artists-long-songs-genre-limit\nSELECT\n ar.artist_id,\n ar.name,\n max(milliseconds) * interval '1 ms' AS duration\nFROM\n track t\n INNER JOIN album al USING (album_id)\n INNER JOIN artist ar USING (artist_id)\n INNER JOIN genre g USING (genre_id)\nWHERE\n g.name = $1\nGROUP BY\n ar.artist_id\nORDER BY\n -- Descending order because we want the top artists\n duration DESC\nLIMIT $2;\nThe SQL comments before the SQL with name of the query are generated by name_tagger added to the manifest. Learn more about Name tagging.
Also notice that the output SQL is formatted by pg_format.
"},{"location":"user-guide/getting-started/#adding-a-test_template","title":"Adding a test_template","text":"Now that we've defined and rendered queries, let's add test_template. Again there are two changes required - an entry in the manifest file and the Jinja template itself.
Add the following lines to the manifest file.
[[test_templates]]\nquery = \"artists_long_songs@genre*limit\"\npath = \"artists_long_songs-genre-limit_test.sql.j2\"\nHere we're referencing the query artists_long_songs@genre*limit hence this test is meant for that query. The path key points to a test template file that we need to create. So let's create the file templates/tests/artists_long_songs-genre-limit_test.sql.j2 with the following contents:
PREPARE artists_long_songs(varchar, int) AS\n{{ prepared_statement }};\n\nBEGIN;\nSELECT\n plan (1);\n\n-- start(noformat)\n-- Run the tests.\nSELECT results_eq(\n 'EXECUTE artists_long_songs(''Rock'', 10)',\n $$VALUES\n (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n (58, 'Deep Purple'::varchar, '00:19:56.094'::interval),\n (59, 'Santana'::varchar, '00:17:50.027'::interval),\n (136, 'Terry Bozzio, Tony Levin & Steve Stevens'::varchar, '00:14:40.64'::interval),\n (140, 'The Doors'::varchar, '00:11:41.831'::interval),\n (90, 'Iron Maiden'::varchar, '00:11:18.008'::interval),\n (23, 'Frank Zappa & Captain Beefheart'::varchar, '00:11:17.694'::interval),\n (128, 'Rush'::varchar, '00:11:07.428'::interval),\n (76, 'Creedence Clearwater Revival'::varchar, '00:11:04.894'::interval),\n (92, 'Jamiroquai'::varchar, '00:10:16.829'::interval)\n $$,\n 'Verify return value'\n);\n-- Finish the tests and clean up.\n-- end(noformat)\n\nSELECT\n *\nFROM\n finish ();\nROLLBACK;\nThe test syntax is SQL only but with some additional functions installed by pgTAP. If you are not familiar with pgTAP you can go through it's documentation. But for this tutorial, it's sufficient to understand that the {{ prepared_statement }} Jinja variable is made available to this template, and when it's rendered it will expand to the actual query.
Let's run the render command again.
tapestry render\nAnd now you should see the pgTAP test file created at output/tests/artists_long_songs-genre-limit_test.sql.
Note
Here the file stem of the test template path itself was used as the output file name. But it's also possible to explicitly specify it in the manifest file (see output in test_templates docs).
This is how the rendered test file looks like,
PREPARE artists_long_songs (varchar, int) AS\nSELECT\n ar.artist_id,\n ar.name,\n max(milliseconds) * interval '1 ms' AS duration\nFROM\n track t\n INNER JOIN album al USING (album_id)\n INNER JOIN artist ar USING (artist_id)\n INNER JOIN genre g USING (genre_id)\nWHERE\n g.name = $1\nGROUP BY\n ar.artist_id\nORDER BY\n -- Descending order because we want the top artists\n duration DESC\nLIMIT $2;\n\nBEGIN;\nSELECT\n plan (1);\n-- start(noformat)\n-- Run the tests.\nSELECT results_eq(\n 'EXECUTE artists_long_songs(''Rock'', 10)',\n $$VALUES\n (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n (58, 'Deep Purple'::varchar, '00:19:56.094'::interval),\n (59, 'Santana'::varchar, '00:17:50.027'::interval),\n (136, 'Terry Bozzio, Tony Levin & Steve Stevens'::varchar, '00:14:40.64'::interval),\n (140, 'The Doors'::varchar, '00:11:41.831'::interval),\n (90, 'Iron Maiden'::varchar, '00:11:18.008'::interval),\n (23, 'Frank Zappa & Captain Beefheart'::varchar, '00:11:17.694'::interval),\n (128, 'Rush'::varchar, '00:11:07.428'::interval),\n (76, 'Creedence Clearwater Revival'::varchar, '00:11:04.894'::interval),\n (92, 'Jamiroquai'::varchar, '00:10:16.829'::interval)\n $$,\n 'Verify return value'\n);\n-- Finish the tests and clean up.\n-- end(noformat)\nSELECT\n *\nFROM\n finish ();\nROLLBACK;\nAssuming that all the above mentioned prerequisites are installed, you can run the tests as follows,
sudo -u postgres pg_prove -d chinook --verbose output/tests/*.sql\nIf all goes well, the tests should pass and you should see output similar to,
1..1\nok 1 - Verify return value\nok\nAll tests successful.\nFiles=1, Tests=1, 0 wallclock secs ( 0.03 usr 0.01 sys + 0.01 cusr 0.00 csys = 0.05 CPU)\nResult: PASS\nIf you've reached this far, you should now have a basic understanding of what tapestry is and how to use it. Next, it'd be a good idea to understand the manifest file in more detail.
Note
The chinook example discussed in this tutorial can also be found in the github repo under the examples/chinook directory (there are a few more tests included for reference).
"},{"location":"user-guide/install/","title":"Installation","text":"Until tapestry is published to crates.io, you can install it directly from github,
cargo install --git https://github.com/naiquevin/tapestry.git\nTapestry can be configured to depend on external SQL formatting tools for formatting the generated SQL files. In that case, it expects the respective tool to be installed on the system.
Note that you need to install the formatting tools on the machine where you'd be rendering the SQL files using tapestry e.g. on your workstation and/or the build server.
But these are not hard requirements as tapestry also ships with a basic inbuilt SQL formatter so that the generated files will be properly formatted without the need of any external dependencies.
"},{"location":"user-guide/install/#dependencies-for-running-tests","title":"Dependencies for running tests","text":"If you are using tapestry to render tests (which you should, because that's what the tool is meant for!) then you'd also need the pgTAP extension and the pg_prove command line tool.
pgTAP can be easily built from source. Refer to the instructions here.
You can install pg_prove from a CPAN distribution as follows:
sudo cpan TAP::Parser::SourceHandler::pgTAP\nRefer to the pgTAP installation guide for more details.
As tapestry is a postgres specific tool, it goes without saying that you'd need a working installation of postgres to be able to run the tests. Please refer to the official documentation for that.
"},{"location":"user-guide/layouts/","title":"Layouts","text":"Tapestry lets you control the layout of the query files i.e. how the generated SQL is organized in files. It supports two ways at present:
one-file-one-query: Each SQL query will be written to a separate fileone-file-all-queries: All SQL queries will be written to a single file To configure this, you need to specify the query_output_layout key in the manifest. The default option if not specified is one-file-one-query.
"},{"location":"user-guide/layouts/#layout-and-queriesoutput-field","title":"Layout and queries[].output field","text":"Users may specify output field for every query, which is the path where the generated SQL output will be written. If output is not specified, it's value is derived from the query id. This works well for the one-file-one-query layout.
When the layout is one-file-all-queries, it's expected that the output field of all queries must be the same. Otherwise the manifest fails to validate. To avoid duplication, a related setting query_output_file is provided.
If layout = one-file-all-queries, it's recommended to set query_output_file and omit the output field for individual queries.
If layout = one-file-one-query, then you must not set query_output_file. Whether or not to set the output field for individual queries is up to you.
"},{"location":"user-guide/manifest/","title":"Manifest","text":"Every tapestry \"project\" has a tapestry.toml file which is called the manifest. It is in TOML format and serves the dual purpose of configuration as well as a registry of the following entities:
query_templatesqueriestest_templates The various sections or top level TOML keys are described in detail below. When going through this doc, you may find it helpful to refer to the chinook example in the github repo. If you haven't checked the Getting started section, it's recommended to read it first.
"},{"location":"user-guide/manifest/#placeholder","title":"placeholder","text":"The placeholder key is for configuring the style of the placeholder syntax for parameters i.e. the values values that are substituted into the statement when it is executed.
Two options are supported:
"},{"location":"user-guide/manifest/#posargs","title":"posargs","text":"posargs is short for positional arguments. The placeholders refer to the parameters by positions e.g. $1, $2 etc. This is the same syntax that's used for defining prepared statements or SQL functions in postgres.
This option is suitable when your db driver or SQL library accepts queries in prepared statements syntax. E.g. sqlx (Rust).
Default: The manifest file auto-generated upon running the tapestry init command will have,
placeholder = posargs\nWhen placeholder=variables placeholders are added in the rendered query using the variable substitution syntax of postgres. The variable name in the query is preceded with colon e.g. :email, :department
This option is suitable when your db driver or SQL library accepts queries with variables. E.g. yesql, hugsql (Clojure), aiosql (Python)
Examples
Templateplaceholder = posargsplaceholder = variables SELECT\n *\nFROM\n employees\nWHERE\n email = {{ placeholder('email') }}\n AND department = {{ placeholder('department') }};\nSELECT\n *\nFROM\n employees\nWHERE\n email = $1\n AND department = $2;\nSELECT\n *\nFROM\n employees\nWHERE\n email = :email\n AND department = :department;\nNote
Note that the prepared_statement Jinja variable available in test templates will always have posargs based placeholders even if the placeholder config in manifest file is set to variables. That's the reason the Jinja var is named prepared_statement.
"},{"location":"user-guide/manifest/#query_templates_dir","title":"query_templates_dir","text":"Path where the query templates are located. The path is always relative to the manifest file.
Default: The manifest file auto-generated upon running the tapestry init command will have,
query_templates_dir = \"templates/queries\"\nPath where the query templates are located. The path is always relative to the manifest file.
Default: The manifest file auto-generated upon running the tapestry init command will have,
test_templates_dir = \"templates/tests\"\nPath to the output dir for the rendered queries. This path also needs to be defined relative to the manifest file.
Default: The manifest file auto-generated upon running the tapestry init command will have,
queries_output_dir = \"output/queries\"\nA common use case to modify this config would be to store SQL files in a directory outside of the tapestry \"project\" dir, so that only the SQL files in that directory can be packaged into the build artifact. There's no need to include the query/test template and the pgTAP test files in the build artifact. E.g.
queries_output_dir = \"../sql_queries\"\nPath to the output dir for rendered pgTAP tests. The path is always relative to the manifest file.
Default: The manifest file auto-generated upon running the tapestry init command will have,
tests_output_dir = \"output/tests\"\nLayout to be used for the generated query files. The two options are:
one-file-one-query: Each SQL query will be written to a separate file
one-file-all-queries: All SQL queries will be written to a single file
It's optional. The default value is one-file-one-query.
"},{"location":"user-guide/manifest/#query_output_file","title":"query_output_file","text":"query_output_file is optional but it's use is valid only when the layout is one-file-all-queries. It basically saves the user from having to define the same output for all queries.
Refer to the Layouts section of the user guide for more info.
"},{"location":"user-guide/manifest/#formatterpgformatter","title":"formatter.pgFormatter","text":"This section is for configuring the pg_format tool that tapestry uses for formatting the rendered SQL files.
There two config params under this section:
"},{"location":"user-guide/manifest/#exec_path","title":"exec_path","text":"Location of the pg_format executable.
"},{"location":"user-guide/manifest/#conf_path","title":"conf_path","text":"Path to the pg_format config file. It can be used for configuring the behavior of pg_format when it gets executed on rendered SQL. As with all paths that we've seen so far, this one is also relative to the manifest file.
Example
[formatter.pgFormatter]\nexec_path = \"pg_format\"\nconf_path = \"./.pg_format/config\"\nAs mentioned in the installation guide, pg_format is not a mandatory requirement but it's recommended.
Upon running the tapestry init command, this section will be included in the auto-generated manifest file only if the executable pg_format is found on PATH. In that case, a default pg_format config file will also be created.
To read more about configuring pg_format in the context of tapestry, refer to the pg_format section of the docs.
"},{"location":"user-guide/manifest/#name_tagger","title":"name_tagger","text":"name_tagger is a TOML table, which if present in the manifest will cause the generated SQL queries to be name tagged.
"},{"location":"user-guide/manifest/#style","title":"style","text":"name_tagger.style can be used to control how name tags will be derived from query id. The two options are:
kebab-casesnake_caseexact Any special characters in the query id will be replaced with an appropriate character based on the above option \u2014 hyphen in case of kebab-case and underscore in case of snake_case. The third option exact is different in the sense that the query id will be used as it is as the name tag.
Example:
[name_tagger]\nstyle = \"kebab-case\"\nNote
Note the autological naming of options kebab-case (with a hyphen) v/s snake_case (with an underscore).
"},{"location":"user-guide/manifest/#query_templates","title":"query_templates","text":"query_templates is an array of tables in TOML parlance. So it needs to defined with double square brackets and can be specified multiple times in the manifest file.
For every query template, there are two keys to be defined:
"},{"location":"user-guide/manifest/#path","title":"path","text":"It's where the Jinja template file is located relative to the query_templates_dir defined earlier in the manifest. path itself is considered as the unique identifier for the query template.
Use .j2 extension as the convention for the query template file.
"},{"location":"user-guide/manifest/#all_conds","title":"all_conds","text":"It's a set of values that will be converted to cond__ Jinja variables that can be referenced inside the template. Note that they are defined in the manifest without the cond__ suffix.
This field is optional. If not specified, an empty set is considered as the default.
For documentation on how to write a query_template, refer to Writing query templates
Example:
[[query_templates]]\npath = \"artists_long_songs.sql.j2\"\nall_conds = [ \"genre\", \"limit\" ]\n\n[[query_templates]]\npath = \"songs_formats.sql.j2\"\nall_conds = [ \"artist\", \"file_format\", \"album_name\" ]\nNote
When all_conds is not specified, it essentially means that the query is a valid SQL statement and not a Jinja template. Then why define it as a template? The answer to that is \u2014 so that it can be embedded in tests.
"},{"location":"user-guide/manifest/#queries","title":"queries","text":"queries is an array of tables in TOML parlance. So it needs to defined with double square brackets and can be specified multiple times in the manifest file.
A query can be defined using the following keys,
"},{"location":"user-guide/manifest/#id","title":"id","text":"id is an identifier for the query.
"},{"location":"user-guide/manifest/#template","title":"template","text":"template is a reference to a query_template defined previously in the manifest.
"},{"location":"user-guide/manifest/#conds","title":"conds","text":"conds is a subset of the all_conds key that's defined for the linked query template. It's an optional and if not specified, an empty set will be considered by default.
"},{"location":"user-guide/manifest/#output","title":"output","text":"output is the path to the output file where the SQL query will be rendered. It must be relative to the queries_output_dir config.
It's optional to specify the output. If not specified, the filename of the output file will be derived by slugifying the id. This property allows us to use certain Naming conventions for giving suitable and consistent names to the queries.
Example:
[[queries]]\nid = \"artists_long_songs@genre*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"genre\", \"limit\" ]\nThe derived value of output for the above will be artists_long_songs-genre-limit.sql.
"},{"location":"user-guide/manifest/#name_tag","title":"name_tag","text":"name_tag can be optionally set to specify a custom name tag for the query. Name tags are prefixed to the SQL queries as comments and they are used by SQL loading libraries such as yesql, aiosql etc. Read more about in Name tagging queries.
Note
A query will be tagged with the specified name_tag only if name_tagger is set.
"},{"location":"user-guide/manifest/#test_templates","title":"test_templates","text":"test_templates is an array of tables in TOML parlance. So it needs to defined with double square brackets and can be specified multiple times in the manifest file.
A test_template can be defined using the following keys,
"},{"location":"user-guide/manifest/#query","title":"query","text":"query is a reference to query defined in the manifest.
"},{"location":"user-guide/manifest/#path_1","title":"path","text":"path is the path to the jinja template for the pgTAP test. It must be relative to the test_templates_dir.
Use .j2 extension as the convention for the test template file.
"},{"location":"user-guide/manifest/#output_1","title":"output","text":"output is the path where the pgTAP test file will be rendered. It must be relative to the tests_output_dir.
Specifying output for test_templates is optional. If not specified, it will be derived from the file stem of path i.e. by removing the .j2 extension.
For detailed documentation on how to write a test_template, refer to Writing test templates
"},{"location":"user-guide/naming-conventions/","title":"Query naming conventions","text":"One of the problems that I have encountered when using SQL loading libraries such as yesql and aiosql is that the queries defined in SQL files need to be given unique names. Often, one ends up writing a group of queries that are mostly similar to each other and differ only slightly. Giving unique and consistent names to each query can become tricky.
A tool like tapestry cannot automatically give a name to a query. However, since the queries are listed in the manifest file, we can partly address the problem with the use of naming conventions.
These naming conventions involve clever use of special characters such as @, +, & and *. Let's look at some examples from examples/chinook dir.
[[queries]]\nid = \"artists_long_songs@genre*limit\"\ntemplate = \"artists_long_songs.sql.j2\"\nconds = [ \"genre\", \"limit\" ]\n\n[[queries]]\nid = \"songs_formats@artist&file_format+album\"\ntemplate = \"songs_formats.sql.j2\"\nconds = [ \"artist\", \"album_name\", \"file_format\" ]\nIn the above queries, the id is defined using an alphanumeric prefix (artists_long_songs and songs_formats) followed by suffix that's an encoding of the conditional Jinja variables relevant to the query.
The convention is as follows,
@ precedes \"cond\" vars used for conditionally including a filter i.e. a WHERE clause. + precedes \"cond\" vars used for conditionally returning a column * precedes \"cond\" vars used for conditionally including any other part of the query e.g. LIMIT, ORDER BY etc. & is used as a delimiter between two \"cond\" vars of same type e.g. @artist&file_format. The name of the output file for the SQL query will be generated by slugifying the id i.e. by replacing the above special characters with hyphen (-). In case of the above two queries, the output file names will be artists_long_songs-genre-limit.sql and songs_formats-artist-file_format-album.sql respectively.
Note that these naming conventions are only recommended by tapestry and are not mandatory.
"},{"location":"user-guide/pg-format/","title":"Configuring pg_format","text":"tapestry can be configured to use pg_format for formatting the rendered SQL files. This makes sure that,
the rendered SQL files have consistent indentation you don't need to worry about SQL indentation when writing Jinja templates It can be installed on MacOS as follows,
brew install pgformatter\nDuring project initialization, if tapestry finds pg_format installed on your system (and in $PATH), it will show it as one of the formatter options. If you choose it, then following lines will be added to the tapestry.toml manifest file.
[formatter.pgFormatter]\n## (required) Location of the pg_format executable\nexec_path = \"pg_format\"\n## (optional) path to the pg_format conf file.\nconf_path = \"./.pg_format/config\"\nThe behavior of pg_format tool in the context of tapestry can be configured by adding a config file. The sample config file in the pg_format github repo can be used for reference.
The tapestry init command also generates a default config file, located at .pg_format/config (relative to the manifest file) and looks like this,
# Lines between markers 'start(noformat)' and 'end(noformat)' will not\n# be formatted. If you want to customize the markers, you may do so by\n# modifying this parameter.\nplaceholder=start\\(noformat\\).+end\\(noformat\\)\n\n# Add a list of function to be formatted as PG internal\n# functions. Paths relative to the 'tapestry.toml' file will also work\n#extra-function=./.pg_format/functions.lst\n\n# Add a list of keywords to be formatted as PG internal keywords.\n# Paths relative to the 'tapestry.toml' file will also work\n#extra-keyword=./.pg_format/keywords.lst\n\n# -- DANGER ZONE --\n#\n# Please donot change the following config parameters. Tapestry may\n# not work otherwise.\nmultiline=1\nformat=text\noutput=\nAs you can see, the generated file itself is well documented.
"},{"location":"user-guide/pg-format/#disallowed-configuration","title":"Disallowed configuration","text":"In the context of tapestry, some pg_format config params are disallowed (or they need to configured only in a certain way) for proper functioning of tapestry. These are explicitly defined with the intended value in the config file and annotated with DANGER ZONE warning in the comments. These must not be changed.
"},{"location":"user-guide/pg-format/#selectively-opting-out-of-sql-formatting","title":"Selectively opting out of SQL formatting","text":"A commonly faced problem with formatting pgTAP tests using pg_format is that hard coded expected values get formatted in a way that could make the test case unreadable for humans.
Example: Consider the following pgTAP test case written in a test template file,
SELECT results_eq(\n 'EXECUTE artists_long_songs(''Rock'', 2)',\n $$VALUES\n (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n (58, 'Deep Purple'::varchar, '00:19:56.094'::interval)\n $$,\n 'Verify return value'\n);\nBy default pg_format would format the above SQL snippet as follows,
SELECT\n results_eq ('EXECUTE artists_long_songs(''Rock'', 2)', $$\n VALUES (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval), (58, 'Deep Purple'::varchar, '00:19:56.094'::interval) $$, 'Verify return value');\nTo retain the readability, we need to preserve the user's custom indentation. This is where the placeholder config param of pg_format is useful
Note
pg_format's placeholder config is not to be confused with placeholder config key in tapestry's manifest.
This can be done by adding noformat markers before and after the snippet.
-- start(noformat)\nSELECT results_eq(\n 'EXECUTE artists_long_songs(''Rock'', 2)',\n $$VALUES\n (22, 'Led Zeppelin'::varchar, '00:26:52.329'::interval),\n (58, 'Deep Purple'::varchar, '00:19:56.094'::interval)\n $$,\n 'Verify return value'\n);\n-- end(noformat)\nIf you want to customize the markers for whatever reason, you can modify the placeholder param in the pg_format config file.
"},{"location":"user-guide/query-tags/","title":"Query tags","text":""},{"location":"user-guide/query-tags/#name-tagging-queries","title":"Name tagging queries","text":"Typically, the output query files rendered by tapestry are intended to be used by libraries such as yesql, aiosql etc. These libraries require the queries to be \"name-tagged\". Tagging is done by simply adding a comment before the query as follows,
-- name: my-query\n-- A simple query\nSELECT 1;\nThis way, these libraries can map the queries with the functions that it generates in code. These functions wraps around the database client/driver code and provides an easy interface for the user.
The following example is taken from yesql's README:
-- name: users-by-country\nSELECT *\nFROM users\nWHERE country_code = :country_code\n...and then read that file to turn it into a regular Clojure function:
(defqueries \"some/where/users_by_country.sql\"\n {:connection db-spec})\n\n;;; A function with the name `users-by-country` has been created.\n;;; Let's use it:\n(users-by-country {:country_code \"GB\"})\n;=> ({:name \"Kris\" :country_code \"GB\" ...} ...)\nTapestry does support name tagging of queries, but it's disabled by default. To enable it, just add the following lines in the manifest,
[name_tagger]\nstyle = \"kebab-case\"\nThis will result in name tags added to queries. The name tags are derived from the query ids. The style setting allows us to control how the id should be slugified to derive the name tag. For e.g. kebab-case will cause all non-alphanumeric characters in the id to be replaced by hyphens.
The other options for style are snake_case and exact.
"},{"location":"user-guide/query-tags/#custom-name-tags","title":"Custom name tags","text":"The above method derives name tags from query ids. But yesql and aiosql sometimes require the query names to be suffixed with specific characters to indicate specific operations. Example: In yesql, the name tags for INSERT/UPDATE/DELETE statements need to be suffixed with !.
-- name: save-person!\nUPDATE person\n SET name = :name\n WHERE id = :id\nThere are two ways to achieve this:
Specify exact as the name_tagger.style. Then the query id itself to be used as the name tag (as it is).
Specify the optional queries[].name_tag field when defining the queries.
While it may seem like the first approach involves less effort, the downside is that we'd be giving up on the Naming conventions that tapestry recommends.
Libraries such as yesql and aiosql usually don't allow special characters in the name tags as they use them to generate functions in code. So yesql recommends the name tags to be in kebab-case as Clojure functions follow that convention, whereas aiosql needs the name tags to be in snake_case as that's the requirement and also the convention in Python.
"},{"location":"user-guide/query-templates/","title":"Writing query templates","text":"Query templates are Jinja template files. One query template can be used for generating multiple SQL queries.
Often, an application needs to issue mostly similiar (or slightly different) queries to the db based on user input. Some examples:
two queries that are exactly similar, except that one returns all columns i.e. * whereas the other returns only selected rows two queries that are exactly the same, except that one has a limit multiple similar queries but different combination of WHERE clauses Using Jinja templates, it's possible to write a single query template that can render multiple SQL queries. This is possible with a combination of Jinja variables and {% if .. %}...{% endif %} blocks. This is pretty much the main idea behind query templates.
"},{"location":"user-guide/query-templates/#cond-variables","title":"\"cond\" variables","text":"Query templates need to be defined in the manifest where we specify all_conds which is a set of \"cond\" vars that the template supports.
Let's look at a query template from the chinook example distributed with the github repo.
SELECT\n track.name as title,\n artist.name as artist_name,\n {% if cond__album_name %}\n album.title as album_name,\n {% endif %}\n media_type.name as file_format\nFROM\n album\n JOIN artist USING (artist_id)\n LEFT JOIN track USING (album_id)\n JOIN media_type USING (media_type_id)\n\n{% if cond__artist or cond__file_format %}\n WHERE\n {% set num_conds = 0 %}\n {% if cond__artist %}\n artist.name = {{ placeholder('artist') }}\n {% set num_conds = num_conds + 1 %}\n {% endif %}\n\n {% if cond__file_format %}\n {% if num_conds > 0 %}\n AND\n {% endif %}\n media_type.name = {{ placeholder('file_format') }}\n {% set num_conds = num_conds + 1 %}\n {% endif %}\n{% endif %}\n;\nThe entry in the manifest file for the above query_template is,
[[query_templates]]\npath = \"songs_formats.sql.j2\"\nall_conds = [ \"artist\", \"file_format\", \"album_name\" ]\nBecause of the 3 all_conds defined in the manifest file, we have the following Jinja variables available inside the Jinja template.
cond__artistcond__file_formatcond__album_name The cond__artist and cond__file_format vars are used for conditionally including WHERE clauses. Because we want to add the WHERE clause only if either of the two vars are true, and because we want to add the AND operator only if both are true, nested if blocks are used and a temp \"counter\" variable num_conds is defined i.e. it's assigned to 0 and then incremented by 1 if the cond__artist var is true.
The third variable cond__album_name is used for conditionally including a column in the returned result.
"},{"location":"user-guide/query-templates/#query","title":"Query","text":"Now let's look at how a query associated with this template is defined in the manifest.
[[queries]]\nid = \"songs_formats@artist+album\"\ntemplate = \"songs_formats.sql.j2\"\nconds = [ \"artist\", \"album_name\" ]\noutput = \"songs_formats__artist__album.sql\"\nIn this query, only 2 of the 3 \"cond\" variables will be true.
As a total of 3 all_conds values are supported by the query template, 8 different queries can be generated from it using different subsets of all_conds.
[ ]\n[ \"artists\" ]\n[ \"artists\", \"file_format\" ]\n[ \"artists\", \"album_name\" ]\n[ \"file_format\" ]\n[ \"file_format\", \"album_name\" ]\n[ \"album_name\" ]\n[ \"artist\", \"file_format\", \"album_name\" ]\nsql-formatter is a Javascript library and command line tool for pretty printing SQL. It supports multiple SQL dialects including postgresql, and hence makes for a pretty good external tool that tapestry can use for formatting the generated SQL.
You can easily install it using npm,
npm install -g sql-formatter\nDuring project initialization, if sql-formatter is found installed on your system (and in $PATH), it will be shown as one of the formatter options. Upon choosing it, following lines will be added to the tapestry.toml manifest file.
[formatter.sql-formatter]\n# (required) Location of the sql-formatter executable\nexec_path = \"sql-formatter\"\n# (optional) path to the json conf file.\nconf_path = \"./.sql-formatter/config.json\"\nsql-formatter can be configured through a JSON file. The init command also dumps a default JSON file at ./.sql-formatter/config.json, relative to the manifest file, with the following contents:
{\n \"language\": \"postgresql\",\n \"tabWidth\": 4,\n \"keywordCase\": \"upper\",\n \"linesBetweenQueries\": 2\n}\nRefer to the sql-formatter documentation for more configuration options.
"},{"location":"user-guide/sqlfluff/","title":"Configuring sqlfluff","text":"sqlfluff is a feature rich SQL formatter that's written in Python and hence is an external dependency for tapestry.
sqlfluff recognizes a file named ./.sqlfluff a standard configuration file to load config from. Tapestry capitalizes on this so that there's no need to invent a new config format.
The configuration options for sqlfluff are quite extensive and well documented - https://docs.sqlfluff.com/en/stable/configuration/index.html.
During project initialization, if sqlfluff is found installed on your system (and in $PATH), it will be shown as one of the formatter options. Upon choosing it, following lines will be added to the tapestry.toml manifest file.
[formatter.sqlfluff]\n# (required) Location of the sqlfluff executable\nexec_path = \"sqlfluff\"\nAdditionally, it will also create the .sqlfluff config file alongside the manifest file.
[sqlfluff]\ndialect = postgres\nYou may refer to sqlfluff documentation to configure formatting as per your preferences.
Note
Unlike in pgFormatter's config, the path to the sqlfluff config file doesn't need to be explicitly specified in the manifest file. Similar to normal functioning of sqlfluff, config will be implicitly loaded from a file named ./.sqlfluff in the current directory. Since tapestry commands are run from the same dir that this file is created in, it just works.
"},{"location":"user-guide/sqlformat-rs/","title":"Configuring sqlformat","text":"sqlformat is the inbuilt formatter supported by tapestry. It's implemented using the the sqlformat crate.
It provides 3 basic config options:
indentation: Default is 4 spaces
uppercase: Whether or not reserved keywords should be converted to UPPERCASE.
lines_between_queries: No. of empty lines between two queries.
During project initialization, sqlformat is shown as one of the options, besides other external formatters. Upon choosing it as the preferred formatter, following lines are added to the tapestry.toml manifest file.
[formatter.sqlformat-rs]\n# (optional) No. of spaces to indent by\nindent = 4\n# (optional) Use ALL CAPS for reserved keywords\nuppercase = true\n# (optional) No. of line breaks after a query\nlines_between_queries = 1\nJust like query_templates, test_templates are also Jinja template files. But while one query template could be used to generate several queries, one test template can be used to generate only one pgTAP test file.
However, many test templates can be associated with a single query. In other words, if multiple pgTAP test suites are to be written for the same query, that's possible.
The test syntax is SQL only but with some additional functions installed by pgTAP. If you are not familiar with pgTAP you can go through it's documentation. Important thing to note is that the Jinja variable {{ prepared_statement }} is made available to every test template, and at the time of rendering, it will expand to the actual query.
Let's look at a templates from the chinook example.
Refer to the test template songs_formats-afa_test.sql.j2. The first few lines are:
PREPARE song_formats (varchar, varchar) AS\n{{ prepared_statement }};\nHere we're using the prepared_statement Jinja variable to create a prepared statement for the user session. The name of the prepared statement is song_formats and it takes two positional args, both of type varchar.
Later in the same file, the prepared statement is executed as part of a pgTAP test case,
SELECT results_eq(\n 'EXECUTE song_formats(''Iron Maiden'', ''Protected AAC audio file'')',\n $$VALUES\n ...\n ...\n $$,\n 'Verify return value'\n);\nCheck the songs_formats-afa_test.sql output file to see how the actual test file looks like.
Note
Note that the SQL query that prepared_statement Jinja var expands to will always have posargs based placeholders, even if the placeholder config in manifest file is set to variables. That's the reason why the Jinja var is named prepared_statement
"},{"location":"user-guide/test-templates/#function-instead-of-ps","title":"Function instead of PS","text":"Sometimes it's tedious to test for result sets returned by the query. In such cases, it helps to manipulate the result returned by the query and compare a derived property. E.g. If a query results too many rows, it's easier to compare the count than the actual values in the rows.
One limitation of prepared statements and the EXECUTE syntax for executing them is that it's not sub-query friendly i.e. it's not possible to execute a prepared statement as part of another query.
The following is NOT valid SQL
SELECT\n count(*)\nFROM (EXECUTE song_formats ('Iron Maiden', 'Protected AAC audio file'));\nIn such cases, we can define a SQL function using the same prepared_statement Jinja variable.
An example of this can be found in the chinook example - all_artists_long_songs_test.sql.j2
CREATE OR REPLACE FUNCTION all_artists_long_songs ()\nRETURNS SETOF record\nAS $$\n{{ prepared_statement }}\n$$ LANGUAGE sql;\n\nBEGIN;\nSELECT\n plan (1);\n\n-- start(noformat)\n-- Run the tests.\nSELECT is(count(*), 204::bigint) from all_artists_long_songs() AS (artist_id int, name text, duration interval);\n-- Finish the tests and clean up.\n-- end(noformat)\n\nSELECT\n *\nFROM\n finish ();\nROLLBACK;\nWhen it comes to automated tests, It's a very common requirement to setup some test data to be able to write test cases. pgTAP tests are not any different. In case of pgTAP one needs to create test data in the database.
Since pgTAP tests are just SQL files, test data creation can be done using SQL itself in the same file. Reusable setup code can also be extracted into SQL functions that can be created as part of importing the database schema.
The chinook directory doesn't include an example of this. But here's an example from one of my real projects that uses tapestry.
In my project, there are two entities categories and items (having tables of the same names) with one-to-many relationship i.e. one category can have multiple items.
In several pgTAP tests, a few categories and items need to be created. To do this, a function is defined as follows,
CREATE OR REPLACE FUNCTION tapestry.setup_category_n_items (cat_id varchar, item_idx_start integer, item_idx_end integer)\n RETURNS void\n AS $$\n INSERT INTO categories (id, name)\n VALUES (cat_id, initcap(replace(cat_id, '-', ' ')));\n INSERT INTO items (id, name, category_id)\n SELECT\n 'item-' || t AS id,\n 'Item ' || t AS name,\n cat_id AS category_id\n FROM\n generate_series(item_idx_start, item_idx_end) t;\n$$\nLANGUAGE sql;\nAnd then it's used in pgTAP tests like this,
...\n\nBEGIN;\nSELECT plan(1);\n\n-- Fixtures\n-- create 2 categories, 'cat-a' and 'cat-b' each having 5 items\nSELECT\n tapestry.setup_category_n_items ('cat-a', 1, 5);\nSELECT\n tapestry.setup_category_n_items ('cat-b', 6, 10);\n\n-- Test cases\n\n...\n
+
+
+
+
+
+
+
+ SQL formatting
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/user-guide/docker/index.html b/user-guide/docker/index.html
index b34d85d..520e0a1 100644
--- a/user-guide/docker/index.html
+++ b/user-guide/docker/index.html
@@ -11,7 +11,7 @@
-
+
+
+
+
+
+
+
+ SQL formatting
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/user-guide/formatting.md~ b/user-guide/formatting.md~
new file mode 100644
index 0000000..c7e52aa
--- /dev/null
+++ b/user-guide/formatting.md~
@@ -0,0 +1,57 @@
+# Formatting SQL
+
+If your SQL queries are even moderately complex, you'd want them to be
+formatted, mainly for readability. But with tapestry, you don't write
+the actual SQL by hand. Instead you write SQL code in bits and pieces
+within Jinja2 templates. Trying to get the SQL formatted the way you
+like using jinja2's whitespace control doesn't work well.
+
+For that reason, tapestry takes care of formatting SQL at the time of
+rendering. For that, it supports a bunch of popular SQL formatting
+tools that the user may already have installed on their system. The
+currently supported formatting tools are:
+
+1. pg_format
+2. sqlfluff
+3. sqlformat (inbuilt)
+
+The first two are external tools that tapestry "shells-out" to. Hence
+they are expected to be installed on your system.
+
+Tapestry also comes with it's own inbuilt formatter that can be used
+in case none of the above tools are installed. It's powered by the
+sqlformat-rs crate. You may choose this if you don't prefer to install
+an additional system level dependency. Although the level of config
+supported by sqlformat is quite rudimentary.
+
+## Selecting a formatter
+
+When you initialize a new tapestry project by running `tapestry init`,
+it will try to find if any supported formatting tools are installed on
+your system. Based on that, it will show a prompt for selecting the
+tool of your choice.
+
+The option selected by default is `sqlformat` which is the
+aforementioned inbuilt formatter.
+
+`None` is also an option in case you'd like to opt out of SQL
+formatting. In that case, tapestry will skip the formatting step
+altogether.
+
+## Configuring the formatter
+
+### sqlformat
+
+### pg_format
+
+### sqlfluff
+
+## Support for more formatters
+
+The underlying formatting component of tapestry is designed to be
+extensible, so that support for more tools can be added without much
+effort. If you want a particular SQL formatting tool to be supported,
+feel free to open an issue or a PR on github.
+
+
+
diff --git a/user-guide/formatting/index.html b/user-guide/formatting/index.html
new file mode 100644
index 0000000..c296cdf
--- /dev/null
+++ b/user-guide/formatting/index.html
@@ -0,0 +1,1004 @@
+
+
+
+
+
+ Overview - tapestry
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+If your SQL queries are even moderately complex, you'd want them to be
+formatted, mainly for readability. But with tapestry, you don't write
+the actual SQL by hand. Instead you write SQL code in bits and pieces
+within Jinja2 templates. Trying to get the SQL formatted the way you
+like using jinja2's whitespace control doesn't work well.
+For that reason, tapestry takes care of formatting SQL at the time of
+rendering. For that, it supports a bunch of popular SQL formatting
+tools that the user may already have installed on their system. The
+currently supported formatting tools are:
+
+pg_format
+sqlfluff
+sqlformat (inbuilt)
+
+The first two are external tools that tapestry "shells-out" to. Hence
+they are expected to be installed on your system.
+Tapestry also comes with it's own inbuilt formatter that can be used
+in case none of the above tools are installed. It's powered by the
+sqlformat-rs crate. You may choose this if you don't prefer to install
+an additional system level dependency. Although the level of config
+supported by sqlformat is quite rudimentary.
+
+When you initialize a new tapestry project by running tapestry init,
+it will try to find if any supported formatting tools are installed on
+your system. Based on that, it will show a prompt for selecting the
+tool of your choice.
+$ tapestry init myproject
+? Choose an SQL formatter
+ None (no formatting)
+> sqlformat (built-in)
+ pg_format
+ sql-formatter
+ sqlfluff
+[The above SQL formatters were found on your system and available for use. Choose one or None to opt out of formatting]
+The option selected by default is sqlformat which is the
+aforementioned inbuilt formatter.
+None is also an option in case you'd like to opt out of SQL
+formatting. In that case, tapestry will skip the formatting step
+altogether.
+
+
+
+
+
+
+The underlying formatting component of tapestry is designed to be
+extensible, so that support for more tools can be added without much
+effort. If you want a particular SQL formatting tool to be supported,
+feel free to open an issue or a PR on github.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ SQL formatting
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
@@ -853,6 +983,18 @@ Init
cd ~/
tapestry init chinook
New in version 0.2.0
+Running init will prompt you to choose an sql formatter.
+? Choose an SQL formatter
+ None ( no formatting)
+ sqlformat ( built-in)
+> pg_format
+ sqlfluff
+[ The above SQL formatters were found on your system and available for use. Choose one or None to opt out of formatting]
+This example assumes that pg_format is chosen as the preferred
+formatter. For more details about SQL formatting support, see SQL
+formatting .
This will create a directory named chinook with following structure,