Duct is a framework for developing server-side applications in the Clojure programming language.
Duct does not rely on a project template; the skeleton of a Duct application is defined by an immutable data structure. This structure can then be queried or modified in ways that would be difficult with a more traditional framework.
While Duct has more general use cases, it’s particularly well-suited for writing web applications. This documentation will take you through Duct’s setup and operation, using a web application as an example project.
|
Warning
|
This documentation assumes knowledge of Clojure. Some commands assume a Unix-like shell environment. |
This section will introduce the fundamentals of Duct’s design and implement a minimal ‘Hello World’ application. While it may be tempting to skip ahead, a sound understanding of how to use Duct will make later sections easier to follow.
This section will cover setting up a new Duct project. You’ll first need
to ensure that the
Clojure CLI is installed.
You can check this by running the clojure command.
$ clojure --version
Clojure CLI version 1.12.0.1479Next, create a project directory. For the purposes of this example,
we’ll call the project tutorial.
$ mkdir tutorial && cd tutorialThe Clojure CLI looks for a file called deps.edn. To use Duct, we need
to add the Duct Main tool as a
dependency, and setup an alias to execute it.
To achieve this, create a new deps.edn file with the following
content:
{:deps {org.clojure/clojure {:mvn/version "1.12.0"}
org.duct-framework/main {:mvn/version "0.1.5"}}
:aliases {:duct {:main-opts ["-m" "duct.main"]}}}Duct can now be run by invoking the :duct alias.
$ clojure -M:duct
Usage:
clojure -M:duct [--main | --repl]
Options:
-c, --cider Start an NREPL server with CIDER middleware
--init Create a blank duct.edn config file
-k, --keys KEYS Limit --main to start only the supplied keys
-p, --profiles PROFILES A concatenated list of profile keys
-n, --nrepl Start an NREPL server
-m, --main Start the application
-r, --repl Start a command-line REPL
-s, --show Print out the expanded configuration and exit
-v, --verbose Enable verbose logging
-h, --help Print this help message and exitWe’ll be using this command a lot, so it’s highly recommended that you
also create an shell alias. In a POSIX shell such as Bash, this can be
done using the alias command.
$ alias duct="clojure -M:duct"For the rest of this documentation, we’ll assume that this shell alias has been defined.
The final step of the setup process is to create a duct.edn file. This
contains the data structure that defines your Duct application. The
Duct Main tool has a flag to generate a minimal configuration file for
us.
$ duct --init
Created duct.ednThis will create a file: duct.edn.
{:system {}}As mentioned previously, Duct uses a file, duct.edn, to define the
structure of your application. We’ll begin by adding a new component
key to the system.
{:system
{:tutorial.print/hello {}}}If we try running Duct, it will complain about a missing namespace.
$ duct --main
✗ Initiating system...
Execution error (IllegalArgumentException) at integrant.core/eval1191$fn (core.cljc:490).
No such namespace: tutorial.printDuct is searching for a definition for the component, but not finding anything. This is unsurprising, as we haven’t written any code yet. Let’s fix this.
First we’ll create the directories.
mkdir -p src/tutorialThen a minimal Clojure file at: src/tutorial/print.clj.
(ns tutorial.print)
(defn hello [_options]
(println "Hello World"))Now if we try to run the application, we get the expected output.
$ duct --main
✓ Initiating system...
Hello WorldCongratulations on your first Duct application!
Duct has two ways of running your application: --main and --repl.
In the previous section we started the application with --main, which
will initiate the system defined in the configuration file, and halt
the system when the process terminates.
The REPL is an interactive development environment.
$ duct --repl
✓ Loading REPL environment...
• Type :repl/help for REPL help, (go) to initiate the system and (reset)
to reload modified namespaces and restart the system (hotkey Alt-E).
user=>In the REPL environment the system will not be initiated automatically.
Instead, we use the inbuilt (go) function.
user=> (go)
Hello World
:initiatedThe REPL can be left running while source files updated. The (reset)
function will halt the running system, reload any modified source files,
then initiate the system again.
user=> (reset)
:reloading (tutorial.print)
Hello World
:resumedYou can also use the Alt-E hotkey instead of typing (reset).
The configuration defined by duct.edn can be accessed with config,
and the running system can be accessed with system.
user=> config
#:tutorial.print{:hello {}}
user=> system
#:tutorial.print{:hello nil}A module groups multiple components together. Duct provides a number
of pre-written modules that implement common functionality. One of these
modules is :duct.module/logging.
We’ll first add the new dependency:
{:deps {org.clojure/clojure {:mvn/version "1.12.0"}
org.duct-framework/main {:mvn/version "0.1.5"}
org.duct-framework/module.logging {:mvn/version "0.6.5"}}
:aliases {:duct {:main-opts ["-m" "duct.main"]}}}Then we’ll add the module to the Duct configuration.
{:system
{:duct.module/logging {}
:tutorial.print/hello {}}}Before the components are initiated, modules are expanded. We can see
what this expansion looks like by using the --show flag. This will
print out the expanded configuration instead of initiating it.
$ duct --main --show
{:duct.logger/simple {:appenders [{:type :stdout}]}
:tutorial.print/hello {}}The logging module has been replaced with the :duct.logger/simple
component.
|
Note
|
Data in the configuration file will override data from expansions. |
The --show flag also works with the --repl command.
$ duct --repl --show
{:duct.logger/simple
{:appenders
[{:type :stdout, :brief? true, :levels #{:report}}
{:type :file, :path "logs/repl.log"}]}
:tutorial.print/hello {}}But wait a moment, why is the expansion of the configuration different
depending on how we run Duct? This is because the --main flag has an
implicit :main profile, and the --repl flag has an implicit :repl
profile.
The :duct.module/logging module has different behaviors depending on
which profile is active. When run with the :main profile, the logs
print to STDOUT, but this would be inconveniently noisy when using a
REPL. So when the :repl profile is active, most of the logs are sent
to a file, logs/repl.log.
In order to use this module, we need to connect the logger to our ‘hello’ component. This is done via a ref.
{:system
{:duct.module/logging {}
:tutorial.print/hello {:logger #ig/ref :duct/logger}}}The #ig/ref data reader is used to give the ‘hello’ component access
to the logger. We use :duct/logger instead of :duct.logger/simple,
as keys have a logical hierarchy, and :duct/logger fulfils a role
similar to that of an interface or superclass.
|
Note
|
The ‘ig’ in #ig/var stands for
Integrant. This is the
library that Duct relies on to turn configurations into running
applications.
|
Now that we’ve connected the components together in the configuration
file, it’s time to replace the println function with the Duct logger.
(ns tutorial.print
(:require [duct.logger :as log]))
(defn hello [{:keys [logger]}]
(log/report logger ::hello {:name "World"}))The duct.logger/report function is used to emit a log at the :report
level. This is a high-priority level that should be used sparingly, as
it also prints to STDOUT when using the REPL.
You may have noticed that we’ve replaced the "Hello World" string with
a keyword and a map: ::name {:name "World"}. This is because Duct is
opinionated about logs being data, rather than human-readable strings. A
Duct log message consists of an event, a qualified keyword, and a map
of event data, which provides additional information.
When we run the application, we can see what this produces.
$ duct --main
✓ Initiating system...
2024-11-23T18:59:14.080Z :report :tutorial.print/hello {:name "World"}But when using the REPL, we get a more concise message.
user=> (go)
:initiated
:tutorial.print/hello {:name "World"}Sometimes we want to supply options from an external source, such as an
environment variable or command line option. Duct allows variables, or
vars, to be defined in the duct.edn configuration.
Currently our application outputs the same log message each time it’s run. Let’s create a configuration var to customize that behavior.
{:vars
{name {:arg name, :env NAME, :type :str, :default "World"
:doc "The name of the person to greet"}}
:system
{:duct.module/logging {}
:tutorial.print/hello {:logger #ig/ref :duct/logger
:name #ig/var name}}}Then in the source file we can add the :name option that the var is
attached to.
(ns tutorial.print
(:require [duct.logger :as log]))
(defn hello [{:keys [logger name]}]
(log/report logger ::hello {:name name}))The default ensures that the application functions the same as before.
$ duct --main
✓ Initiating system...
2024-11-23T23:53:47.069Z :report :tutorial.print/hello {:name "World"}But we can now customize the behavior via a command-line flag, --name,
or via an environment variable, NAME.
$ duct --main --name=Clojurian
✓ Initiating system...
2024-11-24T04:45:19.521Z :report :tutorial.print/hello {:name "Clojurian"}
$ NAME=Clojurist duct --main
✓ Initiating system...
2024-11-24T04:45:54.211Z :report :tutorial.print/hello {:name "Clojurist"}Vars are defined as a map of symbols to maps of options. The following option keys are supported:
:arg
|
a command-line argument to take the var’s value from |
:default
|
the default value if the var is not set |
:doc
|
a description of what the var is for |
:env
|
an environment variable to take the var’s value from |
:type
|
a data type to coerce the var into (one of: |
A Duct application has some number of active profiles, which are
represented by unqualified keywords. When run via the --main flag, an
implicit :main profile is added. When run via (go) at the REPL, an
implicit :repl profile is added.
You can add additional profiles via the --profiles argument. Profiles
are an ordered list, with preceding profiles taking priority.
$ duct --profiles=:dev --mainMost of the modules that Duct provides use profiles to customize their
behavior to the environment they’re being run under. We can also use the
#ig/profile data reader to create our own profile behavior.
Let’s change our component to allow for the log level to be specified.
(ns tutorial.print
(:require [duct.logger :as log]))
(defn hello [{:keys [level logger name]}]
(log/log logger level ::hello {:name name}))In duct.edn we can use a profile to change the log level depending
on whether the application uses the :main or :repl profile.
{:vars
{name {:arg name, :env NAME, :type :str, :default "World"
:doc "The name of the person to greet"}}
:system
{:duct.module/logging {}
:tutorial.print/hello
{:logger #ig/ref :duct/logger
:level #ig/profile {:repl :report, :main :info}
:name #ig/var name}}}So far we’ve used functions to implement components. The
:tutorial.print.hello component was defined by:
(ns tutorial.print
(:require [duct.logger :as log]))
(defn hello [{:keys [level logger name]}]
(log/log logger level ::hello {:name name}))But this is just convenient syntax sugar for Integrant’s init-key
method. The following code is equivalent to the previous component
definition:
(ns tutorial.print
(:require [duct.logger :as log]
[integrant.core :as ig))
(defmethod ig/init-key ::hello [_key {:keys [level logger name]}]
(log/log logger level ::hello {:name name}))Duct uses Integrant for its component definitions, and Integrant
provides several multimethods to this end. The most common one is
init-key. If no such method is found, Integrant searches for a
function of the same name.
There is also halt-key!, which defines a teardown procedure for a key.
This can be useful for cleaning up files, threads or connections that
the init-key method (or function) opened. The return value from
init-key will be passed to halt-key!.
(ns tutorial.print
(:require [duct.logger :as log]
[integrant.core :as ig))
(defmethod ig/init-key ::hello [_key {:keys [level logger name] :as opts}]
(log/log logger level ::hello {:name name})
opts)
(defmethod ig/halt-key! ::hello [_key {:keys [level logger name]}]
(log/log logger level ::goodbye {:name name}))For more information on the multimethods that can be used, refer to the Integrant documentation.
While Duct can be used for any server-side application, its most common use-case is developing web applications and services. This section will take you through writing a ‘todo list’ web application in Duct.
We’ll begin by creating a new project directory.
mkdir todo-app && cd todo-appThe first thing we’ll need is a deps.edn file that to provide the
project dependencies. This will include Duct main and two additional
modules: logging and web.
{:deps {org.clojure/clojure {:mvn/version "1.12.0"}
org.duct-framework/main {:mvn/version "0.1.5"}
org.duct-framework/module.logging {:mvn/version "0.6.5"}
org.duct-framework/module.web {:mvn/version "0.12.0"}}
:aliases {:duct {:main-opts ["-m" "duct.main"]}}}With that done, we need to ensure that the src directory exists. This
is the default directory Clojure uses to store source files.
$ mkdir src|
Important
|
It is especially important to ensure the source directory exists before starting a REPL, otherwise the REPL will not be able to load source changes. |
As this is a Duct application, we’ll need a duct.edn file. This will
contain the two modules we added to the project’s dependencies.
{:system
{:duct.module/logging {}
:duct.module/web {}}}We can now start the application with --main.
$ duct --main
✓ Initiating system...
2024-11-25T02:51:08.279Z :report :duct.server.http.jetty/starting-server {:port 3000}The web application should now be up and running at: http://localhost:3000/
Visiting that URL will result in a ‘404 Not Found’ error page, because we have no routes defined. The error page will be in plaintext, because we haven’t specified what features we want for our web application.
We’ll fix both these issues, but before we do we should terminate the application with Ctrl-C and start a REPL. We’ll keep this running while we develop the application to avoid costly restarts and to give us a way of querying the running system.
$ duct --repl
✓ Loading REPL environment...
• Type :repl/help for REPL help, (go) to initiate the system and (reset)
to reload modified namespaces and restart the system (hotkey Alt-E).
user=> (go)
:duct.server.http.jetty/starting-server {:port 3000}
:initiatedClojure has many excellent libraries for writing web applications, but it can be difficult to put them all together. Duct’s web module handles that for you, but like all modules, we can always override any default that we don’t like.
For now, we’ll tell the web module to configure the application for use
as a webside, using the :site feature. We’ll also add in a single
route to handle a web request to the root of our application.
{:system
{:duct.module/logging {}
:duct.module/web
{:features #{:site}
:routes [["/" {:get :todo.routes/index}]]}}}Then we’ll create a handler function for that route.
(ns todo.routes)
(defn index [_options]
(fn [_request]
[:html {:lang "en"}
[:head [:title "Hello World Wide Web"]]
[:body [:h1 "Hello World Wide Web"]]]))Finally, we trigger a (reset) at the REPL.
user=> (reset)
:reloading (todo.routes)
:resumedNow when we go access http://localhost:3000/ we find a HTML page instead. Congratulations on your first Duct web application!
In the previous section we set up a route and a handler function, but you may rightly wonder how the route finds the function.
In the [_fundamentals] section we learned that key/value pairs in the Duct configuration have definitions in the application’s source files, or from a library.
The function we defined was called todo.routes/index, and therefore
we might assume that we’d have a matching key in the configuration.
{:todo.routes/index {}}This component key could then be connected to the routes via a ref. In other words:
{:duct.module/web {:routes [["/" {:get #ig/ref :todo.routes/index}]]}
:todo.routes/index {}}And in fact, this is almost exactly what is going on behind the scenes.
The Duct web module expands out to a great number of components, including a web server, middleware and error handlers, all which can be customized. Amongst these components, it creates a router and a number of route handlers.
A web module configured the following routes:
{:duct.module/web {:routes [["/" {:get :todo.routes/index}]]}}Will expand out to:
{:duct.router/reitit {:routes [["/" {:get #ig/ref :todo.routes/index}]]}
:todo.routes/index {}}The router component uses Reitit, a popular data-driven routing library for Clojure. Other routing libreries can be used, but for this documentation we’ll use the default.
Let’s take a closer look at function associated with the route.
(ns todo.routes)
(defn index [_options]
(fn [_request]
[:html {:lang "en"}
[:head [:title "Hello World Wide Web"]]
[:body [:h1 "Hello World Wide Web"]]]))This function returns another function, known as a Ring handler. Usually this function will return a response map, but in this case we’re returning a Hiccup vector.
Hiccup is a format for representing HTML as a Clojure data structure. Elements are represented by a vector starting with a keyword, followed by an optional attribute map and then the element body.
The :site feature of the web module adds middleware to turn Hiccup
vectors into HTML response maps. If the response is a vector, it wraps
the vector in response map. If the response is already a map, it checks
the :body of the response for a vector.
If we wanted a custom status code or headers, then the full response map could be returned.
(defn index [_options]
(fn [_request]
{:status 200
:headers {}
:body [:html {:lang "en"}
[:head [:title "Hello World Wide Web"]]
[:body [:h1 "Hello World Wide Web"]]]))|
Note
|
The :status and :headers keys map optionally be omitted.
|
Or we could return the string directly:
(defn index [_options]
(fn [_request]
{:status 200
:headers {"Content-Type" "text/html;charset=UTF-8"}
:body "<!DOCTYPE html>
<html lang=\"en\">
<head><title>Hello World Wide Web</title></head>
<body><h1>Hello World Wide Web</h1></body>
</html>"}))All of these examples are equivalent, but returning a vector is the most convenient and concise.
The next step is to add a database to our application. We’ll use SQLite, which means we need the corresponding JDBC adapter as a dependency.
To give us a Clojure-friendly way of querying the database, we’ll also add a dependency on next.jdbc.
Finally, we’ll add the Duct SQL module. This will add a connection pool to the system that we can use to access the database.
Our project dependencies should now look like this:
{:deps {org.clojure/clojure {:mvn/version "1.12.0"}
org.duct-framework/main {:mvn/version "0.1.5"}
org.duct-framework/module.logging {:mvn/version "0.6.5"}
org.duct-framework/module.web {:mvn/version "0.12.0"}
org.duct-framework/module.sql {:mvn/version "0.7.1"}
org.xerial/sqlite-jdbc {:mvn/version "3.47.0.0"}
com.github.seancorfield/next.jdbc {:mvn/version "1.3.955"}}
:aliases {:duct {:main-opts ["-m" "duct.main"]}}}We can load these new dependencies either by restarting the REPL, or by
using the sync-deps function.
user=> (sync-deps)
[...]The next step is to add :duct.module/sql to our Duct configuration.
{:system
{:duct.module/logging {}
:duct.module/sql {}
:duct.module/web
{:features #{:site}
:routes [["/" {:get :todo.routes/index}]]}}}Then reset via the REPL:
user=> (reset)
:reloading ()
Execution error (ExceptionInfo) at integrant.core/unbound-vars-exception (core.cljc:343).
Unbound vars: jdbc-urlWait, what’s this about an unbound var? Where did that come from?
Modules can add vars, and the SQL module adds one called jdbc-url.
This var can be set via:
-
A command-line argument,
--jdbc-url -
An environment variable,
JDBC_DATABASE_URL
We can also set a default value for this var via the configuration. As SQLite uses a local file for its database, we can add a default to be used in development.
{:vars {jdbc-url {:default "jdbc:sqlite:todo.db"}}
:system
{:duct.module/logging {}
:duct.module/sql {}
:duct.module/web
{:features #{:site}
:routes [["/" {:get :todo.routes/index}]]}}}If we want to change this in production, we can use the corresponding command-line argument or environment variable to override this default.
user=> (reset)
:reloading ()
:user/added (db sql)
:resumed|
Note
|
The :user/added message informs you about convenience functions
that have been added to the REPL environment in the user namespace.
|
The SQL module adds a database connection pool under the key
:duct.database.sql/hikaricp, which derives from the more general
:duct.database/sql key. We can use this connection pool as a
javax.sql.DataSource instance.
In order to give our route handlers access to this, we’ll use a ref. We could manually add the ref to each of the handler’s option map, as shown below.
{:todo.routes/index {:db #ig/ref :duct.database/sql}This is useful if only some routes need to access the database. However,
in this case, we expect that all routes will need database access in
some fashion. To make this easier, the web module has an option,
:handler-opts that applies common options to all route handlers it
generates.
{:vars {jdbc-url {:default "jdbc:sqlite:todo.db"}}
:system
{:duct.module/logging {}
:duct.module/sql {}
:duct.module/web
{:features #{:site}
:handler-opts {:db #ig/ref :duct.database/sql}
:routes [["/" {:get :todo.routes/index}]]}}}This will add the DataSource instance to the :db key of the
component options. We can access this from the route handler function we
created earlier.
(ns todo.routes)
(defn index [{:keys [db]}]
(fn [_request]
[:html {:lang "en"}
[:head [:title "Hello World Wide Web"]]
[:body [:h1 "Hello World Wide Web"]]]))Before we go further, however, we should set up the database schema via a migration.
Part of the SQL module is to add a migrator, a component that will
manage database migrations. By default the
Ragtime library is used, and
looks for a migrations.edn file in your project directory.
Let’s create a migration for a table to store the todo list items.
[[:create-table todo
[id "INTEGER PRIMARY KEY"]
[description "TEXT"]
[checked "INTEGER DEFAULT 0"]]]When we reset the REPL, the migration is automatically applied.
user=> (reset)
:reloading (todo.routes)
:duct.migrator.ragtime/applying {:id "create-table-todo#336f15d4"}
:resumedIf the migration is modified in any way, its ID will also change. At the REPL, this will result in the old version of the migration being rolled back, and the new version applied in its place.
Running the application via --main will also apply any new migrations
to the database. However, if there is any mismatch between migrations,
an error will be raised instead.
This difference reflects the environments that --main and --repl are
anticipated to be used in. During development a REPL is used and
mistakes are expected, so the migrator will work to sync the migrations
with the database. During production migrations need to be applied with
more care, and so any discrepancies should halt the migration process.
In some production environments, there may be multiple instances of the
application running at any one time. In these cases, you may want to run
the migrations separately. The --keys option allows you to limit the
system to a subset of keys. We can use this option to run only the
migrations and logging subsystems.
$ duct --main --keys=:duct/migrator:duct/loggerThis will run any component with a key that derives from
:duct/migrator or :duct/logger, along with any mandatory dependants.
|
Note
|
:duct/logger is often defined as an optional dependency, via a
refset. Without explicitly specifying this as one of the keys, the
migrator will run without logging.
|
Now that we have a database table and a web server, it’s time to put the two together. The database we pass to the index function can be used to populate an unordered list. We’ll change the index function accordingly.
(ns todo.routes
(:require [next.jdbc :as jdbc]))
(def list-todos "SELECT * FROM todo")
(defn index [{:keys [db]}]
(fn [_request]
[:html {:lang "en"}
[:head [:title "Todo"]]
[:body
[:ul (for [rs (jdbc/execute! db [list-todos])]
[:li (:todo/description rs)])]]]))|
Tip
|
It’s often a good idea to factor out each SQL string into its own
var. This allows them to be treated almost like function calls when
combined with execute!.
|
We can reset via the REPL and add some test data with the sql
convenience function.
user=> (reset)
:reloading (todo.routes)
:resumed
user=> (sql "INSERT INTO todo (description) VALUES ('Test One')")
[#:next.jdbc{:update-count 1}]
user=> (sql "INSERT INTO todo (description) VALUES ('Test Two')")
[#:next.jdbc{:update-count 1}]If you visit http://localhost:3000/ you’ll be able to see the todo items that were added to the database table.
The next step is to allow for new todo items to be added through the web interface. This is a little more involved, as we’ll need a HTML form and a route to respond to the form’s POST.
First, we add a new handler, new-todo, to the configuration to handle
the POST.
{:vars {jdbc-url {:default "jdbc:sqlite:todo.db"}}
:system
{:duct.module/logging {}
:duct.module/sql {}
:duct.module/web
{:features #{:site}
:handler-opts {:db #ig/ref :duct.database/sql}
:routes [["/" {:get :todo.routes/index
:post :todo.routes/new-todo}]]}}}Then we need incorporate the POST handler and the form into the codebase.
(ns todo.routes
(:require [next.jdbc :as jdbc]
[ring.middleware.anti-forgery :as af]))
(def list-todos "SELECT * FROM todo")
(def insert-todo "INSERT INTO todo (description) VALUES (?)")
(defn- create-todo-form []
[:form {:action "/" :method "post"}
[:input {:type "hidden"
:name "__anti-forgery-token"
:value af/*anti-forgery-token*}]
[:input {:type "text", :name "description"}]
[:input {:type "submit", :value "Create"}]])
(defn index [{:keys [db]}]
(fn [_request]
[:html {:lang "en"}
[:head [:title "Todo"]]
[:body
[:ul
(for [rs (jdbc/execute! db [list-todos])]
[:li (:todo/description rs)])
[:li (create-todo-form)]]]]))
(defn new-todo [{:keys [db]}]
(fn [{{:keys [description]} :params}]
(jdbc/execute! db [insert-todo description])
{:status 303, :headers {"Location" "/"}}))There are two new additions here. The create-todo-form function
creates a form for making new todo list items. You may notice that it
includes a hidden field for setting an anti-forgery token. This prevents
a type of attack known as a
Cross-site
request forgery.
The second addition is the new-todo function. This inserts a new row
into the todo table, then returns a “303 See Other” response that will
redirect the browser back to the index page.
If we reset via the REPL and check http://localhost:3000/, you should see a text input box at the bottom of the todo list, allowing more todo items to be added.