boolean·knot

09 May 2017

Advancing Duct

Duct started life as a project template for writing web applications in Clojure, based around Stuart Sierra’s Component micro-framework. The next version of Duct replaces Component with Integrant, a micro-framework that I developed with Duct in mind.

The core structure of Duct remains the same; it still makes use of the reloaded workflow described by Stuart Sierra, and it still has a central configuration that defines the running system. However, the introduction of Integrant has allowed Duct to go much further. The latest alpha introduce several new and powerful concepts that transform Duct from a template into a fully-fledged framework, similar in scope to Arachne.

The next section will walk you through the steps of creating a project with Duct 0.9.0-alpha7, and from there we’ll cover the key concepts of Duct’s operation.

Setting up

Let’s start by using Leiningen to create a new duct-alpha template called “foo”:

$ lein new duct-alpha foo
Generating a new Duct project named foo...
Run 'lein duct setup' in the project directory to create local config files.

This creates a new directory called “foo”, and suggests we run a command in the project directory. Let’s do just that:

$ cd foo
$ lein duct setup
Created profiles.clj
Created .dir-locals.el
Created dev/resources/local.edn
Created dev/src/local.clj

This command generates local files that should be kept out of source control. These are not necessary to run the application, but allow the developer to add local customisations.

Next, we run the application:

$ lein run
17-05-09 09:10:11 localhost REPORT [duct.server.http.jetty:15] - :duct.server.http.jetty/starting-server {:port 3000}

Note that the logs are written to STDOUT, as recommended by the Twelve-Factor App methodology. Rather than use English or another natural language for logs, Duct uses an event keyword followed by an optional map of data.

The logs tell us the server has started on port 3000, but we can test that by sending a HTTP request with curl or a similar tool:

$ curl http://localhost:3000
Resource Not Found

We get a 404 response, but this is expected; we have yet to actually set up any routes.

Before we move on, let’s first terminate the running server with ctrl-c.

Pulling back the curtain

So what’s happening behind the scenes when we execute lein run? Let’s take a look at src/foo/main.clj:

(ns foo.main
  (:gen-class)
  (:require [clojure.java.io :as io]
            [duct.core :as duct]))

(defn -main [& args]
  (duct/exec (duct/read-config (io/resource "foo/config.edn"))))

We can see there’s not a lot here. We find the resource at foo/config.edn, read a configuration from it, then execute the configuration.

In Duct, the configuration doesn’t just store a map of options; it also defines the structure of the application. The configuration tells us what to run, as well as how to run it. This differs from most Clojure applications where the program structure is defined by code.

Let’s look at the mysterious configuration file located in resources/foo/config.edn:

{:duct.core/project-ns  foo
 :duct.core/environment :production

 :duct.module/logging {}
 :duct.module/web     {}

 :duct.router/cascading
 []}

It’s concise, but it also doesn’t show us much of the structure. What part of this map actually creates the HTTP server?

To understand how this works we need a REPL to help us. First we open one up:

$ lein repl

Then we switch to the dev namespace:

user=> (dev)
:loaded
dev=>

By separating the dev namespace from user, we can guarantee that the REPL will always load.

Next we run the prep function:

dev=> (prep)
:prepped

This function loads the configuration and prepares it for execution. The prepped configuration can be found in the config var. Here’s the contents of that var, cleaned up and formatted:

{:duct.core/project-ns  foo
 :duct.core/environment :development
 :duct.core/include     ["foo/config" "dev"]
 
 :duct.module/logging {}
 :duct.module/web     {}

 :duct.logger/timbre
 {:level     :debug
  :appenders {:duct.logger.timbre/spit  #ig/ref :duct.logger.timbre/spit
              :duct.logger.timbre/brief #ig/ref :duct.logger.timbre/brief}}
              
 :duct.logger.timbre/spit  {:fname "logs/dev.log"}
 :duct.logger.timbre/brief {:min-level :report}
 
 :duct.core/handler
 {:router     #ig/ref :duct/router
  :middleware [#ig/ref :duct.middleware.web/not-found
               #ig/ref :duct.middleware.web/defaults
               #ig/ref :duct.middleware.web/log-requests
               #ig/ref :duct.middleware.web/log-errors
               #ig/ref :duct.middleware.web/stacktrace]}

 :duct.server.http/jetty
 {:port    3000
  :handler #ig/ref :duct.core/handler
  :logger  #ig/ref :duct/logger}
  
 :duct.handler.static/bad-request
 {:headers {"Content-Type" "text/plain; charset=UTF-8"}
  :body    "Bad Request"}

 :duct.handler.static/not-found
 {:headers {"Content-Type" "text/plain; charset=UTF-8"}
  :body    "Not Found"}

 :duct.handler.static/method-not-allowed
 {:headers {"Content-Type" "text/plain; charset=UTF-8"}
  :body    "Method Not Allowed"}

 :duct.handler.static/internal-server-error
 {:headers {"Content-Type" "text/plain; charset=UTF-8"}
  :body    "Internal Server Error"}

 :duct.middleware.web/not-found
 {:error-handler #ig/ref :duct.handler.static/not-found}

 :duct.middleware.web/hide-errors
 {:error-handler #ig/ref :duct.handler.static/internal-servererror}

 :duct.middleware.web/log-requests {:logger #ig/ref :duct/logger}
 :duct.middleware.web/log-errors   {:logger #ig/ref :duct/logger}
 
 :duct.middleware.web/stacktrace {}
 
 :duct.middleware.web/defaults
 {:params    {:urlencoded true, :keywordize true}
  :responses {:not-modified-responses true
              :absolute-redirects     true
              :content-types          true
              :default-charset        "utf-8"}}

 :duct.router/cascading []}

You can see that this configuration is far larger than the one we saw in foo/config.edn. There are many additional keys, including ones that describe the logger (:duct.logger/timbre) and the web server (:duct.server.http/jetty). There are also references between keys, which are denoted by the #ig/ref tag.

These keys were added mostly by the modules, :duct.module/logger and :duct.module/web. Modules are pure functions that extend the configuration they’re in with additional keys and structure. Modules are one of the advantages to having a program structure defined in data.

To run this configuration, we use the init function:

dev=> (init)
:duct.server.http.jetty/starting-server {:port 3000}
:initiated
dev=>

This initiates the configuration. The Integrant method init-key is called for each entry in the configuration map. Some keys will initiate into functions, like :duct.core/handler, while others will have side effects like :duct.server.http/jetty.

Keep the REPL open for the next section.

Customizing the configuration

Modules are powerful, but they are also well-behaved. They are guaranteed to be pure, and guaranteed to never remove keys from the configuration. Your own keys should have precedence over any keys added by the module.

We can test this by changing the message set by the 404 error handler. Open up resources/foo/config.edn again and add a :duct.handler.error/not-found key:

{:duct.core/project-ns  foo
 :duct.core/environment :production

 :duct.module/logging {}
 :duct.module/web     {}

 :duct.router/cascading
 []

 :duct.handler.static/not-found
 {:body "I can't find what you want!"}}

In your REPL, run the reset function:

dev=> (reset)
:reloading (foo.main dev user)
:resumed
dev=>

The 404 message has now changed. We can check this with curl:

$ curl http://localhost:3000
I can't find what you want!

Resetting the system will reload the configuration and any namespaces that have changed.

Often it’s necessary to configure the development environment differently to the production environment. When we use lein run, we use the configuration at resources/foo/config.edn, but when we run the application from the REPL, the dev/resources/dev.edn file is used instead:

{:duct.core/environment :development
 :duct.core/include ["foo/config"]}

This is even smaller than the production configuration, but that’s fine because the :duct.core/include key will merge in a vector of configuration resources. Keys in the base configuration will override keys from included configurations, so the :duct.core/environment key in dev.edn will override the one in config.edn.

We can go one step further. There’s also a dev/resources/local.edn file that was created when we ran lein duct setup. This file is kept out of source control, and allows you to make local changes to the configuration.

By default, however, local.edn just includes dev.edn and does nothing else:

{:duct.core/include ["dev"]}

Adding routes

Now that we have an overview of the configuration, we can try adding a route. The default and most basic router is :duct.router/cascading. This router takes an ordered vector of handler functions, and returns the first response that isn’t nil.

We start by creating a new key, :foo.handler/example, that will contain our route. This key is linked to the router using the #ig/ref tag:

{:duct.core/project-ns  foo
 :duct.core/environment :production

 :duct.module/logging {}
 :duct.module/web     {}

 :duct.router/cascading
 [#ig/ref :foo.handler/example]

 :duct.handler.error/not-found
 {:response "I can't find what you want!"}

 :foo.handler/example {}}

Now that we have an entry in the configuration, we need a corresponding multimethod to supply the code.

Duct uses the name of the key to guess where the multimethod is defined. For the key :foo.handler/example, Duct will first try the foo.handler namespace, and then the foo.handler.example namespace. The latter sounds like a good choice, so let’s create a new file at src/foo/handler/example.clj:

(ns foo.handler.example
  (:require [compojure.core :refer [GET]]
            [integrant.core :as ig]))

(defmethod ig/init-key :foo.handler/example [_ _]
  (GET "/" [] "Hello World"))

We use Compojure to construct our route, because it will return nil if the route doesn’t match. This makes it compatible with :duct.router/cascading.

To load this change, we call reset:

dev=> (reset)
:reloading (foo.handler.example)
:resumed

And we can see that our new namespace has been loaded. If we use curl to access the web server, we can see our route:

$ curl http://localhost:3000
Hello World
$ curl http://localhost:3000/invalid
I can't find what you want!

Connecting to a database

Most web applications have some sort of database. To demonstrate how this works in Duct, we’ll connect our Duct application up to a SQLite database.

To do this, we first need some additional dependencies, and adding dependencies is one of the few times we need to exit the REPL:

dev=> (exit)
Bye for now!

Two dependencies, one for the module and one for the drive, need to be added to the project file:

[duct/module.sql "0.2.0"]
[org.xerial/sqlite-jdbc "3.16.1"]

Then restart the REPL:

$ lein repl

And load in the dev namespace:

user=> (dev)
:loaded
dev=>

Earlier, we used prep to read and prepare the configuration and init to initate it. Because these two steps are commonly executed together, a third function is provided, go, that runs prep then init:

dev=> (go)
:duct.server.http.jetty/starting-server {:port 3000}
:initiated

Now that we have the right dependencies, we can add the :duct.module/sql key to the configuration map:

{:duct.core/project-ns  foo
 :duct.core/environment :production

 :duct.module/logging {}
 :duct.module/web     {}
 :duct.module/sql     {}

 ...}

By default the SQL module takes the database connection from the DATABASE_URL environment variable. That’s good for production, but for development we want to override this, either in dev.edn or local.edn.

Since SQLite is self-contained, there’s no reason to keep it out of source control. Let’s extend dev.edn:

{:duct.core/environment :development
 :duct.core/include ["foo/config"]

 :duct.module/sql {:database-url "jdbc:sqlite:dev.sqlite"}}

Reset the application:

user=> (reset)
:reloading (foo.main foo.handler.example dev user)
:resumed
dev=>

We’re not using the database yet, but we can see that the dev.sqlite file has been created in the project directory:

$ ls *.sqlite
dev.sqlite

Now we have a database connection set up, we can use it in our handler. We first update the handler configuration in foo/config.edn to add a reference to the :duct.database/sql key:

:foo.handler/example {:db #ig/ref :duct.database/sql}

Next we update our handler code. Let’s change the web application to display a list of table names in our SQL database. We could just write a function, but the preferred way to do this in Duct is to write a boundary protocol.

Let’s create a new file called src/foo/boundary/tables.clj:

(ns foo.boundary.tables
  (:require [clojure.java.jdbc :as jdbc]
            [duct.database.sql]))

(defprotocol Tables
  (get-tables [db]))

(extend-protocol Tables
  duct.database.sql.Boundary
  (get-tables [{:keys [spec]}]
    (jdbc/query spec ["SELECT name FROM sqlite_master WHERE type = 'table'"])))

This extends the duct.database.sql.Boundary record and gives it a new protocol method. This has two benefits:

1. It clearly delimits where our application ends and where the SQL database begins.
2. We can use a tool like Shrubbery to create a mock or stub for the protocol for testing.

Once we have our boundary, we can add it to our handler:

(ns foo.handler.example
  (:require [compojure.core :refer [GET]]
            [foo.boundary.tables :as tables]
            [integrant.core :as ig]))

(defmethod ig/init-key :foo.handler/example [_ {:keys [db]}]
  (GET "/" []
    (pr-str (tables/get-tables db))))

Then reset the system:

dev=> (reset)
:reloading (foo.boundary.tables foo.handler.example)
:resumed

And now curl will return the results of our database query:

$ curl http://localhost:3000/
({:name "ragtime_migrations"})

Wait a minute; where did that ragtime_migrations table come from?

Migrating the database

The :duct.module/sql key does more than set up a database connection; it also adds a database migrator based on Ragtime.

As the final part of this post, we’ll create a database migration to add a new table. To do this, we need to add some more keys to foo/config.edn:

:duct.migrator/ragtime
{:migrations [#ig/ref :foo.migration/example]}

[:duct.migrator.ragtime/sql :foo.migration/example]
{:up ["CREATE TABLE example (id int)"]
 :down ["DROP TABLE example"]}

There are two keys here, but one of them is a vector. Vectors denote composite keys in Integrant, and are often used to give a unique identifier to keys we want to use multiple times.

In this case, the base key is :duct.migrator.ragtime/sql, which denotes a SQL migration. Because it’s likely we’ll want multiple migrations eventually, we make the key unique by adding :foo.migration/example as an identifier.

In order to apply this migration, we call our old friend, reset:

dev=> (reset)
:reloading ()
:duct.migrator.ragtime/applying :foo.migration/example#88bcf00d
:resumed

We can see the migration has been applied, and we can check it with curl:

$ curl http://localhost:3000/
({:name "ragtime_migrations"} {:name "example"})

Migrations are applied whenever we init or reset. Rather than manually migrate and rollback migrations, Duct does that for us. For example, let’s say we change the table name from example to users:

[:duct.migrator.ragtime/sql :foo.migration/example]
{:up ["CREATE TABLE users (id int)"]
 :down ["DROP TABLE users"]}

The migrator is smart enough to recognize the change and act accordingly when we reset:

dev=> (reset)
:reloading ()
:duct.migrator.ragtime/rolling-back :foo.migration/example#88bcf00d
:duct.migrator.ragtime/applying :foo.migration/example#f0d81966
:resumed

In the development environment, the old migration is rolled back and the new one applied. In a production environment an error would be raised instead; in production, we only migrate, we don’t rollback.

Conclusion

This post doesn’t cover all of Duct’s features, and Duct is still in early alpha testing. However, it’s hopefully given you a basic understanding about what Duct can do and how it’s structured.

Questions about the Duct alpha can be raised in the Google Group or as an issue. You can also contact me directly.

If you’re planning on using the Duct alpha for a project, either commercially or just to try it out, I’d be interested in your comments and feedback.