technical blog

29 May 2017

Building Web Services with Duct

If you’re new to the 0.9 alpha version of Duct, I suggest you take a detour and read Advancing Duct, which covers the new architecture of Duct more completely.

By contrast, this post won’t cover old ground, but instead focus on a more specific (but very common) use-case: web services. I’ll also introduce Ataraxy, a data-driving routing library that integrates well with Duct.

Let’s start by creating a new Duct project, and we’ll use the +api, +ataraxy and +sqlite profile hints to speed things up:

$ lein new duct-alpha foo +api +ataraxy +sqlite
Generating a new Duct project named foo...
Run 'lein duct setup' in the project directory to create local config files.

Let’s enter the project, and run the setup script to generate the local files that are kept out of source control:

$ cd foo
$ lein duct setup
Created profiles.clj
Created .dir-locals.el
Created dev/resources/local.edn
Created dev/src/local.clj

Next, we start a REPL:

$ lein repl

And initiate the system for development:

user=> (dev)
dev=> (go)
:duct.server.http.jetty/starting-server {:port 3000}

We can use another terminal to check that the server is running:

$  curl http://localhost:3000/

We get a “not found” error message, but this is expected as we haven’t set up any routes. By default it returns JSON data, but we can choose to accept edn instead by adding in an Accept header:

$ curl -H Accept:application/edn http://localhost:3000/
{:error :not-found}

Let’s take a look at the config.edn file for this project:

{:duct.core/project-ns  foo
 :duct.core/environment :production

 :duct.module/logging {}
 :duct.module.web/api {}
 :duct.module/sql {}


Instead of the :duct.module/web module, we’re using the more specific :duct.module.web/api. That gives us content negotiation with JSON and edn.

There’s also a :duct.module/ataraxy module. That’s where we’re going to add some routes, but before we do that, let’s attend to the database first and add a couple of new keys to our configuration:

{:migrations [#ig/ref :foo.migration/create-things]}

[:duct.migrator.ragtime/sql :foo.migration/create-things]
{:up   ["CREATE TABLE things (id INTEGER PRIMARY KEY, name TEXT)"]
 :down ["DROP TABLE things"]}

This create a migration that generates a things table. To run this migration, we just reset:

dev=> (reset)
:reloading (foo.main dev user)
:duct.migrator.ragtime/applying :foo.migration/create-things#d811dff6

Our migration is loaded, and our things table is now in our development database.

One important feature of migrations is that until they’re committed to the production database, we can edit them freely. Say we wanted to add a quantity column. We could just change the migration accordingly:

[:duct.migrator.ragtime/sql :foo.migration/create-things]
{:up   ["CREATE TABLE things (id INTEGER PRIMARY KEY, name TEXT, quantity INTEGER)"]
 :down ["DROP TABLE things"]}

When we run reset, Duct will roll back the older version of the migration, then apply the newer version:

dev=> (reset)
:reloading ()
:duct.migrator.ragtime/rolling-back :foo.migration/create-things#d811dff6
:duct.migrator.ragtime/applying :foo.migration/create-things#ae0427b0

Next, let’s add some basic routes to access and update the data in this table:

 {[:post {thing :body-params}] [:thing/create thing]
  [:get]                       [:thing/list]
  [:get "/" id]                [:thing/fetch ^int id]}}

This sets up the routes. Notice that each key in the map denotes a route to be matched, and each value denotes a result. The result collects data matched from the route, and in some cases transforms it. For example, the id matched in the route is coerced into an integer.

Now we have the routes, we need to add keys for the corresponding handler functions:

:foo.handler.thing/create {:db #ig/ref :duct.database/sql}
:foo.handler.thing/list   {:db #ig/ref :duct.database/sql}
:foo.handler.thing/fetch  {:db #ig/ref :duct.database/sql}

All of these handlers depend on the database, but in other cases we might have routes that depend on different resources, such as caches, queues, or mailers.

The Ataraxy module is smart enough to know that the routing result :thing/* should be connected to the handler function created by the :foo.handler.thing/* key.

So far we’ve just been writing configuration, but now it’s time to write some actual code. We start by creating a foo.handler.thing namespace and adding some essential requires:

(ns foo.handler.thing
  (:require [ataraxy.response :as response]
            [ :as jdbc]
            [integrant.core :as ig]))

The next step is to define our boundary protocol. We could put this in a separate namespace (like foo.boundary.thing), but because this boundary is used by only one handler, we instead place it in the handler namespace:

(defprotocol Things
  (create-thing [db thing])
  (list-things  [db])
  (fetch-thing  [db id]))

We can implement these functions in various ways. Clojure has a variety of different SQL libraries, such as HoneySQL for those who like to write SQL in Clojure, and HugSQL for those who like to write SQL in SQL.

In this example, we’ll just use the no-frills library:

(extend-protocol Things
  (create-thing [{db :spec} thing]
    (val (ffirst (jdbc/insert! db :things thing))))
  (list-things [{db :spec}]
    (jdbc/query db ["SELECT * FROM things"]))
  (fetch-thing [{db :spec} id]
    (first (jdbc/query db ["SELECT * FROM things WHERE id = ?" id]))))

The boundary defines the grey area between where our application logic ends and the database logic begins. We can also mock the boundary in tests, so that our handler tests can be isolated from the database tests.

Speaking of handlers, it’s time we implement them:

(defmethod ig/init-key ::create [_ {:keys [db]}]
  (fn [{[_ thing] :ataraxy/result}]
    (let [id (create-thing db thing)]
      [::response/created (str "/things/" id)])))

(defmethod ig/init-key ::list [_ {:keys [db]}]
  (fn [_] [::response/ok (list-things db)]))
(defmethod ig/init-key ::fetch [_ {:keys [db]}]
  (fn [{[_ id] :ataraxy/result}]
    (if-let [thing (fetch-thing db id)]
      [::response/ok thing]
      [::response/not-found {:error :not-found}])))

Ataraxy stores the result of the route match in the :ataraxy/result key on the request map. The URL “/things/10”, for example, would have the result [::fetch 10]. We can pull the result out of the request map with some simple destructuring.

It’s important to note that this destructuring serves to filter out unnecessary information. We know that only the :ataraxy/result key matters, and when testing we can omit every other part of the request map.

Similarly, Ataraxy allows variants (vectors that contain a key/value pair) to be returned instead of response maps. This makes the return value easier to check in the REPL and to write tests around.

Let’s test this works. We first create a new resource:

$ curl -X POST -H Content-Type:application/json \
   -d '{"name": "Book", "quantity": 10}' \

Then by check the created resource is in the list:

$ curl http://localhost:3000/things

And the resource has it’s own URI:

$ curl http://localhost:3000/things/1

Looks like everything works as expected.

As Duct gains more modules, we can expect the process of creating an API to become more automated. Because Duct is a framework based around a data structure, we can extensively automate common operations, without losing the ability to customize when necessary.

As you build an API with Duct, keep modules in mind. If you find yourself writing very similar sets of handlers and routes, there may be a simple way of factoring out the code into a module.

For more about modules, read the duct/core documentation.