29 May 2017
Building Web Services with Duct
If you’re new to the 0.9 alpha version of Duct, I suggest you take a detour and read Advancing Duct, which covers the new architecture of Duct more completely.
By contrast, this post won’t cover old ground, but instead focus on a more specific (but very common) use-case: web services. I’ll also introduce Ataraxy, a data-driving routing library that integrates well with Duct.
Let’s start by creating a new Duct project, and we’ll use the +api,
+ataraxy and +sqlite profile hints to speed things up:
$ lein new duct-alpha foo +api +ataraxy +sqlite
Generating a new Duct project named foo...
Run 'lein duct setup' in the project directory to create local config files.
Let’s enter the project, and run the setup script to generate the local files that are kept out of source control:
$ cd foo
$ lein duct setup
Created profiles.clj
Created .dir-locals.el
Created dev/resources/local.edn
Created dev/src/local.clj
Next, we start a REPL:
$ lein repl
And initiate the system for development:
user=> (dev)
:loaded
dev=> (go)
:duct.server.http.jetty/starting-server {:port 3000}
:initiated
We can use another terminal to check that the server is running:
$ curl http://localhost:3000/
{"error":"not-found"}
We get a “not found” error message, but this is expected as we haven’t
set up any routes. By default it returns JSON data, but we can choose
to accept edn instead by adding in an Accept header:
$ curl -H Accept:application/edn http://localhost:3000/
{:error :not-found}
Let’s take a look at the config.edn file for this project:
{:duct.core/project-ns foo
:duct.core/environment :production
:duct.module/logging {}
:duct.module.web/api {}
:duct.module/sql {}
:duct.module/ataraxy
{}}
Instead of the :duct.module/web module, we’re using the more
specific :duct.module.web/api. That gives us content negotiation
with JSON and edn.
There’s also a :duct.module/ataraxy module. That’s where we’re going
to add some routes, but before we do that, let’s attend to the
database first and add a couple of new keys to our configuration:
:duct.migrator/ragtime
{:migrations [#ig/ref :foo.migration/create-things]}
[:duct.migrator.ragtime/sql :foo.migration/create-things]
{:up ["CREATE TABLE things (id INTEGER PRIMARY KEY, name TEXT)"]
:down ["DROP TABLE things"]}
This create a migration that generates a things table. To run this
migration, we just reset:
dev=> (reset)
:reloading (foo.main dev user)
:duct.migrator.ragtime/applying :foo.migration/create-things#d811dff6
:resumed
Our migration is loaded, and our things table is now in our
development database.
One important feature of migrations is that until they’re committed to
the production database, we can edit them freely. Say we wanted to add
a quantity column. We could just change the migration accordingly:
[:duct.migrator.ragtime/sql :foo.migration/create-things]
{:up ["CREATE TABLE things (id INTEGER PRIMARY KEY, name TEXT, quantity INTEGER)"]
:down ["DROP TABLE things"]}
When we run reset, Duct will roll back the older version of the
migration, then apply the newer version:
dev=> (reset)
:reloading ()
:duct.migrator.ragtime/rolling-back :foo.migration/create-things#d811dff6
:duct.migrator.ragtime/applying :foo.migration/create-things#ae0427b0
:resumed
Next, let’s add some basic routes to access and update the data in this table:
:duct.module/ataraxy
{"/things"
{[:post {thing :body-params}] [:thing/create thing]
[:get] [:thing/list]
[:get "/" id] [:thing/fetch ^int id]}}
This sets up the routes. Notice that each key in the map denotes a
route to be matched, and each value denotes a result. The result
collects data matched from the route, and in some cases transforms
it. For example, the id matched in the route is coerced into an
integer.
Now we have the routes, we need to add keys for the corresponding handler functions:
:foo.handler.thing/create {:db #ig/ref :duct.database/sql}
:foo.handler.thing/list {:db #ig/ref :duct.database/sql}
:foo.handler.thing/fetch {:db #ig/ref :duct.database/sql}
All of these handlers depend on the database, but in other cases we might have routes that depend on different resources, such as caches, queues, or mailers.
The Ataraxy module is smart enough to know that the routing result
:thing/* should be connected to the handler function created by
the :foo.handler.thing/* key.
So far we’ve just been writing configuration, but now it’s time to
write some actual code. We start by creating a foo.handler.thing
namespace and adding some essential requires:
(ns foo.handler.thing
(:require [ataraxy.response :as response]
[clojure.java.jdbc :as jdbc]
[duct.database.sql]
[integrant.core :as ig]))
The next step is to define our boundary protocol. We could put this in
a separate namespace (like foo.boundary.thing), but because this
boundary is used by only one handler, we instead place it in the
handler namespace:
(defprotocol Things
(create-thing [db thing])
(list-things [db])
(fetch-thing [db id]))
We can implement these functions in various ways. Clojure has a variety of different SQL libraries, such as HoneySQL for those who like to write SQL in Clojure, and HugSQL for those who like to write SQL in SQL.
In this example, we’ll just use the no-frills clojure.java.jdbc library:
(extend-protocol Things
duct.database.sql.Boundary
(create-thing [{db :spec} thing]
(val (ffirst (jdbc/insert! db :things thing))))
(list-things [{db :spec}]
(jdbc/query db ["SELECT * FROM things"]))
(fetch-thing [{db :spec} id]
(first (jdbc/query db ["SELECT * FROM things WHERE id = ?" id]))))
The boundary defines the grey area between where our application logic ends and the database logic begins. We can also mock the boundary in tests, so that our handler tests can be isolated from the database tests.
Speaking of handlers, it’s time we implement them:
(defmethod ig/init-key ::create [_ {:keys [db]}]
(fn [{[_ thing] :ataraxy/result}]
(let [id (create-thing db thing)]
[::response/created (str "/things/" id)])))
(defmethod ig/init-key ::list [_ {:keys [db]}]
(fn [_] [::response/ok (list-things db)]))
(defmethod ig/init-key ::fetch [_ {:keys [db]}]
(fn [{[_ id] :ataraxy/result}]
(if-let [thing (fetch-thing db id)]
[::response/ok thing]
[::response/not-found {:error :not-found}])))
Ataraxy stores the result of the route match in the :ataraxy/result
key on the request map. The URL “/things/10”, for example, would have
the result [::fetch 10]. We can pull the result out of the request
map with some simple destructuring.
It’s important to note that this destructuring serves to filter out
unnecessary information. We know that only the :ataraxy/result key
matters, and when testing we can omit every other part of the request
map.
Similarly, Ataraxy allows variants (vectors that contain a key/value pair) to be returned instead of response maps. This makes the return value easier to check in the REPL and to write tests around.
Let’s test this works. We first create a new resource:
$ curl -X POST -H Content-Type:application/json \
-d '{"name": "Book", "quantity": 10}' \
http://localhost:3000/things
Then by check the created resource is in the list:
$ curl http://localhost:3000/things
[{"id":1,"name":"Book","quantity":10}]
And the resource has it’s own URI:
$ curl http://localhost:3000/things/1
{"id":1,"name":"Book","quantity":10}
Looks like everything works as expected.
As Duct gains more modules, we can expect the process of creating an API to become more automated. Because Duct is a framework based around a data structure, we can extensively automate common operations, without losing the ability to customize when necessary.
As you build an API with Duct, keep modules in mind. If you find yourself writing very similar sets of handlers and routes, there may be a simple way of factoring out the code into a module.
For more about modules, read the duct/core documentation.