boolean·knot

technical blog

15 July 2016

Asynchronous Ring

You might already know about the current proposal to introduce support for asynchronous handlers in Ring. In this post I’m going to try to summarise the proposed design, and talk a little about why it was chosen. I can’t cover everything, but I’ll try to provide an overview.

Asynchronous handlers

In previous versions of Ring, handlers were written:

(defn handler [request]
  {:status 200, :headers {}, :body "Hello World"})

In the current beta, handlers can also be written to take two additional arguments, respond and raise. Responses are delivered via the respond function, rather than the function’s return value. Exceptions are reported via raise, rather than thrown.

(defn handler [request respond raise]
  (respond {:status 200, :headers {}, :body "Hello World"}))

Handlers can support both synchronous and asynchronous use by dispatching on arity:

(defn handler
  ([request]
   {:status 200, :headers {}, :body "Hello World"})
  ([request respond raise]
   (respond {:status 200, :headers {}, :body "Hello World"}))

Asynchronous middleware

One advantage of this design is that we can write Ring middleware to support both synchronous and asynchronous use with the same function.

In previous versions of Ring, middleware returned a synchronous handler:

(defn wrap-foo [handler value]
  (fn [request]
    (-> (handler request)
        (assoc-in [:headers "X-Foo"] value))))

In the current beta, middleware can return a handler that supports synchronous and asynchronous arities:

(defn foo-response [response value]
  (assoc-in response [:headers "X-Foo"] value))

(defn wrap-foo [handler value]
  (fn
    ([request]
     (foo-response (handler request) value))
    ([request respond raise]
     (handler request #(respond (foo-response % value)) raise))))

Because both synchronous and asynchronous use share functionality, we’ve factored out common behaviour in this example into a function called foo-response. By convention, anything modifying the request map should go into into a function named foo-request, and anything modifying the response goes into foo-response.

This convention has been used in Ring Core for a while in order to help support interceptors in Pedestal. If we adopt the same convention for asynchronous handlers in general, we can benefit the Pedestal ecosystem as well as Ring.

Streaming responses

In previous versions of Ring, there was a choice of four types for the response body: strings, seqs, files and input streams. The current beta expands this to include the StreamableResponseBody protocol, which allows you to define how a type is written to the response OutputStream.

For example, we can extend the protocol to cover core.async channels:

(extend-type ManyToManyChannel
  StreamableResponseBody
  (write-body-to-stream [channel response output-stream]
    (go (with-open [writer (io/writer output-stream)]
          (loop []
            (when-let [msg (<! channel)]
              (doto writer (.write msg) (.flush))
              (recur)))))))

This allows a channel to be used as a response body. The server will keep the response open until the channel is closed.

A handler that makes use of a channel body might look something like this:

(defn hello-world [request respond raise]
  (let [ch (chan)]
    (respond {:status 200, :headers {}, :body ch})
    (go (<! (a/timeout 1000))
        (>! ch "hello\n")
        (<! (a/timeout 1000))
        (>! ch "world\n")
        (a/close! ch))))

Upon receiving a response, the above example waits a second, sends “hello”, waits another second, then sends “world” and ends the response. The go-block returns immediately, so we only block a thread when we’re actively sending data to the client.

Non-blocking I/O

One omission to the Ring beta is support for non-blocking I/O, particularly around writing the response body. An OutputStream blocks until the write has finished, using up a thread while data is being transmitted. While this is better than using up a thread for the entire duration of the response, it’s still not perfect.

Fortunately there’s nothing stopping Ring from supporting multiple protocols, and allowing the adapter to choose the most suitable. For example, a String might satisfy both the current StreamableResponseBody protocol and a future NonBlockingResponseBody protocol. A synchronous adapter could choose the former, an asynchronous one the latter.

However, non-blocking I/O is tricky to implement in Ring, because all Java web servers have a different implementation. Ring will need to find a common protocol that can work with all of them, and so non-blocking I/O has been pushed back to future release. In the meanwhile, adapters can use specialised protocols that are tied to a particular web server. More on that later.

Alternative designs

Now that I’ve covered the changes to Ring, it’s worth briefly covering some alternative designs, and explaining why were not chosen for Ring.

Pedestal is a web framework that uses interceptors in place of middleware. An interceptor is asynchronous, and has the advantage of context; it both can see what other interceptors are queued, and even alter the queue dynamically. The cost is that interceptors are quite a bit more complex than middleware.

Interceptors were avoided largely because they aren’t compatible with Ring’s existing middleware. They wouldn’t be an enhancement to Ring’s handlers and middleware, but a complete replacement.

Aleph is a Ring-compatible web server that supports asynchronous behaviour. For this it uses Manifold, a library that provides asynchronous building blocks. Responses can be returned wrapped in Manifold deferreds, and the response body can be a Manifold stream.

Aleph’s model is backward compatible with Ring, but was was not used because it’s tied to Manifold. Manifold is an impressive library, but it’s also a sizeable dependency and has yet to reach version 1.0.0, making it too large and too risky to be adopted by Ring.

HTTP Kit is another Ring-compatible web server with asynchronous behaviour. Unlike Aleph it doesn’t use asynchronous promises, but instead exposes a channel for writing the response, bypassing the return value. This is similar to Ring’s asynchronous handlers, but HTTP Kit uses the same send! function for both delivering the response and for streaming the response body.

This design wasn’t chosen because it makes it difficult for middleware to handle the response, and because the send! function is too dependent on when it’s called; the first time it’s called on a channel is different to all subsequent times. In Ring, this is handled by two separate functions: the respond argument passed to the handler, and the write-body-to-stream method that’s part of the StreamableResponseBody protocol.

Compatibility

Ring’s asynchronous handlers are designed to be compatible with the broadest possible range of Java and Clojure web servers. Perhaps most importantly, Ring supports the latest stable Java Servlet specification (3.1) out of the box, making it compatible with the majority of Java web servers.

While it doesn’t adopt Aleph’s design, asynchronous Ring handlers can be automatically converted into asynchronous Aleph handlers (and vice versa):

(defn ring->aleph [handler]
  (fn [request]
    (let [response (d/deferred)]
      (handler request #(d/success! response %) #(d/error! response %))
      response)))

Similarly, we can write a compatibility layer for HTTP Kit as well:

(defn ring->httpkit [handler]
  (fn [request]
    (with-channel request channel
      (handler request #(send! channel %) (fn [_] (close channel))))))

When it comes to writing to the response body, adapters should support Ring’s StreamableResponseBody protocol. However, as mentioned previously, they can also add their own protocols, and use the StreamableResponseBody as a fallback.

For example, a future HTTP Kit release there could expose a protocol to support writing a type to the HTTP Kit channel:

(defprotocol ChannelResponseBody
  (write-body-to-channel [body response channel]))

(extend-protocol ChannelResponseBody
  String
  (write-body-to-channel [body _ channel]
    (send! channel body))
  clojure.lang.ISeq
  (write-body-to-channel [body _ channel]
    (doseq [x body]
      (send! channel (str x) false))
    (close channel)))

HTTP Kit could check if a type satisfies ChannelResponseBody before Ring’s native StreamableResponseBody. As HTTP Kit’s channels don’t block on write, we’d expect the more specific ChannelResponseBody protocol to be more performant, and therefore should be preferred by HTTP Kit.

Synchronous vs. Asynchronous

It’s hard to judge how many people will need asynchronous handlers. Even the most modest of web servers is capable of running thousands of threads, and even web applications with a significant number of users often need to support surprisingly low numbers of connections.

For example, in 2009, when Twitter had around 350,000 users, they dealt with an average of 200-300 connections per second, with spikes of 800. In this case, a thread per connection seems perfectly reasonable, even with only one web server, let alone a reasonably-sized cluster.

Zach Tellman points out that web services often fall under much heavier load than user-facing websites, and it’s not just the average load we need to worry about, but the peak. For instance, an API might come suddenly under considerable load as the result of an automated process. Most web servers buffer connections until a thread is available, but clients may still see significant spikes in latency because of this.

That said, in many cases the web server won’t be the performance bottleneck. Web servers can be scaled horizontally, while many databases cannot, so the bottleneck often ends up being the database. The cost of an additional web server is small compared to the cost of a developer’s time, so it’s generally cheaper to spin up another web server than it is to change existing code. While brute forcing a problem in this way may seem inelegant, trading CPU-time for development time is usually a good deal.

Aside from web services that expect significant load, technologies that rely on keeping the response open, such as Server-Sent Events or long polling, may also need to manage large numbers of simultaneous connections. The longer a connection is kept open, the more attractive asynchronous handlers become, even under relatively small loads.

I don’t expect asynchronous HTTP to be necessary for most web applications; but I do think there are enough cases where it could be useful that it’s worth adding optional support for them. If asynchronous HTTP was completely unnecessary, I don’t believe we’d see asynchronous support in web servers like Pedestal, Aleph or HTTP Kit.

The intent is for Ring’s asynchronous handlers to be a specialised tool. Their design makes it relatively straightforward to convert a Ring application that uses synchronous handlers into one that makes selective use of asynchronous handlers. This encourages beginning with a synchronous design, and then moving to an asynchronous one if performance becomes an issue.

Feedback

Ring 1.6.0 represents a significant change, so the beta-testing period will be long. There’s plenty of time to try it out and give feedback, either on the mailing list or in a private email. I’ll also do my best to respond to any comments on r/clojure.

If you want me to go into more detail on any of the aspects of the design, please feel free to ask. This post is intended to be a summary, rather than a complete explanation, and there’s plenty of detail I’ve had to omit or cover only briefly.