It all started with a problem

Back in 2014, Sönke Hahn, Julian Arni and myself were working together in “the Haskell team” at Zalora on all sorts of projects. Many of them involved serving web applications, querying external APIs or our own services from Haskell, PHP, JS and probably a few other languages. At the time, we were using a few of the well established “web frameworks”, among which scotty, whenever we had to offer some service over HTTP.

However, writing all those functions for hitting our own webservices was a lot of manual, error-prone, tedious work. The bigger web applications got, the more tedious it became. And it had to be done once per language in which we wanted to hit the application. This could not continue.

For reference, this is what a simple scotty application looks like:

{-# LANGUAGE OverloadedStrings #-}
import Data.Text (split)
import Web.Scotty

main :: IO ()
main = scotty 8000 $
  get "/repeat/:n" $ do
    n <- param "n"
    json (replicate n n)

  post "/message" $ do
    msg <- jsonData
    json (split "\n" msg)

How could we somewhat automate the creation of one client function per endpoint of the web application? In an ideal world, we would just show this application to some program or library and it would collect all the data it needs about the overall structure of the application from the code itself, in order to produce 2 client functions:

-- Client for the first endpoint.
--
-- The Int is the value you want to set ":n" to (/repeat/23, /repeat/10, ...).
getRepeat :: Int -> ClientMonad [Int]

-- Client for the second endpoint.
--
-- The JSON body (just a naked string) to send is the Text argument.
postMessage :: Text -> ClientMonad [Text]

which would do all the hard work of preparing an HTTP request for us, even taking care of JSON encoding and decoding for us. But… the entire structure of the application is just hidden in the do block and we just cannot programmatically access it.

So… this is not realistically doable. We clearly need to change (a little bit? a lot?) the way we write our applications, making sure we get a description of the web’s application structure (the endpoints, the part of the request they use or depend on, what they return) that we could then hand over to something, which would get us our client functions.

We will now try implementating such a web application description DSL in the most straightforward way possible.

Note: We could define a DSL with an interpreter that spits out Haskell code for us (through Template Haskell or another mechanism). The result would be typed, but the process would not be. The translation from the DSL to the result would be untyped code, therefore easy to get wrong. As Haskell programmers, we prefer static type checking where possible. This therefore excludes Template Haskell and other code generation mechanisms.

A first, non-modular attempt

We want to produce client functions that look like the ones above, that prepare and send HTTP requests for us by taking some pieces of data given as arguments to those functions and encoding then storing them in the right places of the request (request path for URL captures, request body, headers, etc). Let’s perhaps start simple with a data type for describing an endpoint that can be served under some path (which can contain static string fragments and captures), for a given http method, ignoring everything else for now.

It could look like this:

data Method = Get | Post

data Endpoint = Static String Endpoint
              | Capture Endpoint
              | Verb Method

-- GET /hello/:name
getHello :: Endpoint
getHello = Static "hello" (Capture (Verb Get))

and, if we want it to look a little more “servant-y”, we can define:

infixr 5 :>
(:>) :: (Endpoint -> Endpoint) -> Endpoint -> Endpoint
f :> x = f x

getHelloNew :: Endpoint
getHelloNew = Static "hello" :> Capture :> Verb Get

Unlike servant though, as you can see with the type of getHello and getHelloNew, our descriptions are good old Haskell values, both of the Endpoint type.

Given those few definitions, how could we go about, say, generating links to endpoints? Well, here is a straightforward attempt.

-- a link here is just a list of path components
-- (we ignore query parameters in this post)
type Link = [String]

linkTo :: Endpoint -> Link
linkTo (Static str rest) = str : linkTo rest
linkTo (Verb _method)    = []
linkTo (Capture rest)    = ??? : linkTo rest

But… what should we put in place of those ???, if anything?

Well, we definitely want to add some path component, to fill the Capture slot. However, by definition, a captured path fragment is not fixed, it is allowed to vary. In other words, Capture :> Verb Post matches both POST /x and POST /y. We cannot just pick one value and hope that it is the one the user wanted. We need to take it as an argument. But what about Capture :> Capture :> Verb Post? We would need our linkTo function to take 2 arguments for that case. And zero additional argument for Static "hello" :> Verb Post. This is quite problematic.

Indeed, we would like the type of linkTo to be Endpoint -> Link, Endpoint -> String -> Link, Endpoint -> String -> String -> Link and so on depending on what the Endpoint argument is. In other words, we want the return type of linkTo (when really seen as a function of one argument, which it is anyway) to depend on the value of type Endpoint it gets as input. That is, we want a type that depends on a value, i.e dependent types.

The alternative would be:

-- linkTo assumes that the capture values are given in the same order as we want
-- them to appear in the url.
linkTo :: Endpoint -> [String] -> Link
linkTo (Static str rest) captureValues  = str : linkTo rest captureValues
-- when there is at least one value left in the list, we use it.
-- when there isn't... we error out.
linkTo (Capture rest)    (c : cs)       = c   : linkTo rest cs
linkTo (Capture rest)    []             = error "linkTo: capture value needed but the list is empty" -- :-(
linkTo (Verb method)     _              = []

This solution is very unsatisfactory, first and foremost because it is possible for it to error out if we don’t supply the right number of captures. However, it will also silently let us pass too many capture values without telling us that some of them are not used. Such a function should be total, we really don’t want an implementation that can let us down if we’re not very careful.

Fortunately, GADTs can help here. We could turn Endpoint into a GADT that tracks captures and then use some type-level computations to get the type of the link-making function from our list of captures, as well as define the link making function through typeclass instances that would go through the captures and add an argument for each of them. Request bodies, query parameters, headers? We could probably track them too, in a similar way. Or we could unify it all by basically building up and tracking actual servant API types through a GADT version of Endpoint’s type argument, and do some of what servant does at the type-level, with everything else at the value-level.

However, all those approaches have a big problem. Once you’ve made a decision, it is set in stone, in a way. Indeed, in all these approaches, if you want to extend your web app description language, you need to add a new constructor to your Endpoint type. You then need to handle this new constructor in all the functions that pattern match on Endpoint constructors since they’ve instantly become partial. Not to mention that just the act of adding a constructor requires rebuilding the entire library. You cannot explore two different directions simultaneously without breaking code, you cannot add new constructs you hadn’t thought of without touching the library, e.g just locally in a project of yours. Extensibility and modularity were central requirements as we had been bitten by the lack of them in libraries that we were using at the time.

Note that a GADT-based approach would work well (in addition to being more approachable) for very stable domains, and is not considered here because of the kind of flexibility we are asking for.

So… how do people build extensible/modular DSLs in Haskell? The next section talks about the general problem behind this and a solution that I read about that gets us halfway to servant.

The Expression Problem

To quote Phil Wadler: “the expression problem is a new name for an old problem. The goal is to define a datatype by cases, where one can add new cases to the datatype and new functions over the datatype, without recompiling existing code, and while retaining static type safety (e.g., no casts)”.

In Haskell, the standard approach to representing some domain is to define an algebraic data type for it. For a simple type of expressions with additions and integers, we usually do:

data Expr = I Integer | Add Expr Expr

and proceed to write what we call “interpreters”, which in this case are just functions that take expressions as input and do something interesting with them.

eval :: Expr -> Integer
eval (I n)     = n
eval (Add a b) = eval a + eval b

prettyPrint :: Expr -> String
prettyPrint (I n) = show n
prettyPrint (Add a b) = unwords [prettyPrint a, "+", prettyPrint b]

So, given an expression type, we can easily “add new functions over the data type”, to reuse Phil’s wording. We just write a new function. However, when the time comes to “add new cases to the data type”, this approach becomes painful. A “new case” here means a new constructor for our Expr data type. Let’s say we want to support multiplications too:

data Expr = I Integer | Add Expr Expr | Mul Expr Expr

Now, we have to modify every single function that patterns matches on an Expr to handle the Mul constructor, including our eval and prettyPrint “interpreters”. For any non-trivial domain, this becomes very painful, very quickly. Fine, so what other options are there?

Ralf Lämmel’s slides on the topic have been of a great help for me, back when we were looking for a solution suitable to our needs. With Oleg Kiselyov, they show how we can reasonably easily (that is, in Haskell 98) achieve full extensibility in both directions (constructors and interpretations) in Haskell. It boils down to:

• Turn what would be a constructor into its own little data type.
• Turn what would be a simple function that operates on the data type into a typeclass with a method.
• Write instances of those typeclasses for the data types representing the DSL’s constructs.

This effectively means that we won’t have a single type to represent all the valid “endpoint descriptions”. Instead, with this approach, we will be able to process any “reasonable” combination of “endpoint components”. The Expr typeclass below is exactly what lets us say what is a valid endpoint description and what isn’t. Using their approach for our expressions would look like this:

-- our expression constructs, one data type per
-- constructor we had previously.

	-- integer constants
data I = I Integer

-- Since we don't have an 'Expr' type anymore, to use as a type for
-- the fields of Add, we just make them type parameters. Sometimes
-- 'l' and 'r' might be I, some other times they might be 'Add I I',
-- or 'Add (Add I I) I', and so on. The type reflects the recursive
-- structure.
data Add l r = Add l r

-- an "open union" to be able to describe all the
-- valid expression types.
class Expr a
instance Expr I
instance (Expr l, Expr r) => Expr (Add l r)

-- our first interpretation, evaluation
class Expr a => Eval a where
  eval :: a -> Integer

-- evaluating a constant amounts to returning it
instance Eval I where
  eval (I n) = n

-- if we know how to evaluate two things, we know how to evaluate
-- their addition
instance (Eval l, Eval r) => Eval (Add l r) where
  eval (Add a b) = eval a + eval b

-- our second interpretation, pretty printing
class Expr a => Pretty a where
  pretty :: a -> String

instance Pretty I where
  pretty (I n) = show n

instance (Pretty l, Pretty r) => Pretty (Add l r) where
  pretty (Add a b) = unwords [pretty a, "+", pretty b]

Every constructor that we had in our previous Expr data type is now turned into its own little type, and every interpretation becomes a type class that all those little types are then free to provide an instance for. In fact, we do not necessarily have to supply an instance of each interpretation for all of our constructs. If we try to interpret an expression that uses a construct not supported by this interpretation, we get a type error! This is much better than calling error in some corner cases that should in theory not be reached… In theory. Right.

Anyway, if we now want to add support for multiplications, we can do:

data Mul l r = Mul l r
instance (Expr l, Expr r) => Expr (Mul l r)

instance (Eval l, Eval r) => Eval (Mul l r) where
  eval (Mul a b) = eval a * eval b

instance (Pretty l, Pretty r) => Pretty (Mul l r) where
  pretty (Mul a b) = unwords [autoParens a, "*", autoParens b]
    where autoParens a@(Add _ _) = "(" ++ pretty a ++ ")"
          autoParens           a = pretty a

We didn’t have to change any existing function, that’s great! Let’s apply this approach to a very simplified web application description “language” that we could make out of tiny building blocks (static path fragments, captures, etc).

A first modular attempt

Adapting the approach from the previous section to our domain, we can give a shot at decomposing the kind of information we want to represent into a few different “constructs” (i.e data types).

-- static path fragments
data Static = Static String

-- variable path fragments ("captures")
data Capture = Capture

-- HTTP method
data Method = Get | Post
-- Leaf of a chain of :>'s, specifies the HTTP method
data Verb = Verb Method

-- chain a few "endpoint components" with this operator,
-- all chains must be terminated with a 'Verb' component.
infixr 5 :>
data a :> b = a :> b

-- a class to specify all the valid endpoint descriptions
class Endpoint a

-- Verb alone is one.
instance Endpoint Verb

-- if we have a valid description, sticking 'Static :>' in front of it
-- yields another valid description.
instance Endpoint rest => Endpoint (Static :> rest)

-- if we have a valid description, sticking 'Capture :>' in front of it
-- yields another valid description.
instance Endpoint rest => Endpoint (Capture :> rest)

-- GET /hello
endpoint1 = Static "hello" :> Verb Get

OK, why not. Let’s now try to write an interpretation for generating links to endpoints like the one above. This is a lot simpler and self-contained than investigating client generation or server-side routing, while retaining many of the difficulties. The main one is that depending on what we find in the description of the endpoint, we need the type of the link-generating function to change: indeed, if we encounter Captures, then the user has to supply values for them. We will let the user do that through one additional argument per Capture we encounter.

Let’s start with something really simple.

type Link = [String]

-- @renderLink ["hello", "world"] == "/hello/world"@
renderLink :: Link -> String
renderLink xs = '/' : intercalate '/' xs

class HasLink endpoint where
  -- return the path components
  link :: endpoint -> [String]

instance HasLink api => HasLink (Static :> api) where
  link (Static s :> api) = s : link api

instance HasLink api => HasLink (Capture :> api) where
  link (Capture :> api) = ??? : link api

instance HasLink Verb where
  link _ = []

We should be appending something in place of those ??? there. But since Capture represents variable path fragments (like :userid in /user/:userid, in many web frameworks), we do not want to pick a fixed string, we would like for this string to be supplied by the caller of link, as stated above. Let’s introduce a slightly fancier HasLink class to make it seemingly “variadic”.

class HasLink endpoint where
  type LinkType endpoint :: Type
  link :: endpoint -> LinkType endpoint

instance HasLink Verb where
  type LinkType Verb = Link
  link _ = []

instance HasLink api => HasLink (Static :> api) where
  type LinkType (Static :> api) = LinkType api
  link (Static s :> api) = s : link api

instance HasLink api => HasLink (Capture :> api) where
  -- HERE! we introduce a String argument
  type LinkType (Capture :> api) = String -> LinkType api

  -- we expand the type of link:
  -- link :: (Capture :> api) -> String -> LinkType api
  -- we see that our little `LinkType` trick there allows
  -- link to receive arguments when appropriate
  link (Capture :> api) captureValue = captureValue : link api

Looks good. Except that this does not typecheck. The problem is with the Capture :> api and Static :> api instances. While we know that the link function will eventually return a Link, once given arguments for all the Captures, we don’t know whether there is another Capture later in api. If there is, then link api would have type e.g String -> Link, and we cannot cons a String on top of… a function.

We have to be a little smarter and accumulate the path components as we go without building up the final list directly. We will be accumulating the path components in reverse order, to make the accumulation efficient, and reverse the whole list at the end to give the final Link (= [String]) value.

link :: HasLink endpoint => endpoint -> LinkType endpoint
link e = link' e []

class HasLink endpoint where
  type LinkType endpoint :: Type
  link' :: endpoint -> [String] -> LinkType endpoint

instance HasLink Verb where
  type LinkType Verb = Link
  link' _ acc = reverse acc

instance HasLink api => HasLink (Static :> api) where
  type LinkType (Static :> api) = LinkType api
  link' (Static s :> api) acc = link' api (s : acc)
  -- we stick the static path fragment at the top of the list,
  -- so that it appears after the rest when we reverse the list,
  -- in the Verb instance.

instance HasLink api => HasLink (Capture :> api) where
  type LinkType (Capture :> api) = String -> LinkType api
  link' (Capture :> api) acc captureValue =
    link' api (captureValue : acc)

We can finally generate links with the new approach:

-- "/hello"
simpleEndpointLink = renderLink (link endpoint1)

endpoint2 = Capture :> Verb Post
linkFun2 :: String -> Link
linkFun2 = link endpoint2

link2a = renderLink (linkFun2 "foo") -- "/foo"
link2b = renderLink (linkFun2 "bar") -- "/bar"

endpoint3 = Static "hello" :> Capture :> Capture :> Verb Get
link3 = renderLink (link endpoint3 "x" "y") -- "/hello/x/y"

This looks promising. Let’s now try to introduce some more types here, by allowing captures to not be specified just as simple strings, but any Showable type (this is terrible, but simple enough for this post). We need to modify Capture to track that Showable type we will use to specify the value of that path fragment.

data Capture a = Capture

instance (Show a, HasLink api) => HasLink (Capture a :> api) where
  -- HERE! we introduce an argument of type 'a'
  type LinkType (Capture :> api) = a -> LinkType api

  link' (Capture :> api) acc captureValue =
	link' api (show captureValue : acc)

We unfortunately cannot just “track” some type by storing it in a field (which is different from storing a value of that type). Instead we make Capture a clone of Proxy (from Data.Proxy) and just carry around a phantom type parameter. This is a little inconvenient as we will have to type annotate all Captures (or use the TypeApplications language extension), but let’s roll with this approach for now.

Let’s now see an endpoint description using this variant of Capture.

endpoint4  = Static "hello" :> (Capture :: Capture Int) :> Verb Post
-- or, with TypeApplications:
endpoint4' = Static "hello" :> (Capture @Int) :> Verb Post

OK, interesting, why not. It does look a little bit ugly. It would look even uglier if we included the response type in Verb, turning it into data Verb a = Verb Method which would require the same kind of type annotations. And the same problem would manifest itself if we were to add all the similar types from servant (ReqBody, QueryParam, Header, etc). This is quite disappointing.

Unrelatedly, have you noticed that I have not given the type of any of our endpoint descriptions so far? This is on purpose, because those types are a little bit fancy. Fortunately, they should look familiar:

endpoint1 :: Static :> Verb
endpoint2 :: Capture String :> Verb
endpoint3 :: Static :> Capture String :> Capture String :> Verb
endpoint4 :: Static :> Capture Int :> Verb

That’s right, not only do the descriptions (which are good old haskell values) look like servant’s API types, but their types too! We can see that we are only “hiding” the strings (in static path fragments) and the HTTP method (in verbs) from the type-level.

Most of the other bits of information we would want to see in API descriptions will also have to be reflected at the type-level. When we consider content types for example, we have no choice but to keep track of them at the type level too, even with this design. Because we need to make sure suitable encoding/decoding instances are available for the types that will be represented with those MIME types, and this cannot be done when discovering "application/json" in a list somewhere, at runtime.

All in all, there is no value in keeping anything at the value level at this point. And we are already traversing a bunch of types mixed together with funny symbols and computing the type of a link making function as we go, as evidenced by the HasLink instances from above, so we’ve already got one foot in type-level land.

An important tradeoff that we are making here is that while putting more information at the type-level indeed makes things more complex, it does however give a chance to our descriptions to influence more things, including other types. This is noticeable in the last HasLink instance we wrote, where making Capture track the type the url fragment is going to be decoded to allowed us to directly make the link-making function take a value of that type, instead of a string. This is strictly more powerful and will allow us to work in a very strongly typed world and where the typechecker “writes” a whole lot of code for us.

Let’s bite the bullet and finally take a quick look at what servant’s type-level approach looks like.

Servant’s approach (simplified)

First, let me emphasize that any of the designs we have considered so far are interesting on their own and are fruitful in different ways. They were not quite good enough to meet our requirements which were, again, dictated by the projects and needs we had at work. This whole project started because we were sick of getting things wrong when manually constructing (client) or deconstructing (server) HTTP requests and so on.

Now, let’s write our type-level DSL. If you want a longer version of just this section, with more explanations, you may want to read Implementing a minimal version of servant.

-- GHC-flavoured Haskell supports type-level strings, they are the only type-level
-- entity to have kind Symbol. We will therefore just use those,
-- wrapped with 'Static'. See the first link in the "Going further"
-- section if you're not familiar with type-level strings, kinds, etc.
data Static (str :: Symbol)
data Capture (a :: Type)

-- GHC-flavoured Haskell (with the DataKinds extension) supports promoting ordinary data types
-- to kinds and their constructors to types of those kinds. See again the
-- first link in the "Going further" section if this is new to you.
-- In our example, this lets us parametrize Verb not by an ordinary type but by
-- a constructor of 'Method'. Indeed, 'method' can only be instantiated to
-- Get and Post.
data Method = Get | Post
data Verb (method :: Method)

infixr 5 :>
data (a :: Type) :> (b :: Type)

As you can see, there isn’t a single constructor in sight, all the types (but Method) are empty. And now, we proceed with the HasLink class. Since we don’t have any value to give to the link method, given that the description is now a type, we will use data Proxy a = Proxy to act as an intermediate between the value level, where the calls to link will happen, and the type level, where the descriptions live and drive the link interpretation through our typeclass instances.

link :: Proxy api -> LinkType api
link api = link' api []

class HasLink api where
  type LinkType api :: Type
  link' :: Proxy api -> [String] -> LinkType api

instance HasLink (Verb method) where
  type LinkType (Verb method) = Link
  link' _ acc = reverse acc

instance (KnownSymbol str, HasLink api) => HasLink (Static str :> api) where
  type LinkType (Static str :> api) = LinkType api

  -- we call some "magic" GHC function, symbolVal, to turn type-level
  -- strings to good old value level strings, through a Proxy to
  -- the type-level string.
  link' api acc = link' (apiTail api) (str : acc)
    where str = symbolVal (Proxy :: Proxy str)

instance (Show a, HasLink api) => HasLink (Capture a :> api) where
  type LinkType (Capture a :> api) = a -> LinkType api
  link' api acc a = link' (apiTail api) (show a : acc)

-- we're just specifying a very handy type for a function
-- that's in fact much more general (forall a b. Proxy a -> Proxy b).
-- no magic going on, we just decide that this function takes endpoint description
-- shaped types and drops the first component.
apiTail :: Proxy (a :> b) -> Proxy b
apiTail Proxy = Proxy

It is not all that different from the code in the previous section. We can use it all as follows:

type Foo = Static "hello" :> Capture Int :> Capture Double :> Verb 'Get

linkFoo :: Int -> Double -> Link
linkFoo = link (Proxy :: Proxy Foo)

link1, link2 :: Link
link1 = linkFoo 40 0.1
link2 = linkFoo 2987 980.5

And that’s it! The key ingredients to servant’s design are all here. If you want to read more about actually implementing server/client interpretations for the DSL, the Going further section has got you covered with a few relevant links.

Conclusion

I hope this little tour of some of the designs we explored on our way to writing servant was useful and informative, whether from a Haskell EDSL writer perspective or for any Haskeller who has ever wondered about why the descriptions live at the type-level. The real servant libraries of course have a much richer vocabulary for describing endpoints and entire APIs, and offer many interpretations in addition to the type-safe links. But the core ideas behind the design and implementation are the same ones we progressively arrived at in this post.

Going further

• Servant, Type Families, and Type-level Everything - A look at advanced GHC features used in Servant

  I suspect this is a rather useful resource for Haskellers who haven’t yet encountered type-level programming in (GHC) Haskell.

• Implementing a minimal version of servant

  A more approchable and more narrowly focused alternative to the servant paper, which consists in implementing a very simplified version of servant, using however the same “API type” based approach for the EDSL as the real servant.

• the servant paper, published at the Workshop on Generic Programming, 2015.

• Software extensions and Integration with Type Classes

  by Ralf Lämmel and Klaus Ostermann talks in greater depth than the slides about the highly modular approach to embedded domain specific languages in Haskell that we’ve seen above, and uses it on several examples.

• serv and solga are smaller, younger and (I think) humbler relatives of servant which make slightly different choices for the DSL.

  Somewhat relatedly, there is servant-0.1, which wasn’t anything like the current approach with its API types. The link leads to its README, with an example and some explanations about the approach, for the curious reader.

Introduction

We’re happy to announce the release of servant-0.14. This is relatively small release, still containing some new features and breaking changes.

The problem

The issue started as instance XY-problem:

• Y: Docs explaining how to actually create a full combinator (ex. one to create/store a DB connection)
• X: How to pass a db connection to the handlers.

I won’t answer to the Y, how to write combinators is different topic (have to write about that later). Let’s see how to deal with X, by implementing a small Cat CR(UD) API:

-- we should have proper data/newtypes, but then we'll need to write instances.
-- we'll try to keep a boilerplate at the minimum in this example.
type Cat = Text
type CatName = Text

type API = "cat" :> Capture "name" CatName :> Put '[JSON] Cat  -- create
      :<|> "cat" :> Capture "name" CatName :> Get '[JSON] Cat  -- read

api :: Proxy API
api = Proxy

Now we’ll need to implement the api, we’ll write a basic Haskell functions, which we would write anyway, we could reuse them in a console application, for example.

createCat :: MonadIO m => Connection -> CatName -> m Cat
createCat = error "not implemented"

readCat :: MonadIO m => Connection -> CatName -> m Cat
readCat = error "not implemented"

And the problem is that if we try to do

-- THIS DOESN'T WORK
app :: Application
app = serve api $ createCat :<|> readCat

it will fail with a type-error message from GHC. Obviously, GHC cannot conjure Connection for us. We need to pass it in somehow.

Partial application

Partial application is a simple tool. We can partially apply the implementation to fit into type required by serve. We’ll make a situation a bit more interesting by using a connection pool:

app :: Pool Connection -> Application
app pool = serve api $
    withResource1 pool createCat :<|> withResource1 pool readCat
  where
    withResource1 :: MonadBaseControl IO m => Pool a -> (a -> b -> m c) -> b -> m c
    withResource1 pool f b = withResource pool $ \a -> f a b

As you can see we’d need to wrap every handler in withResource1. It’s not very elegant, but it works. And is very simple to understand.

hoistServer

servant offers the hoistServer helper function. which let’s you to remove this kind of boilerplate. We’ll rewrite our handlers in MTL-style, with a MonadDB type class. For the sake of example let’s also add a MonadLog from log-base to the first endpoint.

class MonadDB m where
    withConnection :: (Connection -> m a) -> m a

createCat' :: (MonadDB m, MonadLog m) => CatName -> m Cat
createCat' = error "not implemented"

readCat' :: (MonadDB m) => CatName -> m Cat
readCat' = error "not implemented"

Looks good, but how we’ll pass a connection (and a logger)? The answer is obvious, when you know it: we’ll need to use a concrete monad implementation, for example:

newtype H a = H { runH :: ReaderT (Pool Connection) (LogT Handler) a }
   deriving (Functor, Applicative, Monad, MonadTime, MonadLog)

instance MonadDB H where
    withConnection f = H $ do
        pool <- ask
        withResource pool $ \conn -> runH (f conn)

And now hoistServer will do the magic:

app' :: Pool Connection -> Logger -> Application
app' pool logger = serve api $ hoistServer api nt $ createCat' :<|> readCat'
  where
    nt :: H x -> Handler x
    nt m -> runLogT "api" logger (runReaderT (runH m) pool)

The nt (for natural transformation) tells how to transform the concrete monad H into servant’s Handler. The hoistServer machinery walks through ServerT H value and applies that transformation, resulting into ServerT Handler value. If api has HasServer instance, you can hoistServer it.

The hoistServer is most useful when you have polymorphic handlers defined with mtl-like monad type-classes, so you can instantiate them all with the same concrete monad at then end. Note: that if we had concrete LogT Handler in some handler, and ReaderT (Pool Connection) Handler in some other one, hoistServer won’t help!

So to conclude:

• start with partial application to pass arguments into handlers
• later you may transfer to use fancier hoistServer.

Alp Mestanogullari summarised it well: gradually reach for fancier things as your needs grow, never when it’s not required.

Introduction

We’re happy to announce the release of servant-0.5. Quite a lot of exciting changes have happened since the last major release. A lot of things happened under the hood, but there’s also a number of changes on the surface. This will force servant users that want to upgrade to perform some changes to their code. Therefore the following section contains an upgrade guide. We hope, upgrading will go smoothly. At least when you didn’t write your own combinators it should be fairly straightforward.

The second section of this blog post contains a description of the major new features in servant-0.5.

Upgrading

    -- Old code
    Get '[] ()
    -- New code
    GetNoContent '[JSON] NoContent

-- Previously
class HasServer layout config where
   type ServerT layout (m :: * -> *) :: *
   route :: Proxy layout -> Server layout -> RoutingApplication

-- Now
class HasServer layout config where
   type ServerT layout (m :: * -> *) :: *
   route :: Proxy layout -> Config config -> Delayed (Server layout) -> Router

Major Changes

data Delayed c where
  Delayed :: { capturesD :: IO (RouteResult captures)
             , methodD   :: IO (RouteResult ())
             , authD     :: IO (RouteResult auth)
             , bodyD     :: IO (RouteResult body)
             , serverD   :: (captures -> auth -> body -> RouteResult c)
             } -> Delayed c

class FromHttpApiData a where
    {-# MINIMAL parseUrlPiece | parseQueryParam #-}
    parseUrlPiece :: Text -> Either Text a
    parseUrlPiece = parseQueryParam

    parseHeader :: ByteString -> Either Text a
    parseHeader = parseUrlPiece . decodeUtf8

    parseQueryParam :: Text -> Either Text a
    parseQueryParam = parseUrlPiece

data MyData = MyData Int

-- Old code
instance ToSample MyData MyData where
    toSample _ = singleSample (MyData 42)

instance ToSample MyData where
    toSamples _ = singleSample (MyData 42)

data MyData = MyData Int deriving (Generic)

instance ToSample MyData

type Get contentTypes a = Verb 'GET 200 contentTypes a

type API = ReqBody '[JSON] Int :> Get '[JSON] Int
      :<|> ReqBody '[JSON] String :> Get '[JSON] String

-- Old code
endpoint1 :<|> endpoint2 = client api
-- endpoint1 :: BaseUrl -> X
-- New code
endpoint1 :<|> endpoint2 = client api baseUrl manager
-- endpoint1 :: X

data User = ...
type API = BasicAuth "my-realm" User :> PrivateAPI

serverContext :: Context (BasicAuthCheck User ': '[])
serverContext = authCheck :. EmptyContext

authCheck :: BasicAuthCheck User
authCheck =
  let check (BasicAuthData username password) =
        if username == "servant" && password == "server"
        then return (Authorized (User "servant"))
        else return Unauthorized
  in BasicAuthCheck check

Release Management

servant-0.5 was a long time in the making. We initially hoped to cut a release last summer. In the future we hope that we can move to a much more aggressive release managament. So we hope servant-0.6 is coming soon, with a lot less changes.

Conclusion

We hope you enjoy the new release. It was certainly lots of fun to work on servant-0.5. We’d like to say thanks to all the people that helped, big or small. Happy hacking.

swagger2 and servant-swagger

Thankfully David Johnson and Nickolay Kudasov have written two Haskell libraries, swagger2 and servant-swagger, that automate nearly all of that process for servant APIs. They use the mechanism that guides most of the servant ecosystem — interpreters for the type-level DSL for APIs that is servant — to generate a swagger spec for that API.

Let’s see how it is used; as an example, we’re going to take the Gists part of the GitHub API v3. For the purpose of this post we will ignore authentication and consider only GET requests which do not require one. Furthermore, we’ll use simplified representation for the responses (i.e. we are also ignoring some fields of the response objects).

First the imports and pragmas (this is a literate haskell file):

{-# LANGUAGE DataKinds #-}
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE GeneralizedNewtypeDeriving #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE TypeOperators #-}
module Gists where

import Control.Lens
import Data.Aeson
import Data.Aeson.Types (camelTo2)
import qualified Data.Aeson.Types as JSON
import qualified Data.ByteString.Lazy.Char8 as BL8
import Data.HashMap.Strict (HashMap)
import Data.Proxy
import Data.Swagger
import Data.Text (Text)
import Data.Time (UTCTime)
import GHC.Generics (Generic)
import Servant
import Servant.Swagger

The API:

type GitHubGistAPI
    = "users" :> Capture "username" Username :> "gists" :> QueryParam "since" UTCTime :> Get '[JSON] [Gist]
 :<|> "gists" :> GistsAPI

type GistsAPI
    = "public"  :> QueryParam "since" UTCTime :> Get '[JSON] [Gist]
 :<|> "starred" :> QueryParam "since" UTCTime :> Get '[JSON] [Gist]
 :<|> Capture "id" GistId :> GistAPI

type GistAPI
    = Get '[JSON] Gist
 :<|> Capture "sha" Revision :> Get '[JSON] Gist

api :: Proxy GitHubGistAPI
api = Proxy

Data types:

newtype Username = Username Text deriving (Generic, ToText, FromJSON)

newtype GistId = GistId Text deriving (Generic, ToText, FromJSON)

newtype SHA = SHA Text deriving (Generic, ToText)

type Revision = SHA

data Gist = Gist
  { gistId          :: GistId
  , gistDescription :: Text
  , gistOwner       :: Owner
  , gistFiles       :: HashMap FilePath GistFile
  , gistTruncated   :: Bool
  , gistComments    :: Integer
  , gistCreatedAt   :: UTCTime
  , gistUpdatedAt   :: UTCTime
  } deriving (Generic)

data OwnerType = User | Organization
  deriving (Generic)

data Owner = Owner
  { ownerLogin      :: Username
  , ownerType       :: OwnerType
  , ownerSiteAdmin  :: Bool
  } deriving (Generic)

data GistFile = GistFile
  { gistfileSize      :: Integer
  , gistfileLanguage  :: Text
  , gistfileRawUrl    :: Text
  } deriving (Generic)

FromJSON instances:

modifier :: String -> String
modifier = drop 1 . dropWhile (/= '_') . camelTo2 '_'

prefixOptions :: JSON.Options
prefixOptions = JSON.defaultOptions { JSON.fieldLabelModifier = modifier }

instance FromJSON OwnerType
instance FromJSON Owner    where parseJSON = genericParseJSON prefixOptions
instance FromJSON GistFile where parseJSON = genericParseJSON prefixOptions
instance FromJSON Gist     where parseJSON = genericParseJSON prefixOptions

So far this is what you would usually have when working with servant. Now to generate Swagger specification we need to define schemas for our types. This is done with ToParamSchema and ToSchema instances:

prefixSchemaOptions :: SchemaOptions
prefixSchemaOptions = defaultSchemaOptions { fieldLabelModifier = modifier }

instance ToParamSchema SHA
instance ToParamSchema Username
instance ToParamSchema GistId

instance ToSchema Username
instance ToSchema GistId
instance ToSchema OwnerType
instance ToSchema Owner    where declareNamedSchema = genericDeclareNamedSchema prefixSchemaOptions
instance ToSchema GistFile where declareNamedSchema = genericDeclareNamedSchema prefixSchemaOptions
instance ToSchema Gist     where declareNamedSchema = genericDeclareNamedSchema prefixSchemaOptions

These will give us a generically-derived Swagger schema (which is sort of a deterministic version of JSON Schema).

Part of the swagger2 package, Schema and ParamSchema can be quite useful in their own right if you want to e.g. respond with a schema in case of bad request bodies, or OPTIONS requests.

The next step will traverse the GitHubGistAPI, gathering information about it and swagger2 schemas to generate a Swagger value:

swaggerDoc1 :: Swagger
swaggerDoc1 = toSwagger api

Now we can generate the swagger documentation:

genSwaggerDoc1 :: IO ()
genSwaggerDoc1 = BL8.putStr $ encode swaggerDoc1

You can attach more information to your Swagger doc quite easily, using the lenses provided by swagger2:

swaggerDoc2 :: Swagger
swaggerDoc2 = swaggerDoc1
  & host ?~ "api.github.com"
  & info.title .~ "GitHub Gists API"
  & info.version .~ "v3"

main :: IO ()
main = BL8.putStr $ encode swaggerDoc2

Which results in this.

There’s a lot more you can do with both servant-swagger and swagger2 — write manual ToSchema instances for more detailed information, conveniently add tags or change responses of parts of your API, use convenient lenses to modify any part of your schema, generate automatic tests, etc.

Check out the servant-swagger and swagger2 docs for more.

These two new packages vastly expand the landscape of tools within easy reach of application developers using servant. Time to explore that landscape!

On a related note, Masahiro Yamauchi has recently added Servant codegen for Swagger. So not only can you generate a swagger description for your servant server, but you can also generate the servant description from a swagger one too!

The Application

Our goal is to provide a service that converts images between formats based on the Content-Type and Accept headers of the request:

$ curl localhost:8001 -H "Content-Type: image/png"  \
                      -H "Accept: image/jpeg"  \
                      --data-binary "@haskell-logo.png" \
                      > haskell-logo.jpeg

To get there, we need to do a couple of things. We need to of course run the application:

main :: IO ()
main = run 8001 conversion

And describe the API:

type ConversionApi
     = ReqBody '[BMP, GIF, JPEG 50, PNG, TIFF, RADIANCE] DynamicImage
    :> Post '[BMP, GIF, JPEG 50, PNG, TIFF, RADIANCE] DynamicImage

As you can see, we state that we accept and can return a variety of image formats.

The application is then:

conversion :: Application
conversion = serve (Proxy :: Proxy ConversionApi) handler

And for the clincher, the handler:

    where handler = return

And that’s it!

Conclusion

This is just the limit of the relative gain of abstracting content-types - there is nothing to the application besides them!

Essentially the same idea could of course be applied to other areas. Document conversion with Pandoc, video and audio formats, etc.

Summary

A couple of days ago, marcushg mentioned on the #servant IRC channel that one could probably easily use the information available from API types to “derive” a mock implementation of your request handlers that just generates random values of whatever the return type of the handlers are. Julian and I discussed this a bit today and I just went ahead and wrote down our thoughts in a new branch. The result will be explained in this post, but in short, it lets us take a type describing a web API, such as:

type API = "user" :> Get '[JSON] User

and generate request handlers that just respond with random values of the appropriate type, User in our case. In servant/wai terms, this means we get a mock function with the type:

mock :: HasMock api => Proxy api -> Server api

i.e., “given an API type, please generate a mock server for such an API”. This effectively means “please pull a mock server out of thin air for me”.

Out of thin air, really? Not exactly. But let’s start by clearly stating the problem.

The Problem

servant lets you describe web applications with a Haskell type using the combinators from servant’s type-level EDSL. Such a type would be:

-- In English:
-- the API has one endpoint, under /user, which returns
-- response bodies in JSON that describe values of type User
type API = "user" :> Get '[JSON] User

where User could be defined as:

newtype User = User { username :: String }

The goal would be to “automagically” derive a request handler of the right type that we could use as a placeholder until we properly implement a handler that talks to the database and responds with “the real data”.

For anyone not familiar with servant already, you just need to know that it means we need to somehow automatically implement a computation with the type:

getUser :: EitherT ServantErr IO User

possibly by constraining the user to provide an instance for some random generation class.

The Plan

Just like servant-server, servant-client and others, we need a class whose instances will define the way we interpret each combinator, in a way very specific to this task: we will produce what servant-server takes as input, i.e., request handlers! This all means we are basically looking at a class like:

class HasServer api => HasMock api where
  mock :: Proxy api -> Server api

where Server api just computes the types of the all the request handlers of an API type. In our case, Server api is computed as follows:

-- api = the API type from above in our case
Server API = Server ("user" :> Get '[JSON] User)
           -- static strings in the path do not influence
           -- the type of a handler
           = Server (Get '[JSON] User)
           -- EitherT ServantErr IO is the default monad
           -- in which handlers run
           = Either ServantErr IO User

So we have to implement at least support for static string fragments in the path and the Get combinator (i.e handlers for HTTP GET requests).

HasMock instances

Let’s start with the one for static path fragments, it’s the simplest one: we ignore the string bit and move on to what comes after.

instance (KnownSymbol path, HasMock rest) => HasMock (path :> rest) where
  mock _ = mock (Proxy :: Proxy rest)

Don’t be scared by KnownSymbol, it basically just means “path is a type-level string”, that is, a string that appears in a type.

Next comes the one for Get. This one is trickier: this is the combinator that says what type the handler returns. The returned value then gets encoded into JSON, HTML, CSV or any format of your choice. In our case, the handler returns a User and can only encode it in the JSON format.

Now the heart of the matter really is: we know we need to return an User and our EitherT ServantErr IO monad mentions IO, couldn’t we randomly generate an User? Yes, we can! For the purpose of a mock server, we will simply use QuickCheck’s Arbitrary class, which represents types for which we can generate random values, given a random number generator.

class Arbitrary a where
  arbitrary :: Gen a
  -- and another method, but optional

The Gen type provides instances for the Functor, Applicative and Monad classes and Arbitrary comes with instances for many of the types in base.

This essentially means writing an Arbitrary instance for User is as simple as:

instance Arbitrary User where
  -- we just rely on the arbitrary instance for lists of
  -- chars, i.e., Strings, and use the Functor instance for Gen
  arbitrary = fmap User arbitrary

If you have multiple fields, you can use the usual combo of <$> (i.e., fmap) and <*> (comes with Applicative).

-- a point: x, y coordinates
data Point = Point Double Double

instance Arbitrary Point where
  arbitrary = Point <$> arbitrary <*> arbitrary

Once you have an Arbitrary instance, in order to generate a random value using your instance, you have to call a function called… generate!

generate :: Gen a -> IO a

Putting the two together, we get:

generate arbitrary :: Arbitrary a => IO a

All we need to do is just “lift” that up into our EitherT ServantErr IO monad, which is exactly what Control.Monad.IO.Class.liftIO is about in the transformers package.

liftIO (generate arbitrary) :: Arbitrary a => EitherT ServantErr IO a

In order to automatically “fill” request handlers with this expression we just need to write the HasMock instance for Get, shown below.

instance (Arbitrary a, AllCTRender ctypes a) => HasMock (Get ctypes a) where
  mock _ = liftIO (generate arbitrary)

The AllCTRender constraint just says “we know how to encode values of type a in the formats listed in the Get combinator”.

And that’s it! You can now actually use all of this to put together a mock server for our little API.

Using mock

All we need to do to run the mock server is call servant-server’s serve function. It is illustrated below, along with all of the code you’d have to write if you were to use this mock-generation feature (aside from language pragmas and imports).

-- 1/ define our user type, deriving the Arbitrary instance
--    since it's just a newtype and we can use the
--    GeneralizedNewtypeDeriving extension. We also
--    derive the Generic class to get our JSON encoding
--    functions for free.
newtype User = User { username :: String }
  deriving (Arbitrary, Generic)

-- 2/ we get the JSON encoding for free
instance ToJSON Generic

-- 3/ recall our API type
type API = "user" :> Get '[JSON] User

-- 4/ define this simple Proxy.
-- for any given type 'a', there's only one value of type 'Proxy a'
-- that is not equivalent to error "foo" and the likes, a real honest value.
-- The way to build this value is to use the Proxy constructor.
-- In other words, this value lets us target one precise type. Servant
-- uses this to tie the type-level information with value-level data.
api :: Proxy API
api = Proxy

-- 5/ we magically derive the mock server.
-- the run function comes from the warp webserver,
-- http://hackage.haskell.org/package/warp
-- 'mock' is the method of the `HasMock` class we've
-- developed in this post.
-- This will run a mock web server with an endpoint at
-- http://localhost:8080/user that generates random values
-- of type User
main :: IO ()
main = run 8080 (serve api $ mock api)

Our little program in action:

$ curl localhost:8080/user
{"username":"Yv\rG\u0014±Ssv\u001e>\u001aVF\u001e\u000fM5ø\u000ctefÚJ\u0001K4"}
# yes, a truly original username.

This is really all you have to do to put together a mock server for an API type. You can find the complete code for this in the work-in-progress servant-mock package on github. The example can be found under example/main.hs there.

There are many more HasMock instances than the ones I have shown here of course – there’s one for all the combinators provided by the servant package! So you can take any API type out there and just create a mock server for it, as long as you provide Arbitrary instances for your data types. Nothing too interesting though, but feel free to take a look at src/Servant/Mock.hs in the repository if you want to read the other instances.

I hope this makes clear how simple writing your own servant interpretation can be, and encourages you to try your hand at it!

Other news

• I mentioned in a previous post that we had submitted a paper for the Workshop on Generic Programming, co-located with ICFP’15, in Vancouver this year. Well, the paper has been accepted!
• Therefore, Julian Arni and/or Andres Löh will be giving a talk about servant there.
• In addition to this, Julian Arni recently gave a talk about servant at Curry-On!. The video will be uploaded on their Youtube channel in the upcoming days.
• I have submitted a very hands-on servant talk proposal for the Haskell eXchange 2015 in London, and it has been accepted! See you there, folks.

Multiple content-type support

servant combinators are not JSON-centric anymore.

If you had an API type like the following with servant 0.2.x:

type API = -- list users
           "users" :> Get [User]
           -- update an user
      :<|> "user" :> Capture "username" Text :> ReqBody User :> Put ()

You now have to change it to:

type API = -- list users
           "users" :> Get '[JSON] [User]
      :<|> "user" :> Capture "username" Text :> ReqBody '[JSON] User :> Put '[JSON] ()

Wherever applicable (i.e., ReqBody and all the combinators that correspond to an HTTP method), you can now specify all the content types in which you want to want to be able to encode/decode values. As you can see, we use the DataKinds GHC extension to let you specify a type-level list of content-types, which are simple dummy types:

data JSON

In servant-server, a list of these content-types as the first argument of a method gets translated into a set of constraints on the return type:

Get '[JSON, PlainText] Int
==>
MimeRender JSON Int, MimeRender PlainText Int => EitherT ServantErr IO Int

Which have unsurprising instances:

instance (ToJSON a) => MimeRender JSON a

Thus, servant checks at compile-time that it really can serialize your values as you describe. And of course, it handles picking the appropriate serialization format based on the request’s “Accept” header for you.

(For ReqBody, deserialization is involved. For servant-client, the logic goes the other way around - serialization for ReqBody, deserialization for methods.)

servant-blaze and servant-lucid

Declaring new content-types, and the associated constraints for them, is quite easy. But to make it easier still, we are also announcing two new packages: servant-blaze and servant-lucid. To use them, just import their HTML datatype:

import Servant.HTML.Lucid (HTML) -- or Servant.HTML.Blaze

type MyHTML = Get '[HTML] User

And User will be checked for the appropriate (e.g. ToHtml) instance.

Response headers

There was no easy way so far to have handlers add headers to a response. We’ve since come up with a solution that stays true to the servant spirit: what headers your response will include (and what their types are) is still enforced statically:

type MyHandler = Get '[JSON] (Headers '[Header "Location" Link] User)

myHandler :: Server MyHandler
myHandler = return $ addHeader <someLink> $ <someuser>

servant-docs and servant-client are also response-header aware.

Our current solution isn’t something we are entirely happy with from an internal persepctive. We use overlapping instances for all the handlers, which some might think is already a problem. But more concretely, there’s the threat of an exponential blowup in the number of instances we have to declare. And that can be a problem for end users too, if they decide to further modify behavior via a similar mechanism. But these things thankfully don’t seem to pose any immediate problems.

Running handlers in other monads than EitherT

An often-requested feature has been easy use of datatypes/monads besides EitherT. Now we believe we have a good story for that (thanks in large part to rschatz). To convert from one datatype to another, all you need to do is provide a natural transformation between them. For example:

type ReaderAPI = "a" :> Get '[JSON] Int
            :<|> "b" :> Get '[JSON] String

readerServerT :: ServerT ReaderAPI (Reader String)
readerServerT = return 1797 :<|> ask

readerServer :: Server ReaderAPI
readerServer = enter (Nat $ return . (`runReader` "hi")) readerServerT

The new ServerT type synonym takes an extra paramer that represents what datatype/monad you are using over your handlers (instead of EitherT ServantErr IO).

(Note that we also provide a number of pre-existing Nats, which are an instance of Category. We could have used

readerServer = enter (generalizeNat . (runReaderTNat "hi")) readerServerT

instead (with . being from Control.Category).)

Note that the datatypes you can use now don’t even need to be monads!

mkLink

Somewhere between the 0.2 release and now, mkLink got a whole lot better (thanks Christian Marie!). mkLink makes urls that are statically guaranteed to belong to your API, without any Template Haskell. Combined with response headers, you can now easily create, for instance, type-safe redirect headers. Combined with the new HTML support, you can easily make links that you know will not 404.

Left

We also changed the default type of handlers from EitherT (Int,String) IO a to EitherT ServantErr IO a. Now it is possible to return headers and a response body in the Left case.

We also now export function errXXX (where XXX is a 300-599 HTTP status code) with sensible reason strings.

BaseUrl

We also changed the client function from servant-client so that, instead of returning various functions that each take a BaseUrl argument (often in inconvenient argument positions), the client function itself takes a BaseUrl argument, and the functions it returns don’t. So the type of client went from

client :: HasClient (Canonicalize layout) => Proxy layout -> Client layout

To

client :: HasClient (Canonicalize layout) => Proxy layout -> BaseUrl -> Client layout

Complete CHANGELOGs

• servant
• servant-server
• servant-client
• servant-docs
• servant-jquery

Website

We also decided to switch to hakyll in order to be able to have a blog as well as some static pages that collect tips and tricks that people have found. We also used this opportunity to rewrite the getting started into a more informative tutorial, now available here.

Conclusions

As you can see, more and more information is getting encoded statically - the types are becoming a pretty rich DSL. In order to keep the noise down, do what you normally do: abstract away common patterns! If your endpoints always return the same content-types, make aliases:

type Get' a = Get '[JSON, HTML, PlainText] a

There’s still an outstanding issue with the errors servant returns when a request doesn’t get handled. For example, if the path of a request, but not the method nor the request body, match, rather than returning a 405 (Method Not Allowed) we return a 400 (Bad Request), which is not the desired behavior. Andres Löh made some great suggestions for how to improve our routing time complexity, and hopefully we can integrate a fix for this issue when we tackle that.

We also merged our repos into servant. Please use that repo exclusively for PRs and issues (we’ll get rid of the others eventually).

Special thanks to the Anchor team from Australia, Matthew Pickering, Daniel Larsson, Phil Freeman, Matthias Fischmann, rschatz, Mateusz Kowalczyk, Brandon Martin and Sean Leather who’ve contributed from little fixes to whole new features. Several companies are now running servant-powered web applications.