Welcome to Ceramic Hacker

Hi, and welcome back to my OCaml web development project! In this post, I'll be going over the basics of backend web programming in OCaml using the Dream and Caqti libraries.

As motivation, we'll consider building a basic login system. This roughly corresponds to a simplified version of this stage of my project.

Before we start, any asynchronous code we write, such as querying a database, will use OCaml’s `Lwt` concurrency library. If you’re not familiar with it, this article is a good introduction.

What is a Backend Web Framework?

If you've worked with tools like Django, Flask, Laravel, Ruby on Rails, or .NET Core, you're probably familiar with the idea of a backend web framework. Essentially, it's a set of tools, functions, methods, etc that allow you to handle, and respond to, incoming browser requests in your programming language of choice. Some frameworks are bigger and some are more minimalistic, but at the minimum, most have:

• The notion of "handlers", which are logic (usually functions) that compute the response to some incoming request. For example, the handler for a homepage would return the HTML for that homepage, while the handler for a user's account info page would return an HTML page with the current user's information, or a "not logged in" error page if there's no active user. Handlers don't always need to return HTML: a handler for some path in a REST API would probably return information formatted as JSON.
• Support for a "router", which maps paths / path patterns to handlers. For example, requests to "https://yourwebsite.com" should be routed to the homepage handler, whereas "https://yourwebsite.com/account" would go to the account info handler, and "https://yourwebsite.com/api/discussions/5" would go to "get discussion info" handler.
• Support for "middlewares", which are functions that transform all requests before they get to a handler, or all responses returned by a handler before the response is sent to the user. This could be used to attach/verify HTTP headers, figure out who the current user is, implement rate limiting, or run other logic for every request/response.

Additionally, many web frameworks choose to provide features such as:

• Session Management: since HTTP is a stateless protocol, sessions allow us to keep track of users across multiple requests. This is generally done by storing information in cookies or HTTP headers.
• Cryptography Tools: many frameworks include a set of functions for checking passwords, generating/verifying encrypted keys, etc.
• Error Handling Tools: often, web applications can define custom functions for how errors should be handled.
• Static File Management: some web frameworks can serve a directory of static files, although this should usually be done via web servers such as NGINX in production.

Dream Introduction

Dream is a relatively new, simple, minimalistic web framework for OCaml. It's easy to use, and has extensive documentation, great examples, and even an interactive playground. You can install it with opam install dream. The code we'll use to run our site is as follows:

let sql_uri = "sqlite3:db.sqlite"

let start () =
  Dream.run
  @@ Dream.logger
  @@ Dream.sql_pool sql_uri
  @@ Dream.sql_sessions
  @@ Dream.router Nittany_market.Router.routes

We're using several middlewares: Dream.logger will log any errors and interesting events to the terminal, Dream.sql_pool will set up a connection to our SQLite database (more on that later), and Dream.sql_sessions will enable sessions through our database. Note that I putting the path=>handler routing map in a different module; here's what it looks like:

let routes : Dream.route list = [
  (* This will return HTML for the home page, which includes a login form. *)
  Dream.get "/" Views.Index.get; 
  (* This will handle login requests. *)
  Dream.post "/login" Views.Login.post; 
  (* This will handle logout requests. *)
  Dream.post "/logout" Views.Logout.post; 
]

Relatively simple. Now let's go over the code for each of our handlers.

Homepage Handler

We'll start with the homepage:

open Jingoo

let get request =
  let user_id = Dream.session_field request "user_id" in
  Dream.html
    (Jg_template.from_file "templates/index.jinja"
       ~models:
         [
           ( "user_id",
             match user_id with
             | Some id -> Jg_types.Tstr id
             | None -> Jg_types.Tnull );
           ("csrf_tag", Jg_types.Tstr (Dream.csrf_tag request));
         ])

Here, we get the ID of the current user using Dream's session helper. You'll see soon that this field gets added to a session when a user logs in successfully. We also use Dream's CSRF protection tools for safety. Then, we use the Jingoo library to generate an HTML response using the following template:

{% if user_id %}
    Logged in as {{ user_id }}
    <form action="/logout" method="post">
        {{ csrf_tag|safe }}
        <button type="submit">Logout</button>
    </form>
{% else %}
    Not logged in
    <form action="/login" method="post">
        {{ csrf_tag|safe }}
        <input name="email">
        <input name="password">
        <button type="submit">Login</button>
    </form
{% endif %}

Jingoo is a template language/renderer similar to Django templates or Jinja.

So in other words, if the user is logged in, we'll display "Logged in as [USER_ID]", and a logout button. Otherwise, we'll show a login form.

Login/Logout Handlers

The login handler is a bit more logic heavy, but still not too complicated:

let post request =
  match%lwt Dream.form request with
  | `Ok [ ("email", email); ("password", password) ] ->
      let%lwt u = Dream.sql request (Models.User.get email) in
      (match u with
      | Some user ->
          if Auth.Hasher.verify user.password password then
            let%lwt () = Dream.set_session_field request "user_id" email in
            Dream.redirect request "/"
          else Dream.empty `Unauthorized
      | None -> Dream.empty `Unauthorized)
  | _ -> Dream.empty `Bad_Request

The logic is as follows:

1. We start by using Dream's form tools to get the submitted email and password. If that's not available, we throw a "bad request" error.
2. We try to get a user with that email from the database using Dream's SQL helpers, and a Models.User.get function that we'll discuss in the next section. If no such user exists, we throw an "Unauthorized" error.
3. We check whether the provided password matches the current user's password by comparing hashed versions of the password. The password hash/verify functions are implemented in a separate file. If they match, we mark the user as "logged in" by setting their user id in the session, and redirecting to the homepage. Otherwise, we throw an "Unauthorized" error.

The logout handler is a lot simpler, as all we need to do is clear the current session:

let post request =
  let%lwt () = Dream.invalidate_session request in
  Dream.redirect request "/"

And that's all we need! With several middlewares, a routing map, several request handlers, and a template, we've implemented a basic login/logout system. Next, we'll cover how we can interact with the database using OCaml.

The Database Part

As you probably know, a database is used to store, query, and update data efficiently and safely. If you don't know much about databases or SQL (the language used to interact with many databases), this article is a decent place to start.

In the previous section, we used the Models.User.get function to get a user by email. In this section, I'll go over a possible implementation of this function using Caqti.

Database Setup

For simplicity, I used SQLite for my database as opposed to something more powerful like MariaDB or PostgreSQL. The main benefit is that I didn't need to run the database as a standalone process. All I needed to do to create the database was run sqlite3 db.sqlite in the root folder of my project.

With the database created, I needed to make tables for my data. We can do this by running the following SQL commands while connected to the database:

CREATE TABLE dream_session (
  id TEXT PRIMARY KEY,
  label TEXT NOT NULL,
  expires_at REAL NOT NULL,
  payload TEXT NOT NULL
);

CREATE TABLE user (
  email TEXT,
  password TEXT,
  PRIMARY KEY (email)
);

Respectively, these create tables for storing session data and basic information about the users in our system. A real site would probably have a more complex user model, but we're keeping it simple here.

Migrations (optional)

This is a bit beyond the scope of what I want to cover, but I actually built a trivial database migrations system instead of . It's far from complete, as it doesn't track which migrations have/haven't been applied, but it does support applying / rolling back all migrations, and lets me set up table structure from the codebase. Here's the code.

In a real project, it would probably be preferable to use something more mature and feature-complete, such as omigrate.

Caqti and our User get function

Caqti is an OCaml database client library, kind of like Python's SQLAlchemy or Java's JDBC. It provides a mechanism for running queries against a database in a type-safe way. I highly recommend this article for an intro to Caqti, although note that the Caqti_request.* functions have been deprecated in favor of new infix operators.

Here's an abridged version of code for our User access function:

open Caqti_request.Infix
module type DB = Caqti_lwt.CONNECTION

module R = Caqti_request
module T = Caqti_type

type t = { email : string; password : string } [@@deriving yojson, fields, csv]

let get email =
  let query =
    (T.string -->? T.(tup2 string string))
    @:- "SELECT * FROM user WHERE email=?"
  in
  fun (module Db : DB) ->
    let%lwt unit_or_error = Db.find_opt query email in
    let raw = Caqti_lwt.or_fail unit_or_error in
    Lwt.bind raw (fun u ->
        Lwt.return
          (match u with
          | Some (email, password) -> Some { email; password }
          | None -> None))

Let's go over it piece by piece.

• At the start of the file, we define several shorthand type aliases, and open Caqti_request.Infix so that we can use the new query-building infix operators.
• Our type t for users is a record with email and password fields. This corresponds to the database structure we set up in the last section.
• The get function takes a string email, and returns a function that when given a DB connection, evaluates the query.
  • First, we construct the query object that we will evaluate. Since we are searching by email (string) and looking to get an email/password tuple, we specify these input/output types in addition to the query string. The -->? infix operator means that we are fetching 0 or 1 rows.
  • We use Caqti's find_opt function to evaluate the query using the database connection instance.
  • We run that raw output through Caqti_lwt.or_fail to throw an exception if there's an error. If we reach past this point, we know the query has succeeded.
  • Since queries are evaluated using the asynchronous Lwt library, the outputs are packaged in a Lwt.t type. We use Lwt's map function to transform this from a tuple option to a User.t option. It'll still return None if there's no matching user, but otherwise it'll be in the record form we defined previously.

The final result is a function of the type:

val get: string -> (module Caqti_lwt.CONNECTION) -> t option Lwt.t

Now that we've discussed how this function is implemented, let's revisit its usage:

let%lwt u = Dream.sql request (Models.User.get email) in ...

A few things of interest:

• The let%lwt u = part is syntactic sugar for something called a monadic bind, and essentially allows us to access u as a User.t instead of a User.t Lwt.t.
• Once we pass the email to Models.User.get, it has the type Caqti_lwt.connection -> 'User.t Lwt.t. Dream.sql is a helper that takes the current request, as well as functions of that type, and executes them. That's why this gives us User.t Lwt.t.

If you look through the codebase of my project, you'll see this pattern whenever we need to access the database: we define a query function, bind all arguments other than Caqti_lwt.connection, and then pass that function to Dream.sql which executes it and returns the result. Then, using let%lwt = ... in ..., we can access the result of our query.

A Standardized Model Interface (optional)

In this section, I discussed database implementation for one model with one query function. I used a lot more than that for my actual project, so I ended up making a few code organization changes.

Firstly, I split each model into a Model module, which contained the record type for the model, and a ModelRepository module, which contained all the query methods.

Secondly, and more interestingly, I noticed that the "create", "get", and "all" functions for each model were very similar. So similar, in fact, that I was able to code Make_ModelRepository and Make_SingleKeyModelRepository (for models with a single primary key, such as "email" for users) functors that automatically implemented those functions, as long as each model declared some basic types and conversion functions. Of course, some models needed additional query functions, but because OCaml modules/module types are so flexible, that was easy to add to both the .mli interfaces and the .ml implementations.

As a result, I got basic query operations for all my models for free, massively cutting down on boilerplate.

Conclusion

In this post, we built a simple login system with Dream, with database functionality powered by Caqti. We covered, among other topics:

• What web frameworks are.
• The structure of a simple Dream-backed website.
• The usage of sessions to keep track of the current user.
• How to interact with a database from Dream using Caqti.

I am very excited about Dream because it aligns very well with my vision of what a backend web framework should be: a simple system for handling requests, and a library of convenience functions to accomplish desired tasks.

In the next post, we'll change our backend to use GraphQL mutations for the login/logout endpoints instead of simple handlers.

CLI Tool (optional)

One thing I like about the Django framework is the command line interface (CLI) it provides for managing sites built with Django. It can be used to serve a site, run migrations, send a test email, generate boilerplate code, and compile static assets, among other features.

Inspired by this, I decided to try and build a simple CLI for managing my project's site. Using the cmdliner library, I implemented commands for starting the site, loading in testing data, and applying/undoing database migrations. You can see the implementation on GitHub.

That being said, I probably should have done some googling first, because it turns out I wasn't the first person to have this idea. tmattio's dream-cli generates a CLI which allows running a Dream site with a wide variety of command line config arguments. It even supports custom commands. I think this is a powerful and useful addition to any site implemented in Dream.