This post highlights of one of the core ideas posted in this blogpost. If you’ve already read it and you’re intimately familiar with transducers, this post probably won’t have anything new for you. I’ve posted this to Stackoverflow before and saving this to my blog for archival purposes.

In the pre-transducer era, reading text files was often done like this:

1
2
3
4
5
6

(require '[clojure.java.io :as io])
(with-open [rdr (io/reader "/tmp/work.txt")]
  (->> (line-seq rdr)
       (mapcat #(str/split % #";"))
       (map count)
       (doall))

Given input work.txt:

I;am;a;string
Next;line;please

this would return (1 2 1 6 4 4 6). One caveat with this approach is you have to realize the result inside the with-open macro, else the file would already be closed.

What if we want to use transducers instead of lazy collection transformations? The ingredient you need is something that allows you to treat the lines as a reducible collection and which closes the reader when you’re done reducing:

1
2
3
4
5
6
7
8
9
10
11
12
13

(defn lines-reducible
  [^java.io.BufferedReader rdr]
  (reify clojure.lang.IReduceInit
    (reduce [this f init]
      (try
        (loop [state init]
          (if (reduced? state)
            @state
            (if-let [line (.readLine rdr)]
              (recur (f state line))
              state)))
        (finally
          (.close rdr))))))

Count the length of each ‘split’

1
2
3
4
5
6
7
8
9

(require '[clojure.string :as str])
(require '[clojure.java.io :as io])

(into []
      (comp
       (mapcat #(str/split % #";"))
       (map count))
      (lines-reducible (io/reader "/tmp/work.txt")))
;;=> [1 2 1 6 4 4 6]

Sum the length of all ‘splits’

1
2
3
4
5
6
7

(transduce
 (comp
  (mapcat #(str/split % #";"))
  (map count))
 +
 (lines-reducible (io/reader "/tmp/work.txt")))
;;=> 24

Sum the length of all words until we find a word that is longer than 5

1
2
3
4
5
6
7
8
9
10
11
12

(transduce
 (comp
  (mapcat #(str/split % #";"))
  (map count))
 (fn
   ([] 0)
   ([sum] sum)
   ([sum l]
    (if (> l 5)
      (reduced sum)
      (+ sum l))))
 (lines-reducible (io/reader "/tmp/work.txt")))

or with take-while:

1
2
3
4
5
6
7

(transduce
 (comp
  (mapcat #(str/split % #";"))
  (map count)
  (take-while #(> 5 %)))
 +
 (lines-reducible (io/reader "/tmp/work.txt")))

Read https://tech.grammarly.com/blog/building-etl-pipelines-with-clojure for more details.

Advent of Code is a collection of self-contained programming problems, one for each day during Advent. My favorite problem of 2015’s edition was Day 7: Some Assembly Required. You can read the full description on the site. I’ll explain it briefly. A circuit looks like this:

1
2
3
4
5
6
7
8

123 -> x
456 -> y
x AND y -> d
x OR y -> e
x LSHIFT 2 -> f
y RSHIFT 2 -> g
NOT x -> h
NOT y -> i

On the left you have one or more inputs combined by a logical gate which is then wired to the output on the right.

If you run this circuit, it will produce the following state:

1
2
3
4
5
6
7
8

d: 72
e: 507
f: 492
g: 114
h: 65412
i: 65079
x: 123
y: 456

The problem can be subdivided into parsing and processing. When I solved this problem in 2015 I used good old string/split and regexes. But it can also be solved quite elegantly with clojure.spec as we shall see.

I’ll walk you through the circuit spec in reverse.

An expression is the left hand side followed by the symbol -> followed by the right hand side.

1

(s/def ::expr (s/cat :lhs ::lhs :arrow #{'->} :rhs ::rhs))

The right hand side is always a variable name.

1

(s/def ::rhs ::varname)

The left hand side is a little bit more involved. It is either a simple value (a variable name or concrete value: x, 456, 1337, ac), a binary expression (x AND y) or something which is negated (NOT x).

1
2
3
4

(s/def ::lhs (s/alt :simple-value ::val
                    :binary-expression
                    ::binary-expression
                    :not ::not))

A negated expression is the symbol NOT followed by a value.

1

(s/def ::not (s/cat :not #{'NOT} :operand ::val))

A binary expression is a value followed by an operator followed by another value.

1
2
3
4

(s/def ::binary-expression (s/cat
                            :left-operand ::val
                            :operator ::binary-operator
                            :right-operand ::val))

A binary operator is one of the symbols in the set.

1

(s/def ::binary-operator #{'LSHIFT 'RSHIFT 'AND 'OR})

A value is either a variable name or a non-negative integer.

1
2

(s/def ::val (s/alt :name ::varname
                    :value nat-int?))

Lastly, a variable name is a symbol. I could have specified this in more detail by restricting the allowed characters and the length of the symbol, but this was not needed to succesfully parse my input.

1
2
3

(s/def ::varname
  (s/with-gen symbol?
    (fn [] varname-gen)))

However, to generate a variable name which looks like my input, for the sake of playing around with spec, I need to provide my own generator. Scanning through my input, I discovered that a variable name’s length is either 1 or 2 and only alphabetic characters may be used.

1
2
3
4
5
6
7
8

(def varname-gen
  (gen/fmap (fn [chars]
              (symbol
               (str/lower-case
                (apply str chars))))
            (gen/vector (gen/char-alpha)
                        1
                        2)))

Running the generator yields for example:

1

(gen/sample varname-gen) ;;=> (g dv de pw hi a c j bz y)

We can now generate entire expressions:

1

(gen/sample (s/gen ::expr)) ;;=> ((NOT g -> sw) (NOT 0 -> j) (gl LSHIFT 0 -> q) (NOT 1 -> ly) (NOT 2 -> j) (ug -> o) (p RSHIFT 0 -> p) (NOT oj -> dz) (ih -> m) (NOT 5 -> fc))

Looks good.

To read the lines from the input file I cheated a little bit by using edn/read-string which parses raw strings to a vector of symbols and numbers for me:

1
2
3
4
5
6
7
8
9
10

(defn get-lines []
  (str/split-lines
   (slurp "input-day7.txt")))

(defn parsed-lines [lines]
  (mapv (fn [l]
          (let [edn (edn/read-string
                     (format "[%s]" l))]
            (s/conform ::expr edn)))
        lines))

I could have used line-seq or a text transducer to save some memory, but as Knuth says, if you optimize everything you will always be unhappy.

Let’s peek at the first conformed expression which corresponds to the line bn RSHIFT 2 -> bo:

1

(first (parsed-lines (get-lines))) ;;=> {:lhs [:binary-expression {:left-operand [:name bn], :operator RSHIFT, :right-operand [:value 2]}], :arrow ->, :rhs bo}

Yay! Now we have to write some code that processes these lines and calculates the values for each variable. To do this, we are going to build up a map of symbols to their values:

1

(def context (atom {}))

The right hand side of an expression is always a variable name. So we assoc it to the context where the value is a delay of the evaluation of the left hand side.

1
2
3
4
5
6

(defn evaluate-expr! [expr]
  (let [rhs (:rhs expr)
        lhs (:lhs expr)]
    (swap! context assoc rhs
           (delay
            (evaluate* lhs)))))

The reason we are using a delay for the values of the context map, is twofold: delaying and caching. Firstly, not all values that the variable depends on are already added to the context, so we have to delay calculation until every expression has been processed. Secondly, once a value of a variable is known, we do not want to recalculate it. My circuit file is 339 lines long and without caching this becomes terribly slow.

The API we need to get the solution for our Advent of Code puzzle is a function from symbol to integer.

1
2

(defn value-by-symbol [sym]
  @(get @context sym)))

The double deref is needed because we’re dealing with an atom and a delay.

The last bit we need to is evaluate the left hand side of an expression. Here we pattern match on the kind of expression and evaluate accordingly!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

(defn evaluate* [[kind tree-or-val]]
  (case kind
    :value tree-or-val
    :name (value-by-symbol tree-or-val)
    :simple-value (evaluate* tree-or-val)
    :not (bit-not
          (evaluate*
           (:operand tree-or-val)))
    :binary-expression
    (let [l (evaluate* (:left-operand tree-or-val))
          r (evaluate* (:right-operand tree-or-val))
          operator (case (:operator tree-or-val)
                     AND bit-and
                     OR bit-or
                     LSHIFT bit-shift-left
                     RSHIFT bit-shift-right)]
      (operator l r))))

Finally, after evaluating all the lines we can ask for the value of a symbol in the circuit.

1

(value-by-symbol 'a) ;;=> 46065

which happened to be the correct value for my input!

Thanks for reading. The full code is available on Github. Constructive feedback and criticisms are welcome.

Clojure beginners often hear that inline def is bad style. For code you commit that’s mostly true. Example:

1
2
3

(defn foo [x]
  (def y (+ x 1))
  y)

Here def performs a side effect every time you call foo, by (re)defining a Var y which is global to the current namespace. Instead you should use let to make it a local:

1
2
3

(defn foo [x]
  (let [y (+ x 1)]
    y))

However, inline defs can serve a useful purpose, namely that of a simple debugger. Debuggers for Clojure exist, e.g. in CIDER, but I mostly forget to use them. After println, inline def is my debugging tool of choice.

Example:

1
2

(defn foo [& [{:keys [a b c] :as m}]]
  (+ a b c))

How should I call this function? Of course.

1
2

(foo :a 1 :b 2 :c 3)
;; => java.lang.NullPointerException

Wait, what? Inline def to the rescue.

1
2
3
4
5
6
7

(defn foo [& [{:keys [a b c] :as m}]]
  (def m m) ;; TODO: remove this line before commit
  (+ a b c))

(foo :a 1 :b 2 :c 3)
;; => java.lang.NullPointerException
m ;;=> :a

Oh, of course! What happened is that m was bound to the first argument. I get it, I should have called foo like this:

1

(foo {:a 1 :b 2 :c 3}) ;;=> 6

Time to remove the inline def and get on with life.

I regularly use this technique to capture input that is part of some processing chain and I’m not sure what data is flowing through, or when the input is hard to simulate from the REPL, e.g. coming form a large HTTP request body. Recently I was working on some XML parsing. For every XML file this function was called:

1
2
3
4

(defn process-xml [xml-as-str]
  (def xml-as-str xml-as-str) ;; TODO: remove this line before commit
  ;; processing
  )

That function was part of larger flow that reads from a directory of zipfiles containing XML files. After processing a few files I would get an exception. Let’s inspect xml-as-str, the last input that triggered the exception.

1

xml-as-str ;;=> ""

So the last entry from the zipfile was empty. Turns out the entry was not an XML file, but a directory represented by an empty string. Another example involved some XML file that didn’t have the format I expected. Every time I got a new exception I could quickly see what was going on and retry process-xml with xml-as-str without kicking off the flow from scratch.

1

(process-xml xml-as-str) ;;=> ok, now it works!

Next time you have the urge to reach for a debugger, you might want to become friends with inline def first.

Creating temporary vars with the same name as function arguments has the added benefit of being able to evaluate expressions within your function. This is discussed the blogpost REPL Debugging: no stacktrace required by Stuart Halloway.

*) And somewhat controversial

Did you ever need to know the date and time 30 hours from now, because that is the time you could check into your plane to EuroClojure 2016 (and you’re too lazy to do this in your head)? Or maybe you just saw an interesting Clojure library on Twitter or Reddit that you wanted to try out? How convenient would it be if you didn’t have to create a project for such one off experiments.

There are several good options in Clojure for this. In this post let’s assume we were going to try out clj-time, an excellent date and time library based on Joda Time. We’ll show how to make a script that gives you almost instantaneous access to this library from the command line using Boot. And then we’ll make it even faster using Planck.

Leiningen

For Leiningen, there is lein try. This is a plugin you can install into ~/.lein/profiles.clj. Then from the command line, just type lein try clj-time and we’re good to go:

1
2
3
4
5
6

user=> (require '[clj-time.core :as t])
nil
user=> (require '[clj-time.local :as l])
nil
user=> (t/plus (l/local-now) (t/hours 30))
#object[org.joda.time.DateTime 0x3b3c5f5f "2016-10-25T16:53:58.154+02:00"]

Boot

For Boot this story seems even simpler as there is no need to install a plugin. Boot supports an option for including dependencies from the command line. Just type boot -d clj-time repl to get a REPL with the latest clj-time as a dependency:

1
2
3
4

$ boot -d clj-time repl
;;; output omitted
boot.user=> (require '[clj-time.core :as t])
;;; etc.

Note that Boot’s repl task also supports the --eval option (-e for short), so we can already put the require on the command line:

1
2
3
4
5

$ boot -d clj-time repl -e "(require '[clj-time.core :as t]))"
Clojure 1.8.0
;;; output omitted
boot.user=> (t/plus (l/local-now) (t/hours 30))
#object[org.joda.time.DateTime 0x3489e3ab "2016-10-25T17:11:18.560+02:00"

How convenient this is. This allows us to write it as a script:

1
2
3
4

#!/usr/bin/env bash

echo "Example: (t/plus (l/local-now) (t/hours 30))"
boot -d clj-time repl -e "(do (require '[clj-time.core :as t]) (require '[clj-time.local :as l]))"

Great to have handy for EuroClojure 2017!

Note that Boot allows us to load dependencies dynamically. Suppose you’re experimenting but need another library. No need to restart the REPL. You can just type:

1
2
3
4

boot.user=> (set-env! :dependencies '[[org.clojure/core.async "RELEASE"]])
;;; output omitted
boot.user=> (require '[clojure.core.async :refer [go-loop]])
;;; continue experimenting

Plank

Wouldn’t it be great if we could also experiment with ClojureScript libraries from a REPL? For example cljs-time? The easiest way to get here is Planck, which as of now runs only on OS X.

Planck can use jar files from ~/.m2, but you have to specify the full classpath. This is easily done with the help of Boot:

1
2
3

$ boot --dependencies org.clojars.micha/boot-cp            # load with-cp task that helps exporting minimal classpath to file
       --dependencies com.andrewmcveigh/cljs-time:"0.4.0"  # load dependency you actually want to try
       with-cp -w --file .classpath                        # write classpath to a file `.classpath`

The list of dependencies is now written to .classpath. You can re-use this file if your dependency hasn’t changed.

Now we’re ready to start the Planck REPL. It’s fast! Even faster when you use the K option which caches compiled ClojureScript.

1
2
3
4

$ planck -Kc `cat .classpath` -e "(require '[cljs-time.core :as t])" -r
cljs.user=> (require '[cljs-time.local :as l])
cljs.user=> (str (-> (t/plus (t/now) (t/hours 30)) (t/to-default-time-zone)))
"20161025T220847"

Final thoughts

Typing directly in a REPL only goes so far. For larger expressions it is more convenient to write in a text editor and then send the code to the REPL. For experiments started with Leiningen or Boot you can use an nREPL client. I use CIDER. For Planck you can use inf-clojure.

That’s it. I hope this also helpful to beginners. Performing little Clojure experiments can grow into an addiction. Before you know it, you’re soaked into your first Clojure project.

Clojure is a tool that enables interactive development and runtime inspection. Even when we work in other programming languages, Clojure can still be useful. Especially when that other language lives on the JVM.

Let’s take Scala for example. Scala has a REPL. The REPL can be used to test-drive software in development. But it doesn’t really let you inspect a running program when you didn’t start it with sbt console. So let’s use Clojure for that. We will walk through a simple Scala program that allows runtime inspection of an otherwise unknown state.

We’ll need an sbt project for this example. Make a directory and put a build.sbt in it. The only dependency in this example is Clojure.

1
2
3
4
5

scalaVersion := "2.11.8"

libraryDependencies ++= Seq(
  "org.clojure" % "clojure" % "1.8.0"
)

In src/main/scala/example.scala we add the following imports:

1
2

import clojure.java.api.Clojure
import clojure.java.api.Clojure.{`var` => cvar}

We’ll be using Clojure’s Java API. In Scala var is a reserved keyword, so I’m renaming it to cvar, since I don’t like the backticks in my code.

Next, let’s create an object that will contain some random value:

1
2
3

object BusinessLogic {
  val x = Math.random() // I wonder what this value is at runtime... 
}

Also, let’s create an App so we can run our program with sbt run:

1
2

object Main extends App {
}

If we would execute sbt run, we would never know the value of x in BusinessLogic. We could add a println, but what if x was a var and it’s value would change over time? Clojure lets us inspect this value at any given point in time. We’ll start a socket server that is available since Clojure 1.8.0.

1
2
3
4
5
6
7
8

object Main extends App {
  val require = cvar("clojure.core","require")
  require.invoke(Clojure.read("clojure.core.server"))
  val startServer = cvar("clojure.core.server","start-server")
  val options = Clojure.read("""{:port 4555 :accept clojure.core.server/repl 
    :name repl :server-daemon false}""")
  startServer.invoke(options)
}

This may seem a little intimidating, so I’ll explain it line by line. On line 2 we get a reference to Clojure’s require so we can… yes, require namespaces. On line 3 we read a string so we get the symbol that require needs to load the clojure.core.server namespace. On line 4 we get a reference to the start-server var. On line 5 we define a bunch of settings. Their meaning can be found here.

In Clojure this would read as:

1
2
3
4
5
6

(require 'clojure.core.server)
(clojure.core.server/start-server
  {:port 4555
   :accept clojure.core.server/repl
   :name 'repl
   :server-daemon false})

but since we’re coming from Scala and have to use Clojure’s Java API, it looks a bit more involved.

On the last line we invoke start-server with said options. When we execute sbt run again, the process will block, because :server-daemon was set to false:

1
2
3
4
5

~/dev/scala/cljrepl $ sbt run
[info] Loading global plugins from /Users/Borkdude/.sbt/0.13/plugins
[info] Set current project to cljrepl (in build file:/Users/Borkdude/Dropbox/dev/scala/cljrepl/)
[info] Compiling 1 Scala source to /Users/Borkdude/Dropbox/dev/scala/cljrepl/target/scala-2.11/classes...
[info] Running Main

This gives us the chance to connect to the socket repl with good ol' telnet:

1
2
3
4
5
6
7
8
9
10

~ $ rlwrap telnet localhost 4555
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
user=> (import 'BusinessLogic)
BusinessLogic
user=> (BusinessLogic/x)
0.722431948099764 ;; <-- aah!
user=> :repl/quit
Connection closed by foreign host.

See what we did there?

Clojure’s socket REPL also supports initialization via a JVM property. To try this, add these lines to build.sbt:

1
2

fork := true
javaOptions := Seq("-Dclojure.server.repl={:port 4555 :accept clojure.core.server/repl :server-daemon false}")

The App can now be reduced to something like:

1
2
3

object Main extends App {
  val require = cvar("clojure.core","require")
}

This doesn’t do much except taking care that Clojure is initialized (for more info, read the last paragraph in this Stackoverflow answer).

This Scala example translates fairly straightforward to Java. Now don’t tell your boss you’re using a different programming language. After all, Clojure is just a Java library that gives you superpowers :-).

PS: it may be wise to turn this off in production because of the security risk; on the other hand, a Clojure REPL has saved my day more than once in the past!

Edit: this post made it to the front page of Hacker News. Thanks!

Boot is a new build tool for Clojure. To get acquainted with it, I decided to migrate a fairly non-trivial Leiningen project to Boot.

You can find the entire project including the Leiningen project.clj file and Boot’s build.boot file here.

Disclaimer: this is not a comprehensive Boot tutorial. For a detailed introduction to the concepts of Boot I refer to the Boot website.

Requirements

I wanted my Boot project to have the same features as my Leiningen project:

• Managing dependencies
• Setting source and resource paths
• Building ClojureScript
• Automatic reloading of Clojure and ClojureScript source code during development
• A Clojure and ClojureScript REPL
• Setting the initial namespace for a REPL
• Setting a global var like *print-length*
• Packaging the project as a standalone jar that runs in an embedded server

Walkthrough

First let’s walk through the Leiningen project.clj step by step and see how it translated into a build.boot file.

Paths

Here we tell Leiningen where our source files and resources are. Also we declare what directories must be emptied if we want to clean up generated files:

1
2
3

:source-paths ["src"]
:resource-paths ["assets" "out"]
:clean-targets ^{:protect false} [:target-path :compile-path "out/public/out"]

In the build.boot file this is done by a call to set-env!:

1
2
3
4

(set-env!
  :source-paths #{"src" "src-cljs"}
  :resource-paths #{"assets"}
  ...

Boot has the concept of immutable filesets. Each task receives a fileset and produces one. The last task outputs its fileset to a target directory, which is target by default. Boot will clean stale files from target automatically before it emits a new fileset there. There is never a need to clean something in Boot.

Dependencies

Next we describe which dependencies the project has. In Leiningen this is done as follows:

1
2
3
4
5
6
7
8
9
10
11
12

:dependencies [[org.clojure/clojure "1.6.0"]
               [org.clojure/clojurescript "0.0-3211"]
               [org.clojure/core.async "0.1.346.0-17112a-alpha"]
               [ring-server "0.4.0"]
               [org.webjars/bootstrap "3.2.0"]
               [cljs-http "0.1.30"]
               [compojure "1.3.4"]
               [liberator "0.13"]
               [fogus/ring-edn "0.2.0"]
               [clj-json "0.5.3"]
               [reagent "0.5.0"]
               [prismatic/schema "0.4.3"]]

In Boot this is done similarly, still inside the call to set-env!:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

(set-env!
    ...
    :dependencies '[[org.clojure/clojure "1.6.0"]
                    [org.clojure/clojurescript "0.0-3211"]
                    [org.clojure/core.async "0.1.346.0-17112a-alpha"]
                    [ring-server "0.4.0"]
                    [org.webjars/bootstrap "3.2.0"]
                    [cljs-http "0.1.30"]
                    [compojure "1.3.4"]
                    [liberator "0.13"]
                    [fogus/ring-edn "0.2.0"]
                    [clj-json "0.5.3"]
                    [reagent "0.5.0"]
                    [prismatic/schema "0.4.3"]
                    ...
                    ])

Initial REPL namespace

The next line we encounter in project.clj is:

1

:repl-options {:init-ns animals.server}

This makes the animals.server namespace the starting point for every REPL session. In build.boot it is accomplished like this:

1

(task-options! repl {:init-ns 'animals.server})

Boot has several tasks built in and repl is one of them. It supports the option init-ns. With task-options! you can set some default options per task that are global to the project. To see all the options that repl provides, you can issue -h from the command line:

$ boot repl -h

or call (doc repl) from a Boot REPL session.

Global var setting

Next is this line from project.clj:

1

:global-vars {*print-length* 20}

This sets the var clojure.core/*print-length* to 20. If we print collections we’ll never see more than 20 items:

1
2
3

user=> (println (range))
(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ...)
nil

In Boot I attempted to do it like this:

1

(alter-var-root (var *print-length*) (fn [v] 20))

Unfortunately this caused a bug in Boot’s jar task. Later I learned that it’s not a good idea at all to do this in Boot, because there can be multiple Clojure runtimes (pods) in one JVM. Since I was going to use this setting only in the REPL, this is a better solution:

1
2
3

(task-options!
 repl {:init-ns 'animals.server
       :eval '(set! *print-length* 20)})

Task dependencies

Leiningen has the concept of plugins. Plugins typically perform a task as part of a Leiningen build. Two popular plugins are lein cljsbuild and lein figwheel. lein cljsbuild is an interface to the ClojureScript compiler. lein Figwheel lets you push resources to a browser, typically freshly compiled ClojureScript or CSS. It also gives you a ClojureScript REPL and a web server which allows you to serve some static files or even a Ring handler. In this example I don’t use Figwheel’s web server for running my Ring handler, because I use Ring’s standalone Jetty server that comes with automatic code reloading middleware and allows for an initial function to be executed before the handler is started: features that are lacking from Figwheel as far as I know.

In Leiningen plugins are included like this:

1
2

:plugins [[lein-cljsbuild "1.0.5"]
          [lein-figwheel "0.3.1"]]

In Boot tasks are included as normal dependencies and scoped with :test:

1
2
3
4
5
6

(set-env!
  ...
  :dependencies '[[adzerk/boot-cljs "0.0-3269-2" :scope "test"]
                  [adzerk/boot-cljs-repl "0.1.9" :scope "test"]
                  [adzerk/boot-reload "0.2.6" :scope "test"]
                  [pandeiro/boot-http "0.6.3-SNAPSHOT" :scope "test"]])

The first dependency is Boot’s interface to the ClojureScript compiler. The latter three dependencies together offer more or less the same as Figwheel: a ClojureScript, live reloading of resources in the browser and a web server to serve static files or a Ring handler.

Building ClojureScript

In Leiningen ClojureScript is built using the lein cljsbuild plugin. My configuration for this plugin looks as follows:

1
2
3
4
5
6
7
8
9
10
11

:cljsbuild {:builds {:dev {:source-paths ["src-cljs" "src-cljs-dev"]
                           :figwheel {:on-jsload "animals.main/fig-reload"}
                           :compiler {:output-to "out/public/main.js"
                                       :output-dir "out/public/out"
                                       :optimizations :none
                                       :asset-path "out"
                                       :main "animals.main"
                                       :source-map true}}
                     :prod {:source-paths ["src-cljs" "src-cljs-prod"]
                              :compiler {:output-to "out/public/main.js"
                                         :optimizations :advanced}}}}

In Boot configuring the location of where generated JavaScript will end up is done by placing an .edn file at the corresponding location in the resources tree. In this project I placed it in resources/public and named it main.cljs.edn with the following content:

1
2

{:require [animals.main]
 :compiler-options {:asset-path "out"}}

This gives you the same config as in the Leiningen example with respect to the name of the main JavaScript file, :asset-path and the :main namespace.

I use different source folders for development and production, so I can have some environment specific configuration. For example, in development I enable console print and define a function for Figwheel that will be executed when new ClojureScript is pushed to the browser:

1
2
3
4
5
6
7
8
9
10
11

(enable-console-print!)

(defonce init
  (do (println "starting application")
      (reagent/render [crud/animals]
                      (js/document.getElementById "app"))))

(defn fig-reload []
  (println "reloading reagent")
  (reagent/render [crud/animals]
                  (js/document.getElementById "app")))

In my production ClojureScript I set *print-fn* to identity, because else I would get errors when there was still a println around in my code:

1
2
3
4
5

;; no println output in production code
(set! cljs.core/*print-fn* identity)

(reagent/render [crud/animals]
                (js/document.getElementById "app"))

Developing

In Leiningen I would start my development like this.

In one terminal, I would start the web server:

lein repl
(start-server)

In another terminal, I would start Figwheel:

lein clean && lein figwheel dev

Figwheel invokes the ClojureScript compiler and the ClojureScript compiler outputs JavaScript to the out directory.

Note that using this setup, I need to run two JVMs.

In Boot the development flow is one task composed of multiple tasks:

1
2
3
4
5
6
7
8
9
10
11

(deftask dev []
  (set-env!
   :source-paths #(conj % "src-cljs-dev"))
  (comp
   (serve :handler 'animals.api/handler
          :reload true
          :init 'animals.api/init)
   (watch)
   (reload :on-jsload 'animals.main/fig-reload)
   (cljs-repl)
   (cljs)))

Only one JVM needed!

There is no need to worry about cleaning directies, since each task outputs an immutable fileset that the next task can use. Generated files end up in target by default, which gets cleaned before a new fileset is committed there.

The serve task will be the first one to be invoked. By default serve runs a Jetty server, but it is possible to select http-kit. It will reload Clojure files automatically, is able to run a Ring handler and also executes an initial function before the handler is started.

The next task in our Boot pipeline is watch. This task waits for file changes in any of the source or resource paths and then invokes the rest of the pipeline. The rest of the pipeline is also invoked one time for the initial fileset. An example:

1
2
3
4

(deftask watch-example []
  (comp
    (watch)
    (show :fileset true)))

In this example show prints out the fileset that it received as a tree. When we invoke it we see the initial fileset tree. When we add a file, the watch task will invoke show again and we would see the new fileset tree with the added file.

Back to our development pipeline:

1
2
3
4
5
6
7
8
9
10
11

(deftask dev []
  (set-env!
   :source-paths #(conj % "src-cljs-dev"))
  (comp
   (serve :handler 'animals.api/handler
          :reload true
          :init 'animals.api/init)
   (watch)
   (reload :on-jsload 'animals.main/fig-reload)
   (cljs-repl)
   (cljs)))

Whenever a file changes, the reload task is invoked. This will send changed assets to the browser via a websocket connection. The task after that, cljs-repl starts an nrepl server in which it is possible to start a ClojureScript REPL. This task also covers our requirement to have a normal Clojure REPL session with our program. Finally, the cljs task compile ClojureScript to JavaScript. I am not sure if it matters if cljs-repl comes before or after watch, but cljs surely has to come after it, since it has to see new filesets for incremental compilation.

Standalone jar

The final requirement is producing a standalone jar. In Leiningen this is done with the uberjar task. We need to tell Leiningen where it can find the main namespace that will have the -main function that will be invoked when the jar is run. Also we need to aot that namespace:

1
2

:aot [animals.uberjar]
:main animals.uberjar

Before producing a standalone jar, we must invoke the ClojureScript compiler to produce production worthy JavaScript. For convenience we can make an alias that combines all these steps:

1

:aliases {"build" ["do" "clean" ["cljsbuild" "once" "prod"] "uberjar"]}

Note that uberjar will invoke clean also. One problem that arose while writing this blog was that I had the entire out directory in :clean-targets, so when cljsbuild was done, uberjar would remove its output again. You will never have this kind of problem with Boot.

In Boot our build task looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13

(deftask build-cljs []
  (set-env!
   :source-paths #(conj % "src-cljs-prod"))
  (cljs :optimizations :advanced))

(deftask build []
  (comp
   (build-cljs)
   (aot :namespace '#{animals.uberjar})
   (pom :project 'animals
        :version "1.0.0")
   (uber)
   (jar :main 'animals.uberjar)))

First we invoke the sub-task build-cljs which includes sources from src-cljs-prod and produces optimized JavaScript. The next task performs aot on the main namespace. Then a pom file is produced. The uber task adds jar entries from dependencies to the fileset. Finally, the jar task produces a jar file from the fileset with the main namespace set to animals.uberjar. I love how Boot decomposes these tasks, so you can actually see what is going on when you produce a standalone artifact.

Issues

During this blog post I ran into a couple of issues with Boot.

The first issue had to do with dependency resolution and Clojure versions. This issue has been solved. Thanks Alan Dipert!

Another issue was that the reload task didn’t have the concept of an asset-path. I needed to work around this by creating an extra route in my handler:

1
2
3
4
5

(defroutes routes
  ...
  (resources "/")
  (resources "/public") ;; extra route
  ...

This problem will be solved in a future version of reload. See this issue.

Conclusion

Leiningen is a battle tested tool and probably the safest bet if you’re just starting with Clojure. However, Boot has certainly sparked my interest. It has an elegant design and a more functional feel to it. I’ll certainly use it on a future project.

Here are my recommendations based on my brief experience with Boot.

Use Leiningen if you:

• want to get started fast and like to get help from the majority of the Clojure community
• don’t want to take risks in terms of stability
• like configuration and convenience over programmability and composition

Use Boot if you:

• like programs more than configuration files
• don’t like to think about cleaning directories
• need to run one JVM for the entire development process (in Leiningen I needed two: one for Clojure and one for ClojureScript)
• need to perform builds tasks with mutually exclusive dependencies

Thanks for reading my blogpost. Feedback is appreciated. Did I misunderstand something about Boot? Please let me know!

Credits

Thanks to Alan Dipert, Martin Klepsch, Earl S. Sauver and some other fine folks in the #boot channel on Slack.

tl;dr

November 6th 2014 I was given the chance to speak about ClojureScript and React at the Swedish developer conference Øredev. You will find the videos and slides of my talks below. The talk about ClojureScript and React seems to be more popular and it has better sound quality, so you may want to skip straight to that one.

How I got there

On October 7th I received an e-mail from the organization of Øredev. One of their speakers, Anna Pawlicka, who was going to speak about ClojureScript had cancelled (for urgent private reasons). She had passed them some names of people they could ask to replace her. Apparently I was on this list and received an e-mail. After some consideration I accepted this invitation. It would be my first appearance on a bigger (> 1000 participants) developer conference. So in the month leading up to this conference I was pretty nervous and excited. Luckily I could practice my preliminary talks at two local meetups and had lots of people providing me with useful feedback.

The talks

The conference expected two talks of both 40 minutes. 40 minutes is a short time, so I had to select my slides and the level of detail very carefully.

ClojureScript for the web

My first talk was about ClojureScript for web applications in general. I explained the most important features of Clojure(Script). Unfortunately the sound volume of the video is a bit low.

The description of the talk for the conference was:

Over the last few years we have seen the rise of browser applications. Instead of rendering all UI server side, JavaScript driven client applications are now being widely adopted. While JavaScript is a flexible and powerful language, it has its shortcomings. This is where languages that compile to JavaScript step in. ClojureScript is one of them and offers its own powerful features to the front end developer. In this talk you will get an overview of what ClojureScript development looks like and how it may simplify your application.

Video:

CLOJURESCRIPT FOR THE WEB from Øredev Conference on Vimeo.

The slides can be downloaded here. Related code is here.

ClojureScript interfaces to React

The second talk was about using ClojureScript in combination with the React library. I showed two approaches: Om and Reagent. I have spent more time and detail on Reagent, because this library is easier to explain in a 40 minute talk.

The description of the talk for the conference:

React is a JavaScript library for creating declarative UIs. It was created by Facebook to simplify writing applications consisting of many components. React allows you to describe how the UI should look and renders it automatically via one way data binding. It achieves good performance by using a virtual DOM that prevents unnecessary and expensive DOM manipulations. Even better performance can be achieved by leveraging the immutable data structures of ClojureScript. This is an approach taken by Om and Reagent. In this talk you will get an impression of using ClojureScript together with React in practice.

Video:

CLOJURESCRIPT INTERFACES TO REACT from Øredev Conference on Vimeo.

The slides can be downloaded here. Related code is here.

The conference

Øredev is very well organized. I didn’t have to worry about a thing: plane tickets, hotel, vegan food, quality coffee, ice breaking social events: all arranged by them. If you ever have the chance to speak at Øredev: I can highly recommend it!

I wasn’t the only person doing Clojure-related talks. Ryan Neufield, the main author of the Clojure Cookbook and Pedestal contributor was there talking about Datomic and Pedestal. Rob Ashton shared his lessons learned while creating a database with Clojure. Neal Ford talked on functional thinking in Java, Scala and Clojure. Here’s a picture of Ryan presenting about Pedestal:

Thanks

During these 40 minute talks I didn’t have the time to thank people who helped me during my month of preparation in one way or another. Here is a list of people and companies I would like to thank for their help or support:

• Anders Janmyr
• Anna Pawlicka
• Barbara Borkent
• David Nolen
• Denis Fuenzalida
• Emily Holweck
• Finalist (company)
• Jayway (company)
• Martin van Amersfoorth
• Matthijs Steen
• Ustun Ozgur
• Vijay Kiran

Resources

Lastly, for what it’s worth, here is a raw list of resources that I found interesting to study while I was preparing my talks. Have fun with those!

• Clojurescript Up and Running
• http://www.infoq.com/news/2014/01/om-react
• http://www.lexicallyscoped.com/2013/12/25/slice-of-reactjs-and-cljs.html
• http://adamsolove.com/js/clojure/2014/01/06/om-experience-report.html
• http://www.joshlehman.me/rewriting-the-react-tutorial-in-om/
• https://t.co/bzQcj0OsPW - PureScript
• http://2013.jsconf.eu/speakers/pete-hunt-react-rethinking-best-practices.html
• https://www.youtube.com/watch?v=SiFwRtCnxv4 - Nolen on immutability
• https://www.youtube.com/watch?v=-DX3vJiqxm4 - Secrets of the Virtual DOM
• http://www.funnyant.com/reactjs-what-is-it/
• https://twitter.com/swannodette/status/407750614727524352 - historic tweet of David Nolen porting React code to Cljs
• http://spootnik.org/entries/2014/10/26_from-angularjs-to-om-a-walk-through.html - from Angular to Om
• http://www.infoq.com/presentations/ClojureScript-Javelin
• https://github.com/enaqx/awesome-react
• http://stackoverflow.com/questions/21109361/why-is-reacts-concept-of-virtual-dom-said-to-be-more-performant-than-dirty-mode
• https://github.com/swannodette/om/blob/master/src/om/core.cljs#L250
• https://www.youtube.com/watch?v=noiGVQoyYHw#t=72 - Clojure: 10 big ideas
• https://keminglabs.com/blog/cljs-app-designs/ - A sampler of ClojureScript application designs
• http://teropa.info/blog/2013/10/18/single-page-webapps-in-clojurescript-with-pedestal.html - SPA in Pedestal
• https://groups.google.com/d/topic/pedestal-users/jODwmJUIUcg/discussion - Why Pedestal App is (probably) discontinued
• http://blog.cognitect.com/blog/2014/10/24/analysis-of-the-state-of-clojure-and-clojurescript-survey-2014 - Clojure 2014 survey

How to combine figwheel, Om and Ring in one application

tl;dr

In this article you will find a configuration of figwheel with Om and a Ring server in one application.

The following issues will be addressed:

• Om root component will be remounted upon code reload
• App-state is lost when code is reloaded
• Code changes in other cljs files do not cause a re-render

Let’s get Om with it

For a few weeks I have been using Clojurescript and Om for front end development. Om is a Clojurescript library based on the famous React. Figwheel is a tool that can speed up Clojurescript and Om development by reloading code in the browser without refreshing an entire page.

Figwheel comes in the form of a leiningen plugin that uses lein-cljsbuild to compile Clojurescript and pushes the resulting Javascript to the browser, which is then reloaded. Together with React this is a powerful combination. As you change code in your editor and press save, the changes can be instantly reflected in the page you were viewing… if you configure things properly.

The project I’m currently working on consists of several Clojurescript files. Often an Om component resides in a file of its own. For my workflow it would make sense to be able to edit a component in one file, save the code and see the change in the browser immediately. So here is what I’ve done.

Let’s make a project based on the wrom template.

1
2
3
4
5
6
7
8
9
10
11
12
13

$ lein new wrom example
$ find .
.
./.gitignore
./project.clj
./resources
./resources/public
./resources/public/index.html
./resources/public/style.css
./src
./src/example
./src/example/client.cljs
./src/example/server.clj

The application comprises a server and client part. All sources files reside in one directory src. Let’s add figwheel to our project. Add [figwheel "0.1.4-SNAPSHOT"] to :dependencies and [lein-figwheel "0.1.4-SNAPSHOT"] to plugins. Your project.clj should now look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

(defproject example "0.1.0-SNAPSHOT"
  :description "FIXME: write this!"
  :url "http://example.com/FIXME"

  :dependencies [[org.clojure/clojure "1.6.0"]
                 [org.clojure/clojurescript "0.0-2322"]
                 [org.clojure/core.async "0.1.267.0-0d7780-alpha"]
                 [org.webjars/react "0.11.1"]
                 [om "0.7.1"]
                 [cljs-http "0.1.14"]
                 [ring/ring-core "1.3.1"]
                 [ring/ring-jetty-adapter "1.3.1"]
                 [figwheel "0.1.4-SNAPSHOT"]]

  :plugins [[lein-cljsbuild "1.0.4-SNAPSHOT"]
            [lein-figwheel "0.1.4-SNAPSHOT"]
            [lein-ring "0.8.10"]]

  :source-paths ["src"]

  :cljsbuild {:builds [{:id "example"
                        :source-paths ["src"]
                        :compiler
                        {:output-to "resources/public/example.js"
                         :output-dir "resources/public/out"
                         :optimizations :none
                         :source-map true}}]}

  :ring {:handler example.server/app
         :nrepl {:start? true :port 4500}
         :port 8090}

  :global-vars {*print-length* 20})

Let’s edit client.cljs. In the namespace declaration under :require add [figwheel.client :as fw]. Now we’ll hook up figwheel so it can send changes in our project to a browser. Because we use Ring with the Jetty adapter as an external server, we have to tell figwheel explicitly where it’s websocket is, since it can’t just connect to the same origin.

1
2
3
4
5

(fw/watch-and-reload
 :websocket-url   "ws://localhost:3449/figwheel-ws"
 :jsload-callback
 (fn []
   (println "reloaded")))

Start Ring and Figwheel independently, both from inside the example directory. In one terminal type lein figwheel (or optionally supply the name of the build: lein figwheel example). It’s best to wait for the clojurescript compilation to complete before starting Ring.

1
2
3
4
5
6
7
8
9

$ lein figwheel
Compiling ClojureScript.
Figwheel: Starting server at http://localhost:3449
Figwheel: Serving files from '(dev-resources|resources)/public'
Compiling "resources/public/example.js" from ["src"]...
Successfully compiled "resources/public/example.js" in 20.297 seconds.
notifying browser that file changed:  /example.js
notifying browser that file changed:  /out/goog/deps.js
notifying browser that file changed:  /out/example/client.js

In another terminal type lein ring server. If you’re lucky a browser will open automatically to localhost:8090/index.html and you will see the text Hello world from server!. This text really came from the Ring handler in server.clj. If you open a developer console, you will see that figwheel has connected to the browser:

Now let’s see if figwheel will pick up a code change in client.cljs. Let’s change the render function to:

1
2
3
4
5

(render [_]
       (dom/div
        nil
        (dom/h1 nil (:text app))
        (dom/p nil "Hello from Michiel!")))

If everything worked, in the browser you will almost immediately see that your change is reloaded and re-rendered by Om.

Om root component will be remounted upon code reload

One problem of making changes in the file where the call to om/root resides is that the entire component tree will be unmounted and replaced by a new instance. To verify my claim, let’s add line 6 and 19-21 from the snippet below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

(om/root
 (fn [app owner]
   (reify
     om/IWillMount
     (will-mount [_]
       (println "I will mount")
       (go (let [response (<! (http/get
                               (str "/welcome-message")))]
             (if (= (:status response)
                    200)
               (om/update! app :text (:body response))
               (om/update! app :text "Server error")))))
     om/IRender
     (render [_]
       (dom/div
        nil
        (dom/h1 nil (:text app))
        (dom/p nil "Hello from Michiel!")))
     om/IWillUnmount
     (will-unmount [_]
       (println "I will unmount"))))
 app-state
 {:target (. js/document (getElementById "app"))})

Now make a change anywhere in the source code and save the file. In the browser’s console you will see that Om mounted the component. Now change the source again and press save. You’ll see that Om unmounted the component and mounted a new instance of the component. You really have to pay attention to this, because if you created resources or go loops in will-mount and you didn’t clean them up in will-unmount, things can get really messy when you have lots of code reloads. The go loops from unmounted instances will just keep running and your program can get unpredictable. So, the solution to this problem is to keep track of your resources and take the appropriate actions in the will-unmount.

App-state is lost when code is reloaded

Let’s change our app state to

1

(def app-state (atom {:text "" :button "unclicked"}))

and the render function to

1
2
3
4
5
6
7

(render [_]
       (dom/div
        nil
        (dom/p nil (pr-str app))
        (dom/button #js {:onClick #(om/update! app
                                              :button "clicked")}
                    "Click to change state")))

In the render function we show a printed version of the cursor. Now let’s click the button. The app-state updated and the component is reflecting this. Now let’s change the code of client.cljs and save. What do we see? Our app-state is back to the initial state. This is because app-state is being redefined by the code reload. To avoid this, change def to defonce:

1

(defonce app-state (atom {:text "" :button "unclicked"}))

Now the app-state will survive a code reload. Sometimes it’s that easy to write re-loadable code.

Code changes in other cljs files do not cause a re-render

Let’s make a second file called child.cljs like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

(ns example.child
  (:require-macros [cljs.core.async.macros :refer (go)])
  (:require [om.core :as om :include-macros true]
            [om.dom :as dom :include-macros true]))

(defn child [cursor owner]
  (om/component
   (dom/div nil
            (dom/p nil "I'm a child.")
            (dom/p nil (str "I have local state: "
                            (pr-str (om/get-state owner))))
            (dom/button #js {:onClick
                             #(om/update-state! owner
                                                :clicks inc)}
                        "Click me to update my local state"))))

Let’s require and om/build this child component in our root component, so it will appear on the page. I omitted irrelevant parts.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

(ns example.client
  ...
  (:require [om.core :as om :include-macros true]
            ...
            [example.child :refer (child)]))

...

(om/root
 (fn [app owner]
   (reify
     ...
     om/IRender
     (render [_]
       (om/build child app))
     ...  )))
 app-state
 {:target (. js/document (getElementById "app"))})
...

I’m not entirely sure if figwheel handles changes in namespace declarations well, so just refresh the page. You will see the button from the child component on the screen. Click a few times:

Now let’s change some code of the child component. For example, change the text in the button to “Click me to update me!”. When saved, you won’t see the change reflected in your browser. figwheel has reloaded child.cljs but the problem is that Om doesn’t ‘know’ about this. Let’s tell Om. Let’s summon the power of core.async.

Change the :require entry for core.async to [cljs.core.async :refer [<! chan put!]] so we can create channels and put something in them.

In client.cljs add a channel. Place it below the definition of app-state:

1

(defonce re-render-ch (chan))

In will-mount we’ll now spawn a go loop that keeps reading from re-render-ch:

1
2
3
4
5
6

(will-mount [_]
  (println "I will mount")
  (go (loop []
    (when (<! re-render-ch)
      (om/refresh! owner)
      (recur))))

All we have to do now is put a (truthy) message into the channel and the root component will re-render itself. This can be done in the callback provided to fw/watch-and-reload:

1
2
3
4
5
6

(fw/watch-and-reload
 :websocket-url "ws://localhost:3449/figwheel-ws"
 :jsload-callback
 (fn []
   (println "reloaded")
   (put! re-render-ch true)))

Refresh the page so everything is in place. Now change the text of the button again in child.cljs and you’ll see the component being instantly re-rendered in the browser. Observe however that all local state in the child component is lost. Because the code of the child component changed, Om has to remount the component. Again, be conscious of resources and go loops hat are creating during the mount phase and clean them up if necessary.

Let’s change to child to show and update the cursor instead of local state.

1
2
3
4
5
6
7
8
9
10
11

(defn child [cursor owner]
  (om/component
   (dom/div nil
            (dom/p nil "I'm a child.")
            (dom/p nil (str "My cursor:"
                            (pr-str cursor)))
            (dom/button #js {:onClick
                             #(om/transact!
                               cursor
                               :clicks inc)}
                        "Click me!"))))

Click a few times and you’ll see something like this:

Now change the text of the button in child.cljs again and save. You’ll see that the state of the cursor is preserved, as expected.

Update October 11th 2014

The use of a channel to refresh a component was a little invasive and to be honest, I didn’t like it much. Arne Brasseur pointed out in the comments that refreshing of Om components can be implemented simpler, because om/root is idempotent. This means it may be called multiple times and all will be well.

I changed the code in main.cljs according to the suggestion of Arne:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

(ns example.client
  (:require-macros [cljs.core.async.macros :refer (go)])
  (:require [om.core :as om :include-macros true]
            [om.dom :as dom :include-macros true]
            [cljs.core.async :refer [<! chan put!]]
            [cljs-http.client :as http]
            [figwheel.client :as fw]
            [example.child :refer (child)]))

(enable-console-print!)

(defonce app-state (atom {:text ""}))

(defn main []
  (om/root
   (fn [app owner]
     (reify
       om/IRender
       (render [_]
         (dom/div
          nil
          (om/build child app)))
       om/IWillUnmount
       (will-unmount [_]
         (println "I will unmount"))))
   app-state
   {:target (. js/document (getElementById "app"))}))

(fw/watch-and-reload
 :websocket-url "ws://localhost:3449/figwheel-ws"
 :jsload-callback
 (fn []
   (println "reloaded")
   (main)))

(defonce initial-call-to-main (main))

The differences:

• no invasive channel and call to om/refresh!
• the call to om/root is now wrapped inside a function main
• main is called on initial page load and will be called by figwheel upon code reload (no matter which ClojureScript file, which the point of putting the call to main here)

Arne Brasseur is the author of the excellent Chestnut template, that embeds Figwheel as one of its dev tools. I suggest you try it out if you haven’t. David Nolen, the author of Om, just published a demo video of Chestnut.

The completed (and updated) code of this blog post can be viewed here.

If you liked my post or want to suggest an improvement, please leave a comment. Thanks for reading!