From plumatic’s schema to clojure.spec

In a previous blog post, I showed an example of using plumatic’s schema with test.check and test.chuck. With the introduction of Clojure’s new spec library, I thought it would be interesting to revisit that post and port it from schema to spec. The code from this post is available on github.

Overall, the port was relatively straight-forward, though spec took some getting used to. spec provides similar facilities for what I was using in schema. It integrated with both test.check and test.chuck with no significant modifications!

First, we need to update our project.clj to include the latest alpha of Clojure 1.9 (alpha10 as of this writing):

The first thing you notice is things are slightly more verbose in spec, but there’s also more going on. For example, in the case of the ::rest spec, I need a custom generator which is more restrictive than the one automatically created by spec during generative testing. the :gen option on the spec definition allows you to pass in a generating function as part of the definition – making the generator just as re-usable as the spec. This is really powerful.

The ::note spec is more powerful than it’s schema counterpart too. I no longer need a special NoteGenerator in my test namespace.

Now, let’s compare some of the simple functions. Here are some the schema versions:

1

2

3

(s/defnrest?:-s/Bool[n:-NoteOrRest](neg?n))

(s/defnnote-count:-s/Int[notes:-[NoteOrRest]]

(count(removerest?notes)))

And here are the spec versions:

1

2

3

4

5

6

7

8

9

10

(defnrest?[n](neg?n))

(s/fdefrest?

:args(s/cat:n::note-or-rest)

:retboolean?)

(defnnote-count[notes](count(removerest?notes)))

(s/fdefnote-count

:args(s/cat:notes::notes)

:retinteger?

:fn#(<=(:ret%)(->%:args:notescount)))

The spec versions look a fair bit more verbose at first glance, compared to their compact-looking schema counterparts. Aside from some indentation, the main difference is that the specification of the functions exists separately of the function. But take a look at the function spec for note-count. It actually goes further than the schema version. It encodes an invariant using the arguments and return type of the function. We’ll see later on how this is used during testing.

Now let’s look at the main function we are testing with test.check/chuck. Here’s the schema version:

1

2

3

4

5

6

7

8

9

(s/defnwith-new-notes:-Melody

[melody:-Melodynew-notes:-[Note]]

{:pre[(=(countnew-notes)(note-count(:notesmelody)))]}

(let[notes(first(reduce(fn[[updated-notesnew-notes]note]

(if(rest?note)

[(conjupdated-notesnote)new-notes]

[(conjupdated-notes(firstnew-notes))(restnew-notes)]))

[[]new-notes](:notesmelody)))]

(->Melodynotes)))

And here’s the spec version:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

(defnwith-new-notes[melodynew-notes]

(let[notes(first(reduce(fn[[updated-notesnew-notes]note]

(if(rest?note)

[(conjupdated-notesnote)new-notes]

[(conjupdated-notes(firstnew-notes))(restnew-notes)]))

[[]new-notes](:notesmelody)))]

(->Melodynotes)))

(s/fdefwith-new-notes

:args(s/cat:melody::melody

:new-notes(s/coll-of::note:kindvector?))

:ret::melody

:fn(s/and#(=(->%:args:new-notescount)(note-count(->:ret%:notes)))

#(=(->%:args:new-notes)(removerest?(->:ret%:notes)))))

Once again, there’s more going on in the spec version. We are able to specify the invariants of the function right in the definition of the function spec. Nice.

Testing

At first I didn’t really realize why spec actually wraps test.check. Then I watched this video. The cool part is that the library will not only be able to generate test data based on specs in many cases (like schema does), but it will also use the :fn defined in function specs as a property for the tests so that you don’t have to write them yourself! So if you define a :fn for your fdef, it will be used as a property within the test.

Integrating the specs and their respective generators for use with test.check or test.chuck is straight-forward. In schema we had a few generators we needed to setup our tests:

1

2

3

(defPositiveInt(s/constraineds/Intpos?))

(defRestGenerator(sgen/always-1))

(defNoteGenerator(gen/chooseMIN-NOTEMAX-NOTE))

Here are the spec equivalents:

1

2

(defnnotes-gen[size](gen/vector(s/gen::core/note)size))

(defnrests-gen[size](gen/vector(s/gen::core/rest)size))

Notice we no longer define a spec analog for PositiveInt. We don’t need one since Clojure 1.9 includes a predicate in core called pos-int?.

The notes-and-rests-gen genator is just as before:

1

2

3

4

5

6

(defnnotes-and-rests-gen[sizenum-notes]

(gen/bind(notes-gennum-notes)(fn[v]

(let[remaining(-sizenum-notes)]

(if(zero?remaining)

(gen/returnv)

(gen/fmap(fn[rests](shuffle(intovrests)))(rests-genremaining)))))))

For the melody generator, we need to pass in our overrides. In schema this looked like this:

Here we are creating a generator for our melody spec, but with a custom generator for our notes vector. Our actual test.check and test.chuck tests are unchanged!

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

;;

;; test.check version

;;

(defspecwith-new-notes-test-check1000

(let[test-gens(tcg/let[num-notestcg/s-pos-int

melody-num-reststcg/s-pos-int

total-melody-num-notes(tcg/return(+num-notesmelody-num-rests))

melody(melody-gentotal-melody-num-notesnum-notes)

new-notes(notes-gennum-notes)]

[melodynew-notes])]

(prop/for-all[[melodynew-notes]test-gens]

(let[new-melody(with-new-notesmelodynew-notes)]

(and(=(countnew-notes)(note-count(:notesnew-melody)))

(=new-notes(removerest?(:notesnew-melody)))

(=(count(:notesmelody))(count(:notesnew-melody))))))))

;;

;; test.chuck version

;;

(defspecwith-new-notes-test-chuck1000

(tcp/for-all[num-notestcg/s-pos-int

melody-num-reststcg/s-pos-int

total-melody-num-notes(gen/return(+num-notesmelody-num-rests))

melody(melody-gentotal-melody-num-notesnum-notes)

notes(notes-gennum-notes)]

(let[new-melody(with-new-notesmelodynotes)]

(and(=(countnotes)(note-count(:notesnew-melody)))

(=notes(removerest?(:notesnew-melody)))

(=(count(:notesmelody))(count(:notesnew-melody)))))))

Well, except for one thing actually. During the testing of the code in this post, I actually noticed I was missing an essential property! While I was trying to prove to myself that this still worked, I began intentionally breaking the with-new-notes method. I made a change where it just blindly added only the notes, and didn’t include the rests, but the tests still passed! It turned out I needed one more critical property for correctness:

1

(=(count(:notesmelody))(count(:notesnew-melody)))

This ensures the the returned melody still has the same number of notes as the original ones passed in, including rest notes.

Finally, I added the spec/check version. Spec includes a check function which wraps test.check’s quick-check, first checking function :args, and passing any :fn defined along to quick-check as a property. You can also pass in all the options supported by quick-check through a special options map:

1

2

3

(stest/check`with-new-notes{:gen{::core/melody#(melody-gen54)

::core/notes-only#(notes-gen4)}

:clojure.spec.test.check/opts{:num-tests100}})

This is pretty short!

I did get a bit lazy here. Notice I’m hard-coding the sizes for the melody and notes args. The :gen overrides map requires that the provided overrides be no-arg functions returning generators. Not exactly sure, but I suspect there is a way to do some kind of bind/fmap type of magic to mimic the size generators in the test.check/chuck versions.

Impressions

Compared to spec, schema felt a bit more natural to me at first coming from a more strongly typed background as the schema information is right there next to what it is specifying. But there’s an elegant simplicity to the spec approach which I like as well. It’s more composable and feels like it fits into the language more naturally, and thus can be leveraged in more ways, without needing to lean too much on the capabilities of macros syntactically.

I love how any predicate can be used as a spec. This makes perfect sense and allows for a more concise way of specifying the same thing without the need for special functions from the library. And of course a lot of existing validation and other code out there can be leveraged immediately in the spec system.

The regex approach to defining sequential specs is perhaps the most impressive aspect of the design to me. This feels really powerful and flexible, allowing arbitrarily complex definitions to be put together using a technique that’s already really well understood by programmers.

Of course one of the big wins in spec will be the tooling support afforded by the language itself. Built-in documentation support, etc.

I noticed in 1.9.0 alpha8 a new namespaced map reader syntax was added. Wondering when something equivalent will be defined or somehow supported automatically for records. The :req-un and :opt-un syntax is awkward.

Clojure namespaced keywords initially tripped me up a bit. I hadn’t really used them before. Once you start referencing them in other namespaces you quickly learn a nice little thing that clojure supports. Basically you can alias a namespace as usual and then use the namespace qualifying syntax and it works! I refer :all the forms from the core namespace and add an alias so I can refer to the namespaced symbols conveniently, where I make reference to my specs defined in the main source namespace under test.

I’ve seen some debate about where the best place to put specs are; to me it seems pretty logical to co-locate them directly with the code they spec. This is most analogous to the way typed languages work. Why put them somewhere else? You want them where the code is defined. I suppose one exception might be truly generic specs meant for supporting arbitrary code, say a small library of common specs.

I’m interested to see whether spec can improve Clojure’s error messages. I haven’t looked at the source for the alpha to see whether any of Clojure itself is spec’d, but I would imagine when (if?) this code is spec’d along with common libraries, that there may be fewer seemingly obscure exceptions happening in code like class cast exceptions and the like.

The experience of porting the code from schema reminds me its important to practice REPL-driven development and build things up so you can really understand where a problem is. In spec’s case, use the new tools you get from spec like checking conformance, generating some samples to make sure things work as you expect, etc. as it will make tracking down issues much easier. Not checking your spec’s before using them in property-based tests can lead to some strange test failures where it’s not immediately obvious what’s breaking.

Tracking Alpha Changes

As of this writing, spec is still evolving. The documentation in the guide isn’t always keeping pace with the changes. The best way I’ve found to track it is the announcements in the Clojure google groups forum. There you can find changelogs for each successive alpha.