Saturday, January 28, 2017

I recently released a typed and total configuration language named Dhall that you can use to configure Haskell programs. However, Dhall would be more useful if you could configure other programming languages, like Nix.

Nix powers the Nix package manager and the Nix operating system and if you've never used Nix before then please give Nix a try. We use Nix heavily at Awake Networks for managing our deployments but one of our biggest complaints about Nix is the type system:

Nix expressions cannot be annotated with types to guide the user

You can't use sum types to make invalid states unrepresentable

Type errors are not very helpful and poorly located

Many Nix builtins and idioms are inherently difficult to statically analyze

To mitigate this problem I contributed a new Dhall-to-Nix compiler that lets you carve out a restricted subset of Nix with a more user-friendly type system.

This post covers:

How to use this in your own Nix projects

Details of the translation process from Dhall to Nix

Benefits and drawbacks of using Dhall to program Nix

User interface

I began by creating a Dhall to Nix compiler that can translate arbitrary Dhall expressions to equivalent Nix expressions. This compiler is not limited to simple values like records or literals: you can even compile Dhall functions to Nix functions.

The compiler takes a Dhall expression on standard input and emits the corresponding Nix expression on standard output:

However, you do not need to install or manually run the compiler to use Dhall within your Nix project. I went a step further and added a dhallToNix utility to nixpkgs which automatically converts Dhall expressions to Nix expressions for you. This utility automatically bootstraps and caches the dhall-to-nix compiler as part of the evaluation process.

Here's an example of how you can use the dhallToNix function to embed arbitrary Dhall expressions inside of Nix:

Built-in functions which were missing in Nix required the most effort. I had to translate them to efficient implementations based on other Nix-builtins. For example, Dhall's List/reverse primitive uses Nix's builtins.genList and builtins.elemAt under the hood:

The last tricky part was translating Dhall's explicit type abstraction and type application. "Explicit type abstraction" means that polymorphic (or "generic") functions in Dhall are encoded as ordinary functions that take a type as a function argument. For example, this is how you encode the polymorphic identity function in Dhall:

λ(a : Type) → λ(x : a) → x

"Explicit type application" means that you specialize polymorphic functions by applying them to a type argument specifying which type to use. For example, this is how you use the polymorphic identity function:

Some Dhall built-in functions are also polymorphic, and we treat them the same way. For example, the List/reverse function is polymorphic, which is why the first argument in the corresponding Nix expression is an unused type argument named t:

Finally, floating point numbers are not supported in Nix at all, so the dhall-to-nix compiler must reject Double values:

$ dhall-to-nix<<<"1.0"Error: No doubles
Explanation: Dhall values of type ❰Double❱ cannot be converted to Nix
expressions because the Nix language provides no way to represent floating point
valuesYou provided the following value:
↳ 1.0... which cannot be translated to Nix

Benefits

When first learning Nix, particularly NixOS, you'll frequently run into the issue where you're not sure what values you're expected to provide due to the lack of a type system. Dhall can fix that in several ways:

You can request the inferred types of functions so that you know what type of function argument you need to supply

You can also provide users with a "schema" for an expected value, which in Dhall is just an ordinary type annotation pointing to a remote path

You can replaced weakly typed values (like strings) with more strongly typed representations

The following example will illustrate all of the above points

For example a derivation in Nix can be minimally represented in Dhall as the following type:

{ name : Text, builder : Text, system : Text }

... which you can save to a file named derivation and use to check if other expressions match the expected type:

$ dhall --explain <<EOF
{ name = "empty", builder = "touch $out", sytem = "x86_64-linux" }
: ./derivation
EOF
Error: Expressiondoesn't match annotationExplanation: You can annotate an expression with its type or kind using the❰:❱ symbol, like this: ┌───────┐ │ x : t │ ❰x❱ is an expression and ❰t❱ is the annotated type or kind of ❰x❱ └───────┘The type checker verifies that the expression's type or kind matches the
provided annotation
For example, all of the following are valid annotations that the type checker
accepts:
┌─────────────┐
│ 1 : Integer │ ❰1❱ is an expression that has type ❰Integer❱, so the type
└─────────────┘ checkeraccepts the annotation
┌────────────────────────┐
│ Natural/even +2 : Bool │ ❰Natural/even +2❱ hastype ❰Bool❱, so the type
└────────────────────────┘ checker acceptsthe annotation
┌────────────────────┐
│ List : Type → Type │ ❰List❱ is an expression that has kind ❰Type → Type❱,
└────────────────────┘ so thetype checker accepts the annotation
┌──────────────────┐
│ List Text : Type │ ❰List Text❱ is an expression that has kind ❰Type❱, so
└──────────────────┘ the type checker accepts the annotation
However, the following annotations are not valid and the type checker will
reject them:
┌──────────┐
│ 1 : Text │ The type checker rejects this because ❰1❱ does not have type
└──────────┘ ❰Text❱
┌─────────────┐
│ List: Type │ ❰List❱ does not have kind ❰Type❱
└─────────────┘
You or the interpreter annotated this expression:
↳ { builder = "touch $out", name = "empty", sytem = "x86_64-linux" }
... with this type orkind:
↳ { builder : Text, name : Text, system : Text }
... but the inferred type or kind of the expression is actually:
↳ { builder : Text, name : Text, sytem : Text }
Some common reasons why you might get this error:
● The Haskell Dhall interpreter implicitly inserts a top-level annotation
matching the expected type
For example, if you run the following Haskell code:
┌───────────────────────────────┐
│ >>> input auto "1" :: IO Text │
└───────────────────────────────┘
... then the interpreter will actually type check the following annotated
expression:
┌──────────┐
│ 1 : Text │
└──────────┘
... and then type-checking will fail
────────────────────────────────────────────────────────────────────────────────

We can also take advantage of the fact that Dhall supports sum types so that we can make invalid states unrepresentable. For example, the system field really shouldn't be Text because not all strings are valid systems.

We can fix this by first creating a type more accurately representing all supported platforms. First, we just need to create a sum type for all supported operating systems:

However, in practice you'd like to keep the platform expression as the original strongly typed record instead of converting the platform to a string. The original record lets you more easily extract the architecture and operating system fields and make decisions on their values using exhaustive pattern matching.

Drawbacks

The largest drawback of using Dhall to program Nix is that Dhall cannot encode many common idioms used in nixpkgs. Some examples of idioms that do not translate well to Nix are:

The callPackage idiom that nixpkgs uses very heavily for easily updating dependencies. This relies on reflection and recursive fixpoints, neither of which Dhall supports

Anything which uses builtins.listToAttrs or does reflection on record field names

I don't expect Dhall to be used at all in nixpkgs, but I do believe Dhall can benefit end users or companies for their own internal Nix projects.

Conclusion

The dhallToNix utility is available right now in the nixpkgs-unstable channel if you would like to try this out in your own project:

Also, if you would like to use the dhall-to-nix compiler for other purposes you can find the compiler on Hackage or Github: