The Visitor Pattern is one of the most mis-understood of the classic design patterns. While it has a reputation as a slightly roundabout technique for doing simple processing on simple trees, it is actually an advanced tool for a specific use case: flexible, streaming, zero-overhead processing of complex data structures. This blog post will dive into what makes the Visitor Pattern special, and why it has a unique place in your toolkit regardless of what language or environment you are programming in.

Json: a Strawman Data Structure

The Visitor Pattern operates on structured data: often, but not always, hierarchical or tree-like data structures. For the sake of this article, we will base our discussion on a subset of the JSON data format with only three kinds of value:

This should serve to illustrate the usage of the Visitor Pattern, without the overwhelming complexity of real data formats, and allow us to discuss the techniques and approaches in the limited confines of a blog post. The techniques described on this "Json-lite" format all apply to dealing with more complex real-world formats.

Modeling such a JSON tree structure in Scala code, we might write something like this:

* For each JSON string, acceptStr is called
* For each JSON number, acceptNum is called
* For each JSON dictionary, acceptDict is called
* Within that JSON dictionary, acceptKey is called for each key
* acceptValue() is called before each dictionary value, and the Visitor it returns is used when visiting that valueʼs JSON nodes
* acceptValue(value: T) is called on the result of the visiting that value

Exactly how "for each JSON {string, number, dictionary}" is implemented, is left to a separate dispatch function.

For now, I am assuming that Visitor is generic: itʼs methods return a type T, representing the "output" of this Visitor. T will vary depending on what the concrete Visitor implementation is trying to do:

* If itʼs meant to serialize the Json tree then T might be a String
* If itʼs meant to redact sensitive keys from the input data, or convert number-like strings into proper numbers, then T might be Json.
* If itʼs meant to perform some summary statistics on the Json tree, e.g. summing up all the numbers within it, then T might be an Int

Next, we have to write the dispatch function that takes both a Json tree, and a Visitor object, and calls the various methods on the Visitor depending on what it sees in the tree. For example:

Letʼs look at some concrete implementations for Visitor that accomplish the three things mentioned above:

StringifyVisitor

A simple Visitor that renders the Json structure to a String. Not as efficient as it could be - a production-quality serializer may want to build up the output using a StringBuilder or render directly to an output stream - but as an example it works well enough.

So far, we have seen some ways of using the Visitor Pattern to process our Json trees. Nevertheless, in these simple cases, it seems like a very roundabout way of doing something really simple: I could just as easily have written simple functions that recurse over the Json tree and do what I want.

In doing so, I have accomplished the same outcome as we did earlier using the Visitor Pattern, but with a single function rather than one function and two classes! Here were are using Scalaʼs match pattern matching syntax, but it could just as easily be done using Javaʼs instanceof and (Str) casts, or Pythonʼs isinstance. What, then, is the point of all this Visitor stuff?

Chaining Recursive Functions

The recursive transformation functions written above can be chained: as long as the types line up, the output of one can be trivially fed into another, for example performing a summation after redaction, or stringifying the redacted Json trees:

However, there is a downside to this approach: each recursive transformation you chain in this way creates an entire intermediate Json tree structure to pass to the next transform in the chain. In the above examples thereʼs only one intermediate tree - the output of redact - but you can easily imagine chains of transformations with dozens of stages: redact some sensitive data, convert all strings to lowercase, sort the keys of the dictionaries, etc. In such cases, creating and throwing away dozens of intermediate trees is wasteful and can be a performance bottleneck.

There is a solution to this problem: to fuse the different recursive transformations together manually. For example, you may fuse summation and toInt and redact into a single redactToIntSum function as follows:

This gets us the efficiency we want - we no longer are constructing an intermediate tree in redact just to pass it to toInt, and construction and intermediate tree in toInt just to pass to summation - but at a cost of flexibility. We have to manually fuse the recursive transformation we want into a single function and can no longer mix-and-match the different transforms as weʼd like. Manually fusing all possible combinations would require O(n!) different fused functions, which quickly becomes unfeasible.

Chaining Visitors

Like recursive functions, Visitors can also be chained. A slight modification is necessary to the RedactTreeVisitor above to make this possible:

Note how the above code is slightly different from the RedactVisitor we saw earlier: rather than immediately constructing a Json value to return, it simply forwards to a downstream: Visitor[T] and returns whatever T the downstreamʼs methods return.

Like recursive transformations, Visitors can be chained in arbitrary ways using the downstream argument: this makes it just as easy to compose whatever computation you want out of smaller, independent parts.

Unlike chaining recursive transformations like redact/toInt/summation, chaining Visitors does not produce any intermediate trees: you simply feed in the original tree, and in one pass it computes the redaction/toInt/summation and produces a result. You do not need to manually fuse the computations into a single function if you want it to be efficient!

Streaming Sources

Above, we have already seen how Visitors are much more verbose than manually writing recursive transformation functions, but with an upside: you can chain Visitors together to combine their transformations without needing to construct intermediate data structures, and without needing to manually write code to fuse the transformations you want into a single, big function.

However, one assumption we have made so far is that we are starting from the already-parsed, structured data in-memory: the Json classes defined above. This is often not the case in reality! In reality, often you are starting from some kind of serialized data format: text files, binary data coming over a network, etc.. For now, letʼs just consider the textual form of our Json format:

It might seem obvious that a Parser parses an input string into some kind of tree structure for further processing, but there is another alternative: the Parser could instead take a Visitor to dispatch to! This looks like the following:

Effectively, rather than having a dispatch function recurse over a structured Json tree and call the Visitorʼs methods, we have a dispatchParse function than parses over the un-structured Json text and dispatches calls to the Visitors methods. The Visitors themselves do not care who is calling their methods as long as they are called in the same order, so they should behave the same - and produce the same result - either way.

Whatʼs happening here is worth calling out explicitly: we are processing the Json data without ever parsing it into a full tree structure!

* In the first case, we are constructing the minified output string while the input string is still being parsed.
* In the second case, we are summing up the numbers in the Json during the parse.
* In the third, we are feeding the Json through the redactor, then summing up the remaining numbers

All three computations above happen without constructing any Json tree structure: by paying the verbosity cost of using the Visitor pattern instead of writing recursive computations on the Json tree, in exchange we get the ability to run our computations on raw input data without needing to parse/store the entire data structure in memory.

Tree Construction Visitors

ToIntVisitor and RedactVisitor above can be chained into downstream visitors, to combine their computations but without constructing intermediate data structures. Sometimes this is what you want, but sometimes it isnʼt: sometimes you actually do want to spit out a Json structure. Apart from maintaining two versions of ToIntVisitor and RedactVisitor - one chainable and one not - what can we do?

It turns out the solution is simple: define a Visitor that does nothing but constructs the Json tree structure! It looks something like this:

Previously, defining Visitor-based transformations rather than a recursive-transformation-function based API was a tradeoff: you could chain them to other Visitors for zero-overhead streaming processing, but you lost the ability to construct concrete structures in the case that is what you wanted. With ConstructionVisitor, the tradeoff is gone: your Visitor-based transforms can now be trivially converted into tree-building transformations simply by chaining the together with ConstructionVisitor!

Composability

So far we have defined two dispatchers:

* dispatchParse("...", _)
* dispatch(tree, _)

Two chainable visitors:

* RedactVisitor
* ToIntVisitor

And three non-chainable "terminal" Visitors:

* StringifyVisitor
* SummationVisitor
* ConstructionVisitor

What makes the Visitor Pattern so flexible is the way you can plug & play these simple components any way you like, performing operations on trees:

* dispatch(tree, new SummationVisitor) performs a summation over the tree

* dispatch(tree, new RedactVisitor(new StringifyVisitor)) serializes a tree to a minified string with redacted keys/values removed
* dispatch(tree, new ToIntVisitor(new SummationVisitor)) performs a summation over the tree after converting number-like strings to numbers
* dispatch(tree, new RedactVisitor(new ToIntVisitor(new SummationVisitor))) performs a summation over the tree after converting number-like strings to numbers and removing all redacted keys/values

Directly performing operations on raw text, without constructing a tree:

* dispatchParse(text, new RedactVisitor(new StringifyVisitor)) performs a streaming redaction directly on the raw input text
* dispatchParse(text, new ToIntVisitor(new SummationVisitor)) performs a streaming summation over the raw input text after converting number-like strings to numbers
* dispatchParse(text, new RedactVisitor(new ToIntVisitor(new SummationVisitor))) performs a summation over the raw input text after converting number-like strings to numbers and removing all redacted keys/values

We have paid a cost in complexity: rather than writing e.g. a single recursive def redact(input: Json): Json function, we had to write a dispatch functions and pair of RedactorVisitor classes.

In exchange we get the flexibility of chaining different recursive-transformation functions one after the other, but with zero-overhead as if we had manually merged those transformations into a single function. Furthermore, all our Visitor machinery can work regardless of dispatcher, so we can implement dispatchParse and immediately have the ability to do streaming computations directly on raw input text, without ever parsing it into a complete tree structure, with all of our existing Redact/ToInt/Stringify/`Summation operations immediately available for us to use for free!

Further work using the Visitor Pattern

There are further interesting things you can do with the Visitor pattern that I will touch on but not go into too deeply:

Validating that serialized data follows a particular format is a common task. Most people would use a parser to parse the input and report any errors, and simple throw away the data structure the parser generates: a workable but somewhat wasteful solution. Using the Visitor Pattern, we can simply combine our existing DispatchParser that we used for streaming computations with a NoOpVisitor and get a zero-overhead syntax validator entirely for free

Data Mapping

Another common thing to do is to parse JSON into instances of classes to use in your application. perhaps you want to convert:

Again with zero-overhead from intermediate trees. dispatchInstance itself can be implemented in a variety of different ways: using reflection, using type-classes, etc. but for now is left as an exercise to the reader.

True Streaming IO

Above, our dispatchParse function starts dispatching to the visitor from an in-memory String:

def dispatchParse[T](input: String, visitor: Visitor[T]): T

It is not difficult to make an equivalent dispatchParseStream that can work on JSON coming from arbitrary java.io.InputStreams:

This would allow you to perform streaming processing parses on Json data that may be too large to fit in memory: you could perform a summation, validation or directly off a file on disk without ever loading the whole file into memory:

// Validate JSON on disk
dispatchParseStream(new java.io.FileInputStream("big.json"), new NoOpVisitor) // Sum up numbers from JSON on disk
dispatchParseStream(new java.io.FileInputStream("big.json"), new SummationVisitor) // Redact JSON from disk to another file on disk
dispatchParseStream( new java.io.FileInputStream("big.json"), new RedactVisitor( new OutputStreamVisitor(new java.io.FileOutputStream("big-redacted.json")) )
)

All of these capabilities come entirely free: our previous RedactVisitor, SummationVisitor and NoOpVisitor are entirely unchanged, but thanks to the Visitor Pattern are now able to perform their computations in a 100% streaming fashion.

And of course, once youʼre working with InputStreams and OutputStreams we can now perform streaming JSON processing directly over the network either as well!

The exact implementations of dispatchParseStream and OutputStreamVisitor are left as an exercise for the reader.

Whatʼs the Visitor Pattern All About?

The Visitor Pattern gives you flexible, streaming, zero-overhead processing of complex data structures. While composable tree-transforming functions give you the flexibility but without the efficiency, and manually-fusing operations in one big function gives you the efficiency without the flexibility. The Visitor Pattern gives you the best of both worlds, while allowing your computations to happen in a streaming fashion where there isnʼt a concrete data-structure at all!

Libraries like the [3]ASM Bytecode Engineering Library and the [4]uPickle JSON serialization library make heavy use of the visitor pattern to implement their complex, performant bytecode and JSON transformations. While this blog post uses JSON-lite processing as an example, the same principles apply to any sort of processing of complex data structures

If you are just doing a simple computation using a concrete data structure, it does not make sense to use the Visitor Pattern: just define a recursive function using isinstanceof and do what you want directly.

Where the Visitor Pattern shines is when your computation is neither simple, nor on a concrete data structure:

* If you want to break up a complex transformation on a complex data structure into multiple smaller computations, but donʼt want to wastefully generate a bunch of intermediate data structures
* If you want to perform computations on data that doesnʼt have any concrete data structure at all: performing your computations in a streaming fashion, dispatched directly from the parser reading a file or over the network, and never having the entire input or output dataset in memory

If you have either of these requirements, the Visitor Pattern is for you.

About the Author: Haoyi is a software engineer, an early contributor to [5]Scala.js, and the author of many open-source Scala tools such as the [6]Ammonite REPL and [7]FastParse.

If youʼve enjoyed this blog, or enjoyed using Haoyiʼs other open source libraries, please chip in (or get your Company to chip in!) via [8]Patreon so he can continue his open-source work