Manifold: The reinvention of code generators

This is the first in a series of articles covering Manifold, a unique, open source technology you can use in any Java project to enable cutting edge language features such as type-safe metaprogramming, extension methods, templating, and structural typing. In this segment, Scott McKinney discusses Manifold’s reinvention of code generators, namely Type Manifolds.

☠ Death to Code Generators ☠

While Manifold provides a broad host of high-level features, its primary focus is to eliminate the gap separating source code from metadata. For decades code generators have served to bridge this gap, but not without sometimes debilitating drawbacks. If a project you develop involves one or more code generators, perhaps you know what I mean. Read on for a more productive alternative.

The Metadata Disconnect

Our modern lives are replete with structured information, or metadata. It is everywhere and it is produced by near everything with a power cord. As a consequence the software industry has become much less code-centric and much more information-centric. Despite this transformation the means by which our software consumes metadata has remained virtually unchanged for half a century. Whether it’s JSON, XSD/XML, WSDL, CSV, DDL, SQL, JavaScript, XLS, or any one of a multitude of other metadata sources, most modern languages, including Java, do very little to connect them with your code:

Of course the Java compiler has no idea what to do with abc.Person, so we resort to running a code generator in a separate build step to generate all our JSON classes beforehand so the compiler can readily use them. The effects of this build step on the development lifecycle range from mild irritation to utter devastation, depending on the rate of metadata change, number and size of metadata files, density of usage in project, number of developers, etc. The problems include:

stale generated classes

long build times

code bloat esp. with large metadata domain

changes to structured data don’t invalidate generated code

no support for incremental compilation, all or nothing

can’t navigate from code reference to corresponding element in the structured data

generated code is often cached and shared, which leads to stale cache issues

customers often need to change metadata, which requires access to code generators

The Manifold Framework

The Manifold framework represents a rethinking of code generators. It altogether avoids the many disadvantages often involved with them by directly integrating with the Java compiler via the javac plug-in mechanism.

Implementors of the Type Manifold API, called type manifolds, establish a type supplier relationship with the Java compiler — the Manifold framework hooks into the compiler so that as the compiler encounters type names the type manifolds contribute toward resolving them, generating code in memory as needed. As such your application code can reference metadata sources directly by name as Java types, effectively enabling the prior example to work:

Think of a type manifold as a new domain of types for the compiler to access. As such the Manifold framework serves as a gateway between javac and type manifolds, effectively expanding Java’s type system to include whole new domains of types. Any number of type manifolds can operate in concert; they can also cooperate so that the types contributed from one can feed into the next and so on, forming a type building pipeline.

As the diagram illustrates a type manifold contributes to the definition of types in its domain. For example, the JSON type manifold produces types defined in JSON files. A type manifold can contribute toward a type in different ways. Most often a type manifold registers as a primary contributor — it supplies the main body of the type. The JSON type manifold is a primary contributor because it supplies the full type definition according to a JSON Schema file or JSON sample file.

Alternatively, a type manifold can be a partial or supplementary contributor. The extension type manifold, for instance, is a supplementary contributor because it augments an existing type with methods, interfaces, and other features. Thus both the JSON and Extension type manifolds can contribute to the same type, where the JSON manifold supplies the main body of the type and the Extension type manifold contributes methods and other features provided by extension classes. (I’ll cover extensions in a later article.)

Altogether this strategy eliminates many problems plaguing conventional code generation and metadata access in general. In essence the Type Manifold API redefines what it means to be a code generator. Benefits include:

Further, the Type Manifold API unifies code generation architecture by providing much needed structure and consistency for developers writing code generators. It puts an end to “lone wolf” code generator projects only one developer fully understands. Moreover, you don’t have to invest in one-off IDE integration projects; the Manifold plugin for IntelliJ handles everything for you, from incremental compilation to usage searching to refactoring. Finally, even if you’ve already invested in an existing code generator, you can still recycle it as a wrapped type manifold — the wrapper can delegate source production to your existing framework. Learn more about implementing type manifolds here.

Synergy

Perhaps the most refreshing benefit from using Manifold is the synergy resulting from its presence. With Manifold, a developer can define and use metadata that best suits the needs of a project without having to worry about build implications or IDE integration; he can create a metadata file, use it directly as a type, modify it, and access the changes immediately in his code. No compilation necessary, no build steps to invoke. With comprehensive IDE support, he can readily navigate to and from metadata elements, find usages from metadata, refactor, etc. Finally metadata has first-class representation in the Java development lifecycle! View it in action.

Conclusion

As a long time Java developer I’ve personally worked on several projects involving heavy code generation. I’ve seen the sometimes devastating effects of its use: build times measured in hours at customer sites, dev lifecycle demoralization, code generator development and maintenance consuming precious time, etc. It’s about time for a better solution and I think Manifold makes good progress toward that goal. Type Manifolds offer an inviting and productive development experience. With no code generators to invoke and no separate build steps to integrate, metadata just works.

There’s much more to cover, I’ve only scratched the surface with the Type Manifold API. Future articles in this series will cover:

Scott McKinney is the founder and principle engineer at Manifold Systems. Previously, he was a staff engineer at Guidewire where he designed and created Gosu. He currently pounds code by the truckload while listening to way too much retro synthwave.