Roadmap towards non-experimental macros

Monday 9 October 2017

Ólafur Páll Geirsson

BLOG

This week, the Scala Center joins the multi-year efforts of Eugene Burmako and
his collaborators to establish a non-experimental and portable macro system for
Scala.
With Eugene’s blessing, I will be taking the lead on behalf of the Scala Center
to develop this project in close collaboration with the Scala community, the
Dotty team at EPFL, Scala compiler team at Lightbend and the IntelliJ Scala
Plugin team at Jetbrains.
This initiative follows SCP-014, a proposal that was approved two weeks ago
with an overwhelming majority of the Scala Center Advisory Board.

v1: scala.reflect

Scala.reflect-based macros shipped with Scala 2.10 and have since then become
an integral part of the Scala 2.x ecosystem.
Well-known libraries like ScalaTest, Sbt, Spark, Circe, Slick, Shapeless,
Spire and others use scala.reflect macros to achieve previously unreachable
standards of expressiveness, type safety and performance.

Unfortunately, scala.reflect-based macros have also gained notoriety as an
arcane and brittle technology.
The scala.reflect API is based on Scala 2.x compiler internals, meaning that
scala.reflect in its current form cannot be supported on other compilers such
as Dotty or the IntelliJ Scala Plugin.
Even five years after their introduction, scala.reflect macros still can’t
expand in IntelliJ, leading to proliferation of spurious red squiggles -
sometimes in pretty simple code.

Quoting Eugene Burmako, the author of Scala macros, in SIP-16 on
self-cleaning macros

While trying to fix these problems via evolutionary changes to the current
macro system, we have realized that most of them are caused by the decision
to use scala.reflect as the underlying metaprogramming API. Modelled after
compiler internals, scala.reflect inherits many of its peculiar design
choices. Extensive use of desugarings and existence of multiple independent
program representations may have worked well for compiler development, but
they turned out to be inadequate for a public API.

v2: scala.meta

The Scalameta project was founded to become a better macro system for Scala,
with the vision to replace scala.reflect as the de-facto metaprogramming
toolkit for Scala.

With Scalameta, we managed to support a different flavor of macros: annotation macros.
These macros are used to implement public type providers, which makes it possible
to generate publicly visible classes and methods.
Note that annotation macros have never been officially supported in Scala 2.x,
and have always required an external compiler plugin.

Scalameta annotation macros are the first macros to be supported on multiple
compilers:
Scala 2.x,
IntelliJ Scala Plugin
and
Dotty.
The novelty with Scalameta macros was that they used a “converter-based” approach.
This approach involves converting compiler-specific ASTs into the Scalameta
AST, which is a large collection of “dumb” data containers that leak no
implementation details from the compiler.
The details of this approach are explained in more detail in SIP-29, a
proposal to use Scalameta as the foundation for building macros in Scala.

The Scala community is indeed creative and eager to explore new metaprogramming
facilities.

Scalameta annotation macros are far from perfect, they suffer from integration
problems with, e.g., Scaladoc, Scala IDE/presentation compiler, Scala REPL and
Scoverage.
Most importantly however, we discovered several drawbacks with the converter-based
approach while adding support for def macros with access to the semantic API.
The issues we encountered are documented in more detail in section 3 of the
technical report “Two Approaches to Portable Macros”.

Building a portable macro system with ASTs that don’t leak compiler
implementation details is possible.

Eagerly converting between the compiler and scalameta ASTs introduces
a lot of problems that can be avoided by implementing macros in terms of
extractor/constructor methods on abstract syntax.

With these lessons learned, we decided to retire our efforts to build a macro
system on top of Scalameta.
From now on, Scalameta’s primary focus is to support building developer tools
such as Scalafmt, Scalafix and Metadoc.

v3: scala.macros

In winter and spring 2017, Eugene Burmako at Twitter and Liu Fengyun at EPFL
worked on a new macro system to address the limitations of scala.reflect-based (v1)
and scala.meta-based (v2) macros.
This third iteration of Scala Macros builds on top of the strengths of the
scala-reflect API with the following distinction, it’s

smaller, the API exposes as little as possible from the compiler
while still being able to support most interesting macro applications.

more robust, common pitfalls in scala.reflect-based macros are guarded by
the type system with separation of typed and untyped trees.

The design of this approach is explained in more detail in
section 4 “The Syntax-Based approach” of the technical report
“Two approaches to portable macros”.
The latest implementation of this design is hosted at scalacenter/macros.
I want to thank Twitter for allowing Eugene Burmako to share this
implementation, that he created during work-hours, under a BSD 3 license.
This repository has a working prototype that we can base our future work on.

One pain point in macros v1 that macros v3 does not address is the separate
compilation restriction.
Macro definitions must still be compiled in a separate project from where they
are used.
Nevertheless, we believe that v3 represents a significant enough improvement to
forgive this restriction, which is admittedly an inconvenience for macro authors
but arguably not a blocker for adoption.

Next steps

As the history above shows, establishing a stable macro system for Scala is a
large undertaking.
It has taken many years to reach where we are today, involving a collaboration
between many different parties.
We still have a long way to go to reach the level of expressiveness, robustness
and simplicity that we seek in a stable macro system.

Here below is a rough estimated roadmap for macros v3

in Scala 2.12, we experiment with compiler plugins

in Dotty, Liu Fengyun at EPFL will work on adding support in the compiler

in IntelliJ, Mikhail Mutcianko from the Scala Plugin team at Jetbrains will
work on adding support to expand macros from the IntelliJ editor

in Scala 2.13, we continue to experiment via compiler plugins and compiler
feature flags in later minor releases

in Scala 2.14 macros v3 become no longer “experimental” and scala.reflect is
deprecated

Following the recommendation of the Scala Center Advisory Board, the work on
macros v3 will be an iterative processes between

implementing macro features that have been approved for inclusion into
macros v3

gathering feedback from the community on what macro features merit inclusion
in macros v3

As for the first part, we have immediately begin development to support a limited
subset of blackbox def macros.

they can query the compiler for semantic information such as types and symbols.

they faithfully respect their declared type signatures, making their
implementation irrelevant to understand their behavior. From the end user’s
perspective, they look and behave much like regular Scala methods excluding
the ability to emit compiler errors/warnings

These attributes of blackbox def macros enable them to mix naturally into
Scala codebases and make them prime for inclusion in the Scala Language
Specification.

SIP proposal

Alongside prototyping preliminary support for a limited set of blackbox def
macros, we will immediately begin preparing a SIP proposal to include macros
v3 into the Scala Language Specification.
We plan to address the valuable reviews made to SIP-29 on inline/meta in a
new proposal, so that SIP-29 can be rejected.
In addition, we will document how we aim to solve hygiene using an
innovation discovered by the collaboration of Liu Fengyun and Eugene Burmako.

Documentation

Currently, there is no official guide for exploring the wide landscape of
metaprogramming facilities that are available to Scala developers.
While discussing macros with members of the OSS community and industry,
I repeatedly receive questions such as:
should I use def macros, annotations, shapeless, a compiler plugin or scripted
code generation?
There is no silver bullet, each solution comes with a set of trade-offs.

We plan to document the trade-offs of these different metaprogramming
solutions.
Especially, we want to document which use-cases macros are suitable for, and
which use-cases they’re either overkill, or insufficient.
Macros should in general be used as a last resort.
We care deeply to know what you’d like to see covered in this documentation
effort.
Please share your thoughts.

Share your thoughts

The current set of approved features in macros v3 does not reach feature parity
with the scala.reflect-based macro system.
Some scala.reflect macros rely on advanced capabilities beyond what
blackbox macros support.

By not supporting these advanced features, we put ourselves a fragile situation
where we risk forcing Scala users to remain on old versions of the compiler.
Such a situation is undesirable for both Scala users and those who wish to
evolve the language.
We must debate together as a community whether these advanced features
merit inclusion in the language specification or if they can be replaced
with alternative metaprogramming techniques.

To initiate this debate, I have started two Scala Contributors threads:
one on whitebox def macros and another on annotation macros.
In my posts, I try to reflect on the pros and cons of each feature in
an unbiased manner.
I look forwards to hearing your thoughts.
In particular, we should try to explore

towards what end do you use these advanced features?

why are the macros enabled by these features important for you and your
users?

can you use alternative metaprogramming techniques such as
code generation scripts or compiler plugins to achieve the same
functionality? How would that refactoring impact your macro?

We plan to include the results of these discussions in the SIP proposal for
macros v3.
The end result, I hope, will be a simpler Scala language with yet very capable
metaprogramming facilities.

Acknowledgements

I want to express my great gratitude to Eugene Burmako and his relentless
efforts to make macros in Scala as capable and popular as they are today.
Eugene has been a steward of macros in Scala for over 6 years.
During both professional and personal time, he has generously worked on
exploring new metaprogramming paradigms, mentored dozens of people (including
myself!) and communicated his findings with the Scala community both online and
offline.
I am honored to take the lead from your solid guidance.
I look forward to continuing our fruitful collaboration and I hope we, together
as a community, can stand up to the challenge to complete this project to end.