The moment has finally come… Arachne, while still far from complete, is ready to put in the hands of developers for testing and feedback. I'm excited to see what people think of it, and gather some data about what's awesome and what's not.

First, however, I'm going to state a big list of caveats, so nobody gets their expectations raised too high just yet.

Although many of the major underlying pieces are working, the current application development experience isn't yet close to what it will be. There's a snowball effect; each new module that becomes available will diminish the friction of application development just that much more. For example, the data abstraction layer (codename Chimera) will be huge, and an enabler for a lot of other tools that can't be started until it is ready. But it isn't quite done for this alpha release.

Everything is subject to change! I've developed things in a way that makes sense to me, and suits my tastes and style, but Arachne ultimately isn't about me; it's about putting power in the hands of people who want a fast and efficient path towards building powerful webapps. So if my tastes differ from yours, if there's anything about Arachne that you find awkward or distasteful, please let me know.

Related to the above: this is an alpha version of the software. That means that things can and will change. Until we release the first "production ready" builds, I can make no guarantee, or even put any effort, towards making sure that there are no breaking changes. If you build on top of Arachne, right now, you will be responsible for keeping up to date as things change underneath you.

With that out of the way, if you're still interested, what's in the release?

Write up the "module" documentation, which will be the best source if information for the concepts the modules introduce, their configuration schema, and the DSL they expose.

Write up a rudimentary config browser, to demonstrate the power of of a data-driven application to introspect application config.

Ship the kickstarter rewards. I've got this all lined up and ready to go, but haven't pulled the trigger on moving the physical goods yet. My apologies. I had intended to do this around the same time as the alpha release, but will do it this week instead. At any rate, backers should count on getting their stuff sometime in February.

I hope to finish up everything in the list above in the next two weeks, and then move on to the more long term roadmap. That roadmap includes, in rough priority order:

Finish "Chimera", Arachnes data and persistence layer. This will unlock a whole set of additional modules to work on.

Write modules to automatically expose APIs that can read/store data in a database. This will really start to demonstrate Arachne's immediate value for web services.

Write some automated deployment tools, for "one click deploy" into cloud hosting providers.
p
It's hard to predict how many of these I will get done before my Kickstarter funding runs dry sometime in April, and I am no longer able to work on Arachne as much. But even when that happens I'll continue to work on it on an open-source basis (or figure out some other arrangment that will let me continue to make forward progress.)

At any rate, I hope you enjoy what we've produced so far, and don't forget to post your feedback. Thanks again for your support!

Arachne design and development is progressing since our last update. Since then, due to time constraints, we sent out a brief 'Interim Update' to the Arachne mailing list, which you can sign up for on our home page, arachne-framework.org.

On Dec 2, Luke gave a talk about Arachne at the Clojure eXchange 2016, which explains some more of the high-level reasoning about Arachne and why the status quo of composing small libraries isn't good enough.

The Steering Group met on Dec 16, when Luke shared that he is ready to announce Arachne's initial alpha release date of Jan 20, 2017.

Yay! But, What does this mean? It means that while not all of Arachne's fundamental work is done, the core work is. It's time to move this project into the public square, where you fine folks can start to play with a documented release of Arachne.

It's worth saying directly: Arachne will be alpha software. Production use would be unwise!

The purpose of this release is to open up the feedback process outside the bubble of the Steering Group and into the hands of the people that matter most – all of you!

This alpha release is primarily to start getting feedback on some of the core design elements, and to enable potential module authors to get started thinking about what and how their own plans might integrate with Arachne. Please be aware that it will not provide the "complete" Arachne experience… there are a lot of convenience modules and higher-level abstractions that have yet to be written. So if the alpha release of Arachne is missing a feature you were counting on, don't despair! It's almost certainly still in the queue for delivery. At the time we release the alpha, we will also publish a roadmap of what our current plans around next steps are.

The codebase will still be under a considerable amount of flux, so please check with us before submitting a large or far-reaching pull request. Eventually, once the core design is quite stable, we hope to reach a point where the majority of changes to Arachne are driven by the community itself.

Luke will be working on documentation between now and the release, as well as refining existing code.

As to design tweaks, Luke has been busy. Chimera has been simplified. Please see ADR-017 for more.

Also, Chimera operations are now extensible and overridable. Luke used core.match to take override expressions from the config and compile them into a fast dispatch mechanism, so that it's now possible to override Chimera's default behavior at a very, very granular level, based even on the content of message payloads. This feature is intended to provide extreme flexibility when porting data to or from a legacy database. Essentially, this provides a short-circuit to the default behavior of entity persistence.

Our next update will likely be in 3 weeks. We will resume our normal update schedule, an update every 2 weeks, after the Holidays.

Arachne design and development is progressing since our last update. Since then, due to time constraints, we sent out a brief 'Interim Update' to the Arachne mailing list, which you can sign up for on our home page, arachne-framework.org.

We met again this Monday, where Luke showed us working Chimera code, which is well underway. Many questions were asked, many potential use cases were discussed, and Luke's design satisfied the professional scrutiny of his peers.

Chimera will be ready for more serious prodding and poking by the Steering Group in approximately 3 weeks, due to the pending Holiday and Luke's travel to London to speak about Arachne at Clojure eXchange 2016 on Dec 2.

Arachne's public alpha release has a soft target of January 2017. This date assumes that no major redesign is necessary.

Regarding minor design tweaks, Luke is unifying the design of Arachne Modules and Applications which will eliminate many of their current differences and provide more consistency between them. In fact, an Arachne application will now also be an Arachne Module.

Nola Stowe @clojuregeek and I will be at Clojure/Conj in Austin with Arachne stickers for anyone who asks, so please say hello if you're attending.

Our next update will likely be in 3 weeks. We will resume our normal update schedule after the Holidays.

The Steering Group met on Oct. 14 and Luke presented his work on error reporting. Luke explained that his work on error reporting was motivated to smooth over several friction points that he was experiencing while developing Arachne. This work could potentially benefit the larger Clojure ecosystem and will likely be factored out of Arachne Core. Here's an example screenshot of Arachne's error message handling:

Arachne's Database Abstraction Layer (DAL) is taking form, conceptually. An ADR will be published soon with more details.

My progress on the Static Site Generator has been slow, because I'm learning lots of new things. Russ Olsen has kindly pitched in, not only in writing code, but mentoring me along the way.

A pre-alpha template now exists for both Arachne Modules and Arachne Apps. To clarify, Arachne Modules have modest requirements on the presence and location of certain files (see ADR-004), but Arachne Apps can be any arbitrary Clojure code that calls the appropriate Arachne libraries. Even so, it's still helpful to have a starting template for Arachne Apps. The Arachne App directory structure shouldn't be taken as a proposed convention, just one possible instance that happens to work as a starting point.

These materials are not in polished form, and Arachne docs have yet to be written.

The Steering Group will be meeting this Friday, so please look forward to another update in a week.

Hello Arachne Enthusiasts!

Arachne is taking form. What does its future hold? After sharing some news and updating you on our progress, this post peers into Arachne's future. Our last post was in early July, so Luke has been feeling torn between writing a community update and writing Arachne itself. That's why I'm writing you instead.

My name is Jay Martin. I'm a fledgling web programmer with a sweet tooth for Clojure and Datomic. I'll be helping Luke focus on building Arachne by serving as a community liaison.

So, what does a community liaison do? Mostly listen.

Arachne's community liaisons will listen to feedback from the Community and relay that feedback to Luke in digestible form, striving to honor the feedback's spirit and substance.

Also, we want to learn as much as we can about Luke's mental model for building software systems with Arachne. We intend to share that mental model with you, clearly and concisely.

Luke is passionate about fostering a culture of learning and knowledge sharing, as is the Steering Group.

You can expect updates on our progress about once every two weeks.

Also, keep an eye out for our first technical screencasts and Wiki articles covering Arachne fundamentals. We'll post links to new resources here on this site as they become available.

Many people have stepped forward to offer Luke their help. Luke is deeply inspired by this outpouring of support. The Steering Group sees a risk to the project's success by opening up the development process too early, while Luke is simply getting ideas out of his head and into an initial project structure. The opposite risk for Arachne is that of isolating the development process from the Community, alienating the very people who will help Arachne reach its full potential – you, and every single supporter, code contributor, Kickstarter, developer, designer, author, blogger, speaker and tire kicker out there.

We aim to strike a pragmatic balance between these competing risks and we'll rely on you to let us know when we're off track. If you have ideas about how to improve Arachne, please open a GitHub issue. If you have ideas about how to make our Community Experience the very best in this quadrant of the galaxy, please send me a tweet @webappzero.

We'll open up more lines of communication, as needed to best support Arachne's community with the care and attention it deserves. We'll include Community Developers and Designers in the creative process once Arachne's fundamental pieces are more concrete. The idea being that it's easier to talk en masse about improving tangible code than intangible concepts. Presently, that's the role of the Steering Group. Luke shares his ideas and progress and then listens to our feedback. Gradually, the Community as a whole will replace the Steering Group in that role.

Speaking of Arachne's core pieces, here's a list of things that Luke has gotten done, adding to his original work, since officially starting work on Arachne on July 18th:

Yes! We've got demos. They reveal the inner-workings of the low-level Arachne system. These are largely undocumented so only the adventurous should expect to receive a learning outcome by attempting their execution. Also, keep in mind that these primitives are very low level and much friendlier DSLs (Domain Specific Languages) can and will be built on top of what you'll see there now.

My first impression of Arachne, based on seeing the working demos, is that of a system which vanquishes mystery and embraces accountability. Arachne's validated, acyclic, runtime system graph ignited atop a queryable, validated, schema compliant config value feels like a new level of control and awareness when initializing a software system. And while Arachne is marketed as a web framework, at its core its a framework of frameworks, a novel way of organizing a software application with the potential to influence the way we write server, desktop and even mobile software.

This level of control comes with a price: learning a new way of doing things that is, as yet, unproven in the real world of shipping software. Existing Clojure Libraries and software will need to be wrapped in an interface to comply with the Arachne Module protocol in order to reap the benefits described in the ADR on abstract modules.

When I saw my first Arachne demo I felt a little overwhelmed by all the new words like 'ontology', 'config schema', 'component' and how they all fit together. At that time, I'd never even used a Stuart Sierra Component, so it was a lot to take in.

For some, this price will appear too steep. Fortunately for Arachne, our Industry is awash in change. Driving us toward this new future is the mere possibility that the inherent frustrations of building and managing software systems could be significantly mitigated. Many have already opted for a future that includes Arachne by voting for its existence with their time, talent and treasure. For that we cannot thank all of you enough!

Arachne's immediate future will see Luke writing the Database Abstraction Layer (DAL). The DAL will include the common set of database primitives and commands shared by today's prevalent database systems: entities, queries, etc. I'll be working on the Static Site Generator which will allow us to dogfood Arachne by using it to host this site in the near future.

Concluding with a bit of philosophical conjecture about Arachne's future:

I recently learned about solving problems using recursion from a web page by Carnegie Mellon University:

To solve a problem recursively means that you have to first redefine the problem in terms of a smaller subproblem of the same type as the original problem. […] This step is often called a recursive leap of faith. Before using a recursive call, you must be convinced that the recursive call will do what it is supposed to do. You do not need to think how recursive calls work, just assume that it returns the correct result.

Arachne isn't pulled from the ether. It benefits from a host of innovations, built by people that answered countless challenges and shared their life's work with the world. It has been refined by many generous and thought provoking conversations within the Clojure and programming Communities. Something else will come after Arachne and It will be better than It could have been because of what Arachne is and what it is not. Ultimately, Arachne's success will depend on our collective willingness to convince ourselves that the pleasure it promises will outstrip the pain it hides and move ourselves to that self-evident and demonstrable leap of faith we call – action!

Last Friday, the steering group held its first retrospective and risk
assessment. The goal of this meeting was to analyze what's gone right
and what's gone wrong so far throughout this lead-in period, and to
try and forecast the major risks to avoid as Luke starts full-time
development.

The discussion proceeded with each person identifying a potential
risk, and brief discussion of each item.

Near Term Risks

Luke is the only person full-time on Arachne: will the steering
group have time to be sufficiently prepared for meaningful input?

Is everyone clear on their roles in the project, and what's expected of them?

The only expectation from the steering group is that they spend
a couple hours every few weeks thinking about the project, and
identify any screw-ups or mis-steps they can see. Additional
levels of involvement are welcome but not necessary.

Not only do we need to make sure existing roles are clear, but we
need to make sure these stay clear as we ramp up community
involvement, and let community members participate meaningfully
without derailing forward progress or becoming chaotic.

We could be either too ambitious, or not ambitious enough. We need
to make sure we thread a fairly narrow corridor, and try to stay in
the sweet spot that produces something genuinely new and useful,
while not getting mired down in attempting to solve every problem in
the world.

We need to make sure that there is enough of a feedback loop to help
Luke find that sweet spot. That feedback loop should be more rapid
than just these bi-weekly sync-up meetings. What are some ways people
who are interested can get involved in a more active way?

There is a risk that we could go off on technical tangents instead
of solving the real/core problems that people have.

Having a real client from day 1 (Apex) is a mitigation here.

So far, we've been weighted a bit heavily towards up-front design
rather than iterating on an MVP.

That's true, we will try to move more towards an iterative model
once we're working full-time and the core modules are done. We
definitely want to prefer an iterative, agile model of
development.

Luke might try to take on too much instead of delegating or seeking help.

Steering group can try to help more, as they have time. Ideas
for things that they could do to help Luke focus on core tasks:
website/docs admin, shipping Kickstarter rewards. Anything else?

We need to be careful not to get lost aiming too high and trying too
much to "change the world" - we can do as much as we have time for,
but need to remember that we're actually on the hook to deliver a
web framework first.

We need to get the foundations in place so we and others can use Arachne as
a platform, from which to build the really cool/big stuff.

As a counterpoint, let's be careful to actually aim high enough to
do something meaningful; let's not be so focused on not flying into
the sun that we stay stuck in the mud.

What exactly are we afraid of being too ambitious about?

Aiming too far ahead and designing for too much generality could
slow development down. If we always choose the technical approach
that is "50% more work, but better for the future…" that could
put our schedule at risk.

We might try to be more than a web framework before we even
become a web framework. This increases risk in many ways, both
from a schedule point of view and running the risk that we're
not solving a real problem (or not solving it in the right
way). We need to focus on building and iterating on a web
framework first, and going from there.

Luke's output is about to increase 15x as he goes full-time. Are we
ready for this?

Questions will shrink in scope as time goes on; but these early
questions are vitally important.

Ever design decision will have an ADR for review.

Code reviews from steering group are desirable but optional.

Will soon be a public repo, can "crowdsource" code review.

Long Term Risks

Open source project, especially those that aspire to have community
involvement, risk an explosion of configurations, which can lead to
lots of complexity.

Backwards compatibility can be hard to maintain - how long do we
want to support a bad decision made early on?

We can either make the choice to preserve or break backwards
compatibility, but we should make these choices explicitly
rather than letting them linger ambiguously.

If we don't hit some critical mass for adoption, Luke will have to
write everything, which will limit the amount/quality/utility of
what Arachne (and it's modules) can offer.

There will be other challenges we can't foresee. We need to make
sure we deal with enough of the known problems that we have time for
unknown and unforeseen issues.

We risk having bad or out-of-date documentation. We need to remain
vigilant and set up processes such that the docs closely match the
code. It can be very frustrating to the community if the
documentation is inadequate or out-of-sync.

We need to make sure we stay organized, and will probably need a
more powerful backlog and issue-management system than Trello in the
long term.

Even if we technically succeed, if we haven't succeeded socially, we
will have failed. Arachne's community needs to be more than just an
arrogant group of white dudes looking down their technical noses: we
need to be welcoming, beginner-friendly and diverse.

I am pleased to announce that we finally have a concrete schedule for
starting real work on Arachne. By July 18 I will have wrapped up my
current consulting engagements and turned my full attention to
Arachne.

During this time, I will also be working closely with
Apex Data Solutions. Apex is a
healthcare technology company that has already invested substantially
in Arachne, via a generous donation to the Kickstarter that
significantly helped to ensure that the campaign would succeed.

The current plan is to consult with them for two days of every week,
specifically around Arachne and how it can further their technological
and business goals. During this time I can work either on Arachne's
core (as it is relevant to Apex's needs), Apex-specific modules and
functionality, or other software as the need arises.

The remaining three days will be dedicated exclusively to general
open-source Arachne work, building it out according to the vision
outlined in the Kickstarter pitch. The Kickstarter funds will only
be used to pay me on these days when I'm focused purely on
open-source. Task prioritization during these days will still be
governed by the steering group, to maintain focus on the core
general-purpose product.

By joining forces in this way, we get two significant benefits:

Arachne will have a real world, rigorous use case from inception,
giving an opportunity to "dogfood" the product in realtime.

Presuming this arrangement continues, this gives at least 10 months
of runway for working on Arachne, as opposed to the ~4 that the
Kickstarter funds alone would have provided.

I think this is a great mutually beneficial opportunity, and I look
forward to working with Apex as well as developing the core web
framework.

These are the notes for the second meeting of the Arachne steering
group, which occurred on Friday, May 6.

The attendees this week were:

Tim Ewald

Jay Martin

Russ Olsen

Stuart Sierra

Luke VanderHart

This was a smaller, shorter meeting than the last one. Because Luke's
efforts the last couple weeks were spent on public-facing and
Kickstarter-related materials, there was no new technical content to
discuss in depth. We did, however, cover a number of other topics.

Also of note, this was our first meeting to include Jay Martin, who
backed the Kickstarter campaign at the $10k level and will be joining
the steering group going forward. He has a background in IT and
finance, and the Arachne project resonated with him after getting back
into web development with Python and Clojure. He is enthusiastic about
the project and eager to be involved in development, educational
materials, and ensuring that the project remains easy and accessible
to beginners.

Topic: Engaging better with the Clojure community

Although public reception of the project has been largely positive,
there have also been some negative reactions. We spent some time
discussing the community dynamics and how we can ensure we remain
friendly, inclusive and supportive of the Clojure web-development
community in its entirety. Some suggestions were:

Stay positive

Focus on the novel features and vision of Arachne, not any perceived
deficiencies in alternatives.

Answer questions and stay engaged, but put the primary focus on
releasing software that will speak for itself. Show, don't tell.

Focus on users who already feel the need for something different.
Those who are happy with the status quo will remain happy; Arachne
should focus on helping those who want a more framework-heavy
approach and growing the webdev community, and is not in any way a
contradiction or threat to the existing ecosystem.

Topic: Engaging better with the non-Clojure web development community

So far, we haven't managed to engage at all with the non-Clojure web
development community. We discussed why this is; the consensus was
that there's no reason anyone should care, given that nothing really
exists yet. Once Arachne is released, we can reach out to grow our
user base by telling a compelling story and providing some demos of
Arachne's capabilities. We can't expect to grow outside the Clojure
community until we have this.

We also discussed how people evaluate frameworks based both on "hygiene
factors" (doing basic things in a easy and clean way), and "selling points" (unique capabilities that bring people to the framework.)

Some ideas:

The Rails-esque "write a blog in 15 minutes" is now table stakes for
a modern web framework; we need something comparable.

Create demos showcasing the "hygiene factors" - ease,

Create demos showcasing the "selling points." Some examples for Arachne could be:

Modularity: live-code swapping out major components of an app.

Introspectability: demo a dashboard exploring the config of an
application.

Demo a one-button deploy of the system.

Tell a good story about security, baked in from the ground up.

Topic: brainstorming asset pipelines

Asset pipelining is an extremely important and somewhat difficult
functionality for any web framework.

Some questions:

Do we build something from scratch in Clojure, or offload to an
existing Ruby or Node tool (the approach taken by Phoenix, which
offloads to Brunch)

Do all the tools need to run in the JVM or are we OK shelling out to
external processes? What are the implications of that for setup
complexity?

What are the actual "must have" tools? Can we get away with not
supporting tools that only have an implementation in one language?

What are existing Clojure tools in this space? (Boot tasks, twixt,
stefon, etc.)

What strategy should we use for cache busting?

For development workflows, should we rebuild assets using filesystem watchers, or build
on-demand when a request for an asset is received (or some hybrid
approach?)

It would be nice if the asset pipeline were complete and robust enough
that a project using only the asset pipeline component was a complete,
usable static site generator.

Ultimately we came out with lots more questions and few or no firm
answers, but it was good to have the discussion.

Topic: abstract data layer

At this point in the conversation, we were getting pressed for time,
so we had little opportunity to do more than bring up the topic for
future thought.

In general, the idea is to have a means of specifying persist-able
domain entities in a way that is agnostic to any particular
database. This means defining a common schema format and persistence
API that will work with JDBC, Datomic, various NoSQL databases, etc.

This has three primary use cases.

Modules sometimes need to persist data, but they should not be
strongly coupled to any particular choice of storage, for greater
modularity. Such modules need to be able to program exclusively
against an abstract data storage API.

Modules should be able to re-purpose an entity definition for uses
other than persistence. For example, a module might take an entity
definition schema and expose it as a RESTful resource, or generate a
form to edit it.

Users should not hesitate to exercise the full power of their chosen
database, rather than the greatest common denominator of database
features, which will be the most the abstract layer can do (by
necessity). However, the most common 70% of database interactions
should still be extremely easy via the abstract API.

Several points were raised:

To make this problem tractable, we need to be aggressive about
limiting the scope of the abstract data layer. It simply can't do
everything well and still be manageable.

We need to have a good story about schema generation, but also about
connecting to existing databases with pre-existing arbitrary
schemas.

We need to be able to handle sets of columns/attributes that are
constructed at query time as well as concrete tables; there is not a
necessarily a 1:1 mapping between your application's concept of an
"entity" and the data store's concept (e.g, a Datomic entity or a
SQL table)

At this point, we used up our allotted hour and adjourned until the
next meeting.

Arachne is a web framework for Clojure. Immediately, this provokes two
questions. How is it different from other Clojure web frameworks that
are out there? And why use a framework at all, instead of simply
composing libraries?

These are good questions, and the answers to them inform both the
rationale and the design of Arachne.

First, let's define a few terms, just to get on the same page. These
aren't necessarily the global definitions of these words, or even the
best definitions, but they're very useful for exploring the
similarities and differences between different tools in the web space.

Definition: Library

Libraries are code or programs written and maintained by someone else,
which you can call from your code.

This pattern should be familiar to all programmers – unless you're
writing for micro-controllers in raw assembly language, it's
impossible to build anything without calling into code that you didn't
write yourself.

Definition: Framework

Frameworks are code or programs written and maintained by someone
else, which call your code. The fancy name for this is
Inversion of Control.

Every programmer also uses frameworks pervasively, whether they
realize it or not. Modern operating systems are prototypical
frameworks; you write a program that follows some defined interaction
protocol, and the system initializes it and hands over some degree of
control.

Managed language runtimes (JVM, .NET, etc) are also inherently
frameworks. So are application servers, IoC containers, game engines,
any application that supports plugins, and much more.

Frameworks are almost always coupled with library-style execution
patterns. It's rare to have a framework that controls the execution of
a developer's code without also providing hooks and utilities that
developers can explicitly call themselves. Therefore, even though the
definition of a framework is "calls your code", in practice it's
usually a much more elaborate handing back and forth of control.

Definition: Template

There is one more concept that often comes into play: code generation,
usually from some sort of code template.

Templates are code written by someone else, and have some sort of a
one-time "generate" operation that inserts their code into your
project. There, it resides alongside and on an equal footing with your
hand-written source.

This often takes the form of the initial generation of an entire project (e.g. rails
new or lein new luminus), or only certain files (e.g. rails generate)

In addition to specific, deliberate code generation there are a wide
variety of other common activities that, loosely considered,
constitute templates. These include copying an existing project to
initialize a new one, copying snippets of code directly, or using the
code generation features of an IDE.

The Holy Grail

LEGO®, the ideal programming environment

All of these techniques have one thing in common: they are a tool for
sharing and reusing code, with the purpose of delivering
faster. They allow developers to stand upon eachother's shoulders and
move into new territory, rather than starting over every time.

Contrary to some popular perceptions, programming is an inherently
social activity. It is inevitable. The programmer who starts with the
work of another programmer will nearly always come out ahead of the
lone wolf. It's no accident that collaborative open source models have
become dominant in the industry. The only path to success lies in
understanding and building upon what has come before.

And so, at the most fundamental level, these concepts of "libraries",
"frameworks" and "templates" are not even technical constructs. They
are the social conventions and patterns about how developers
communicate, share, link and combine what they build.

Implications for Web Development

Web development has some specific characteristics which
govern what and how web developers share and reuse.

There are a relatively large number of distinct concerns that need to be addressed.

There is a great deal of functional overlap between projects.

There are strong time and budget constraints.

Quality, polish and visual appeal have a direct correlation with success.

Bugs have a direct financial cost.

Projects have a long lifespan, and are continuously modified and expanded.

These constraints motivate a high levels of reuse, overall. But the
shape that takes – the ratio of library to framework to
template-style reuse – is highly variable. Every solution uses all of
them, to some degree. But in most applications, there is a dominant
notion of reuse which shapes how the application is developed.

Each pattern has its own benefits and drawbacks. Partially, the
choices are aesthetic. Different developers prefer different
interaction styles. But there are also some real, objective tradeoffs
between the different approaches.

Templates

Ancient Mesopotamian Template, c. 2600 BC

The good thing about templates is that they emit specific code
really, really fast.

The bad thing about templates is that they emit specific code.

The degree to which templates are useful depends on how suitable the
generated code is for you. After it's generated, how much of it will
you have to change? Immediately? What about over the lifetime of the
project? If the code is exactly what you need, they can be a huge
speed boost. If they require extensive modification, or include
features that you don't want, they can be more trouble than they're
worth. If the time required to modify them to suit your needs exceeds
the effort to simply implement what you need directly (or using other
techniques) they have a negative value proposition and it's better
not to use them at all.

Templates are also interesting because although you don't write the
code initially, you are still fully responsible for maintaining
it. Once the generation process is complete, it's your code. And your
code is a liability. Every line of code adds to your project's
inertia, and every line contains potential bugs.

Offsetting this, somewhat, is the fact that templates are the only
means of reuse that are self-documenting, or even educational. You
have to read the docs and figure out how to call a library or
integrate with a framework. But after you generate a template, it's
all right there. You can see exactly what it's doing, and learn from
what are (hopefully) established, solid patterns. A template can
actually teach you how to use a library or framework, or at least get
you started.

Libraries

Libraries are probably the simplest and most general way to reuse
code. You can call them using the same programming constructs you use
to call your own code, but you're not responsible for writing or
maintaining it. To the degree that a library does what you want, it's
pure win. It moves your project forward with little cost to you.

This is why programming based on composing libraries is extremely
popular, particularly in the Clojure community. There is a strong meme
that composing many small libraries, not using a monolithic framework,
is the best way to write software in Clojure.

While it's true that the community has gotten a lot of mileage out of
purely library-based solutions, they are not without their downsides.

Their weakness is that they don't know about your code, and they
don't know about other libraries (except ones they depend upon.) They
are unrestricted and widely varied in how they are called, and what
data formats they require. Your program bears the responsibility for
calling each library in the manner in which it expects to be called.

An application and its libraries

In many situations this isn't a problem. Programming against a handful
of libraries is rarely cumbersome. But given the characteristics of
webapps, as described above – large, diverse, and highly dependent
on reuse – the mismatches start to multiply. The effect is magnified
when the libraries are structural, opinionated and impose an idiom
for working with your data (as many web libraries do.)

Quickly, glue code emerges; code that exists only to integrate with
libraries. The more you depend upon libraries to deliver your app's
functionality (as you should, in a webapp) the stronger the effect
becomes until the application is mostly just a broker between
different libraries.

This isn't terrible, of course. It's a much better outcome than not
having libraries available at all. The cost of writing some glue code
is a relatively small price to pay for not having to re-implement primary
functionality.

The worst effect is how it solidifies an application. It means you
can't swap out one part for another, without replacing all the
relevant glue code as well. It adds inertia that must be overcome for
any changes, refactorings and improvements.

And in some cases, even that isn't so bad. After all, how often does
any given application need to switch out the core libraries? For
something built in house, the answer is "not often."

But what's currently holding Clojure web development back is that many
of the common web frameworks and toolkits are not actually frameworks
(given the definition given above). They tend to be templates,
consisting of libraries and the associated glue code. And for these,
it is necessary to swap out libraries, unless the bundled ones
precisely meet your needs.

It's true: they can let you get started quite quickly. Just like
cloning an existing, working application can get you started
quickly. But once you have, modifying the emitted code to suit the
needs of your current task is still a lot of work. And, particularly
if you disagree with some of the technical approaches taken in the
core application, that might not be worth it. By only providing an
easy path for a particular stack of libraries and development styles,
these toolkits limit their applicability.

If one of these solutions meets your needs, wonderful! You should use
it, and be grateful and happy. But if it doesn't, you may find
yourself looking for something else.

Frameworks

So, how are frameworks any better? Are they any better? The answer
is no. Not inherently. Like the other approaches, they have tradeoffs.

Frameworks still have a lot of glue code; indeed, you could argue,
they are the glue code. The difference is that they are glue code
which is written and maintained by someone else.

In a very real sense, if you use an actual framework (using this
definition), you aren't building an application. The framework is
the application. All you are doing, as a developer, is writing plugins
or extensions for it which customize it for your use case.

As such, frameworks often have a tendency to become massive monoliths
that deliver everything and the kitchen sink. They are, of necessity,
extremely opinionated - the way the framework is put together is the
way that the framework is put together, and if you don't like it you
have no recourse except not to use it.

The benefit of frameworks is that, when done correctly, they lower the
amount of work that application developers are responsible for to the
absolute minimum. Developers aren't on the hook for very much at all,
not even interfacing with their chosen libraries. Their sole job is to
conform to the shape and requirements of the framework, and implement
their own unique functionality.

A simple framework

Of course, the cost of this is that if the framework is going to have
any flexibility to use different libraries or ways of solving
problems, enabling such support is exclusively the framework's
responsibility. The user does not have the flexibility to make their
own decisions about what to us or how to use it. Many frameworks don't
do a good job of providing enough options, or integration points, and
so end up as massive crystalline edifices that aren't very flexible.

Some frameworks do actually do a good job of providing integration
points and useful abstractions. They treat extensibility as a feature,
and are designed from the ground up

One good example of an extensible framework is Rails. Some might argue
that the mechanisms of extension it uses aren't the cleanest or most
evident, but it's clear that real-world developers can use Rails very
effectively for different kinds of web applications. There is a
selection of gems for nearly every occasion, and by and large they
don't take too much effort to get to work with eachother.

An even better example of an extensibility-oriented framework, albeit
one that it isn't so popular any more, is Spring. Spring had many,
many problems, but one thing it got mostly right is that everything
was programmed against interfaces, and you could explicitly control
what concrete classes you wanted to use at any point. Not only that,
but there was no top-level, global set of interfaces or extension
point. Each component defined the interfaces for the points where it
integrated with other components, meaning you could drastically change
functionality by swapping out objects at different levels. Because of
Java and Spring's other flaws, it would be a stretch to say that
building extensions or modifying behavior was easy, but it was
always possible, while still retaining most the benefits of having a
framework.

What about Arachne?

Given this landscape, where does Arachne fit? What patterns of
reuse does it leverage, in what proportion, and how does it emphasize
their strengths while mitigating their weaknesses?

Given the definitions above, existing Clojure web stacks tend to be
mostly libraries, held together by a template, with maybe a few
framework-style elements sprinkled in.

Arachne, on the other hand, is unabashedly a framework; it consists of
a prebuilt runtime that calls the code which you provide it. Of
course, it does exhibit properties of all three. It is an attempt to
find a point in the space of possibilities that maximizes the
desirable outcomes: development speed, ease, maintainability and
quality.

Templates in Arachne

The best way to describe Arachne's relationship to code generation is
to say that Arachne has templates, but it is not a template. It
uses templates to get started in seconds, and to provide a ready-made
examples of bare-bones applications, but not to deliver any of the
core functionality. Templates of Arachne projects are fairly small,
and will only contain code that ought to belong to (and be maintained
by) the user, not any general-purpose or glue code.

This also means that there are multiple valid shapes for Arachne
projects. Arachne is not a directory layout, or a particular namespace
structure, or anything that you will find in a template. Arachne will
ship with multiple options: one for a Leiningen project and one for
Boot, at least.

In the future, it's also likely that there will be templates for
different types of Arachne projects. For example, by using different
combinations of modules, Arachne can be a static site generator, a
simple RESTful web service, a form-driven CRUD app or a rich
ClojureScript & React single-page application. Ultimately, if we are
successful, there will be a template for each of these and more.

The current plan is for templates to be extremely low overhead, with
no specific code generation tool. Neither will we rely on a specific
external tool (such as Leiningen project templates.) Instead,
templates will be available as simple GIT repositories which you can
clone, and which include a script to rename themselves to match your
project.

Libraries in Arachne

Like all frameworks, Arachne does offer some libraries. It will
provide a variety of namespaces and functions that you can require and
invoke, as utilities and also to interact with the framework at
runtime.

However, given the plethora of quality libraries that are already
available for most tasks, Arachne will not focus on re-implementing
basic web functionality that already exists in library form. Web
development requires hundreds of distinct tasks ranging from crypto to
data persistence to string manipulation, and re-inventing even a
portion of those would be clearly out of scope given the resources
available.

Instead, it will focus on making it straightforward to incorporate
existing libraries into an Arachne projects. You can always call
libraries as libraries from code that you write, without any
permission or involvement from Arachne. And for libraries that benefit
from deeper integration, or help provide structure to your application
(such as servers, asset converters, etc), Arachne will make it easy to
write lightweight shims that turn them into Arachne modules, with full
awareness of and participation in the framework.

Frameworks in Arachne

As mentioned above, Arachne is a framework. But it is a framework that
takes modularity and reuse seriously in an attempt to not become
monolithic. In fact, the Arachne core isn't even webapp focused at
all; it's a bootstrapping system for a hierarchy of modules, which are
all replaceable. Everything that Arachne does is implemented in
modules, which in turn can depend upon higher-level, more abstract
modules. Reuse and flexibility is baked in from the very beginning.

Arachne, with embedded and shimmed libraries

This is so true that it's perfectly accurate to think of Arachne
itself as a framework for building frameworks. It doesn't have a
finite set of integration points – rather, each module can itself be
extended. Every time you write an extension for Arachne, you increase
the surface area for future extensions. And Arachne provides clean
patterns for doing so.

Even more important, integration points for individual modules don't
have to be created explicitly. Modules don't need to provide hooks or
anticipate the need for every type of future extension. Instead, they
explicitly define a data ontology for how they work themselves. This
is observable and ultimately editable by potential extenders. Used in
combination with more conventional methods of open dispatch in Clojure
(multimethods and protocols), this pattern is intended to provide an
unparalleled level of open extension, and will hopefully foster a
large ecosystem of third-party modules, all extending eachother
hierarchically.

Conclusion

If Arachne is successful, it will be because of this community of
extension. If it is going to meet the massively varied needs of
real-world users, it isn't enough to get started quickly. It isn't
enough to have the theoretical ability to add or swap features quickly.

It will need to have all those things, coupled with the availability
of real, useful extensions. If we can provide easy solutions to the
most common general types of problems, then we can spend that much
more time working on the features unique to our applications, and
take one more small step towards the elusive goal of true, optimal
code reuse.

This quick, informal presentation gives a basic overview of the
Arachne system at a technical level, and explains the broad strokes of
the module system, the configuration system, and how the runtime
launches.

These are the notes from the first meeting of the Arachne steering
group.

The attendees were:

David Nolen

Jamie Kite

Luke VanderHart

Mike Nygard

Nola Stowe

Russ Olsen

Stuart Sierra

Please note: the prose here is a digest of what was said tidied up for
third parties to read, not a literal transcript. Pleasantries and
chit-chat that would not be meaningful for publication has been
ommitted.

Update on Kickstarter

Luke: We're at approximately $9,500, which is good, although the
rate of new backers has slowed somewhat over the last couple
days. Several companies have expressed interest in a corporate
sponsorship, and I have plans to release more materials such as blog
posts and screencasts to drive continued interest.

Russ: The current plan is for the episode of the Cognicast featuring
Arachne to be released within a couple weeks.

Introductions

Luke - I'm a software developer, using Clojure since 2008. Arachne isn't my idea, by any means… we've needed something like this for a long time. But I'm the one who has decided to try to make it happen, now.

Russ - I'm the VP of consulting at Cognitect, a longtime Java and
Ruby programmer. I have wanted something like Arachne since the
second day I started working with Clojure.

Jamie - I'm an engineering manager at Github. I'm a Rubyist at
heart and have been using Rails for many years, with a brief Clojure
interlude while working for Cognitect. I have deeply felt the need
for a sane way to build web applications in Clojure.

Nola - I am an independent consultant, with extensive experience
in Rails, and PHP before that.I worked with Java and GWT when I
worked at Google, but definitely prefer Rails. Recently I've become
burned out on monolithic, huge apps that are hard to update and so
have gotten into Clojure and Clojure over the past few
years. However, it can be difficult and take a long time to get
started so I'm looking forward to Arachne.

Stuart - I work for Cognitect, and have been using Clojure
approximately forever. I don't like webapps, and am glad Luke is
doing Arachne so I don't have to write them any more.

Mike - I'm primarily a backend developer, and have worked in a
large variety of tech stacks over the years including Java, Rails
and Clojure. I'm hoping to use Arachne as a common architecture and
a common layout to bring some standardization to Clojure projects.

David - I'm an engineer at Cognitect, I've been doing Clojure as a
hobby and professionally by a long time. I'm a major contributor to
ClojureScript. Setting up new web stuff in Clojure is a massive
pain, and I'd like to see a reasonable path towards making it a bit
easier.

Project Goals

For Arachne itself, the consensus was that the goals stated on the
Kickstarter page reflects what we're looking for from the Project.

Steering Group Goals

Luke: I don't want to be a open source benevonent dictator. I'm
looking to this group to keep me accountable, and to help decide the
technical direction for the project. I have some strong opinions
about how things should work, but if a plurality of this group tells
me something is a good idea, or a terrible idea, I will adhere to
those guidlines. Essentially, this group of people are the project
stakeholders.

Jamie & Mike: What does that look like specifically?

Luke: There's no upper bound on how much you can contribute. At a
minimum, members of this group should attend meetings whenever they
can, and review published resources and documentation to audit for
adherence to the stated goals.

Michael: It would be good to document architectural decisions
explicitly.

Russ: I'm looking forward to providing an outside perspective,
reviewing the published materials and vetting them to make sure it's
actually easy to use (which is the hard part.)

Logistics

For communication on the the steering and core development groups, we
will use the following mediums:

ADRs checked into the source repository

Trello

Slack

A bi-weekly remote meeting, every other Friday

Tech so Far

Editor's note: We will be releasing a screencast going over these
technical concepts in the near future - these notes serve as a record
of the discussion, but are not intended to fully explain the
underlying tech. Please be patient and wait until that more complete
explanation is available before forming strong opinions.

Luke: It isn't really far enough that I can show working end to
end. A lot of the pieces are there, and there's a bunch of code
already, but it hasn't all been put together yet. Here's the overall
approach:

configuration

a Datomic-style database value

Works on Datomic Free & Datascript

built before the runtime starts

describes the application

everything that could possibly be data

configuration schema

ontology & meaning of application concepts

defined piecewise by modules included in your app

modules can depend on other modules, & have access to the schema
of their dependencies

Mike: Do you have examples of this schema?

Luke: Yes, I'll show some examples at the end of the meeting if we
have time.

Jamie: If the config is immutable, how do plugins contribute
additional entities (for example, routes) to the application?

Luke: Each module has an opportunity to query the config and add
additional data during its initialization.

startup sequence

build application schema from module schemas

loads programmer-supplied configuration

applies module config in dependency order

build runtimes

entity instantiation

dependency injection

start runtime

Lifecycle/start

user-supplied config

imperative DSL in config "scripts"

DSL is defined by modules

DSL consists of forms that (ultimately) bottoms out to the configuration data

Config value is a pure function of user config scripts

Loading pure data from EDN also possible

Luke: shows an example project directory layout

Luke: Note that the config scripts live in a different directory
from the application source code to emphasize the distinction
between configs and application source. Configs are DSL scripts that
generate configuration data, which is a completely separate process
from your running application (and could even run in a different JVM
at a different time and place.)

Jamie: In what order do user config scripts get loaded?

Luke: Probably the application will invoke one, and it will
explicitly load the others.

Mike: Is this the runtime view or the deployment view?

Luke: For deployment you could either run out of the project
directory, largely the same as development, or put all this stuff
into a JAR file.

Nola: What about database migrations? Where do they go?

Luke: I've put some thought into that, and I'd like to talk it
over with you all. I'll write up an ADR and distribute it prior to
our next meeting so we can talk it over.

Jamie: What about the names of these directories? Are they final? Maybe we should name the config directory "config" instead of "app."

Luke: The names are up for grabs. We can chose different names
that make more sense. I named it "app" instead of "config" because
these aren't static config EDN files, they're actually code
(although different from system code.)

Russ: Rails programmers are used to having executable, "code-like"
config, shouldn't be a problem to call it "config."

Code Dive

Editor's note: At this point, we looked over a bunch of code for the
concepts described above and talked about it for another hour. It was
mostly explanatory, and conversational enough that it's difficult to
write detailed notes.

The reliance of this design on Module authors doing good data
modeling and providing well-thought-through DSLs.

What do good DSLs look like?

How do you interact programmatically with the DSL? Answer, you
don't: Programmatic access should interact directly with the
underlying config database and bypass the DSL entirely.

We need to refactor the HTTP modules into two separate modules: one
that encodes HTTP concepts in an abstract way, and one that
implements those using Pedestal.

We will have to spend some time, with this group, deciding on the
concept of "replacability" of modules in general. There are some
general architectural patterns here that we should be aware of so we
can structure our modules appropriately (i.e, when are "abstract
modules" necessary to define a common data model, which is
implemented in a concrete way by other modules?)

Eventually, we ran over time, and one by one people dropped off as
they had to leave. We will resume the conversation next time by
vetting some ADRs I will hopefully have written by then.