228: Drupal 8 Lightning Distro and Module Acceleration Program

John Kennedy and I spoke about two exciting Drupal 8 projects he's running at Acquia in 2016. He's the Program Manager of the $500,000 US dollars Acquia is investing to upgrade important modules from Drupal 7 to 8 as part of the Drupal 8 Module Acceleration Program. He's also the Product Manager of Acquia's Enterprise Authoring Drupal 8 distribution, Lightning.

"I think Drupal is great at a lot of things ... and if we want to create a world where regular people can create great experiences, not just developers, Drupal is a fantastic application for that." - John Kennedy

1st version of Drupal: A Drupal 4.7 site he immediately migrated to Plone :-(

How did you discover Drupal?

John Kennedy: Well, I was doing a little bit of work for a nonprofit organization called Vibewire maybe back in 2006 and they said to me, “We’ve got this website, it keeps going down and we really need to have you look at it for us.” I looked at it, it was on Drupal 4.7, loaded up to the brim with modules and I said “This is awful” and promptly migrated them to Plone.

jam: Thank you, have a nice day.

John Kennedy: Since then, I’ve had some better experiences with Drupal. I ended up running my own Drupal shop for a while and I then came out to the UK to actually start the UK operation of the Commerce Guys and I did that for a little while, and then Acquia brought me on to be head of solutions architecture for Europe and now they’ve brought me over here.

jam: Fill in the blank here between 2006 and 2016, from being a big Plone fan to actually sticking with Drupal all these years. What changed for you?

John Kennedy: I don’t know that I was a big Plone fan. I was a big open source fan. I’d been a systems administrator and I’d been using the range of tools on top of Linux for a long time. Plone at the time seemed more mature. I had some developers who I could use for Plone, but what happened was that I found a couple of projects that were really suitable for Drupal and I worked out how to use it, nontrivial at the time, at least ... still ... and then once I was dug in, I found it more and more useful and I really got in touch with the community. I started coming to DrupalCons. The first DrupalCon I came to was Chicago and I hadn’t missed any since until I had my son 17 months ago and then I’ve missed a couple.

The power of the Drupal Site-Builder

John Kennedy: Absolutely. Drupal creates this role that exists in other ecosystems, but it’s really clear in Drupal of site-builder, and it’s someone who can be, but is not necessarily a developer, and can be, but is not necessarily an author, and they actually create experiences by assembling modules, assembling functionality, and that could be layouts with Panels or it could be business logic with Rules or it could be a range of other functionality bringing it in through the module ecosystem. I think that role is incredibly powerful because it allows little organizations and large organizations to much better leverage their expertise to build great experiences, to build complicated functionality.

It also facilitates this amazing ecosystem of people who scratch their own itch, but also contribute to a wider modular functionality and it’s a lot more, I would say, sophisticated than the module ecosystems you see in things like with Ruby Gems or necessarily just the wider Composer or PHP ecosystem because it actually takes into account that there’s an end user that needs an administration interface and needs guidance on how to implement this. It’s not just a piece of code that you plug in. It’s also, by default, an administration interface and general principles that allow it to slot into Drupal really easily.

jam: I’m going to try and boil that down. What I tell people often in this context is Drupal has taken the incredible power and flexibility of a lot of great code built by a lot of great developers and made a fundamental design decision along the way. “We are going to put the power of that code in the hands of less technical end users and make it available to anyone who wants to build community, build a business and so on.” So the fundamental design decision is empower the user-interface-user and not just the developer, is that fair?

John Kennedy: Yes, I think so. I think when we think about WordPress, they’re definitely empowering a user. They’re empowering an author, but it’s really just an author. It’s someone who just wants to publish simple content or who wants to write words or add media. Drupal enables an expert user - it’s not necessarily a coder, but it’s someone who has a little bit more knowledge and then can create something really sophisticated.

What is Lightning for?

John Kennedy: So the way I like to talk about lightning is that it’s not an out-of-the-box distribution. It’s a framework. It’s a way to cut 20% off any large project that wants to achieve a great authoring experience. Our tagline for it is "enabling developers to create great authoring experiences and empowering editorial teams". It’s not meant to be a beautiful instant experience like Atrium. It’s really meant to be a set of principles, frameworks, code, best practice, documentation that developers can take to leverage their time, to not have to think about that core set of functionality around authoring which, as we defined it, is layout, preview, media integration and workflow. Within those categories, we enable use cases like putting groups of content through a work flow and being able to preview them.

So you can enable scenarios like the election night scenario or the Super Bowl scenario where I might have two or three groups of content that I write and I want to be able to preview, but only one of them is ever going to get published. That was really hard previously, but we brought together in Lightning a few different modules to enable that, bringing together work spaces and workbench moderation and a couple of other things to allow for that. You can do so much more than we’ve done, but by giving you that little piece, I think we’ve enabled some really interesting use cases.

So there are a few of those different things that we’d brought together. It’s really about the fact that we think those four functionalities should be tightly coupled for enterprise authoring, not all the time, but specifically if you’re a large organization with many authors and you’ve got a sophisticated authoring process, then those functionalities should be tightly coupled because they should integrate with each other for that use case.

jam: So it’s a time saver for developers and they also believe it’s an opinionated way of integrating functionality within Drupal, and I imagine - am I right in saying that if people apply it, it will also save a lot of time on maintenance and handing off projects between different dev shops because a set of universal and good choices has been made along the way?

John Kennedy: Absolutely. Our internal PS team now, when they launch a project, they use Lightning by default.

jam: Even if it’s not an authoring heavy site, is that what they’ll start with anyway?

John Kennedy: I would say that Acquia clients tend to have authoring needs, so I don’t think Lightning should be necessarily the 80% use case for all Drupal shops out there, but it is for Acquia because that’s the kind of client we have. I think if you’re thinking about enterprise authoring, not just to post a blog, in fact not just multiple content types and interesting views, I think I’m really talking about if you have multiple authors and work flows and you need to create lots of different layouts and landing pages and all these kinds of slightly more difficult use cases, then yes, you should be thinking about Lightning.

jam: This is fully open source. When can I get it and how can I make it better?

John Kennedy:Drupal.org. In fact, we are being as transparent as we can possibly be. We publish all of our release notes, but also our forward releases have their stories on Drupal.org, so you can go see what we’re targeting, if we’ve already covered that, the issues. If they’re closed, they’re going to have the little line through them like they do on Drupal.org and we’re publishing as much as we can. So really, people aren’t just helping us fix problems, but influencing our forward roadmap as well. We want to hear what people want.

jam: That’s great, okay. So you’re combining deep technical knowledge and intuition with actual reports from people who do this every day.

John Kennedy: That’s it.

The Lightning Top 3

jam: What would you say are the top three things about Lightning that makes it really ideal for your targets with its use cases?

John Kennedy: So if developers are talking to their managers about why they should use Lightning, the obvious ones are the feature sets. I talked about that before, work flow preview, layout, media, but actually what makes this brilliant for developers is a set of principles that we’ve come up with to build Lightning.

One of those is that you never have to undo anything. So a lot of distributions, you’ve got to go in and you’ve got to undo configuration or things that they’ve decided upon earlier and that makes it less useful for you. We’ve really taken an unopinionated approach so that you can take Lightning and build forward.

The next one is automated testing. We’ve built Behat tests into all of our major functionality and that means that as you build on top of Lightning, you can actually test whether you’re breaking any of our stuff which is really useful when you’re building an enterprise authoring system.

The final one is upgradability and this is controversial because a lot of distributions aim at this and don’t quite get there, but we think by keeping a small-ish core of Lightning, we can actually maintain an upgrade path going forward. That means that if you build on Lightning, you can actually get free features as we upgrade Lightning. Those won’t necessarily be turned on automatically, but as a developer, as you upgrade Lightning, you have the ability to incorporate more of this functionality into the experience that you’re giving your users.

jam: So the first part of those really react to problems that we’ve all faced in one situation or another on the web in Drupal consulting and this idea of the build-it-forward and upgradability built into the distro really hits - if you can nail that, that really hits some pain points that I’ve certainly experienced along the way. Cool, great. So build-it-forward sounds like a good way to sum that up.

John Kennedy: Yes. It’s hard. What we’re doing is hard and we recognize that and there’ll be a lot of time spent maintaining upgradability and those concepts, but we think they’re key to the success of the distribution, so we’re putting the effort in.

jam: Now, in real time it is early March 2016. We’ve had Drupal 8 for a few months now and Drupal 8.1 is coming up. Is the Lightning Drupal 8 version already out and ready?

John Kennedy: So we’re in beta. I think we’re in beta four right now and as I said, our professional services department is already using our beta for building real sites, but we’re not going to release our GA version of Lightning until the 31st of March [this has slipped a bit since we recorded this conversation]. There’re a couple more features we want to put in there, things we want to tighten up, and actually we’re now thinking about what that looks like in 8.1 and how we bring 8.1 into our platform as well.

The Drupal 8 Module Acceleration Program

jam: So I’ve been doing Drupal, I noticed, for 11 years and I’ve figured out recently that a huge percentage of our community has never experienced a new major version release. Drupal 7 was our major version for more than five years and while we did this enormous rebuild, re-architecture, that has produced something really wonderful - I think Drupal 8 is fantastic. I’m enjoying every moment I use it.

Those of us who’ve been in since 4.6 which is actually - so we’ve been doing Drupal about the same amount of time. We’ve seen a lot more releases. For example, the release of Drupal 6, when it came out, was completely unusable. Every Drupal 5 site worth talking about used panel views CCK, and at the time, the views maintainer did not want to port Views One to Drupal 6 and took another six or eight months to finish Views Two for Drupal 6, and Drupal 6 was kind of dead in the water for a long time.

Drupal 7 had really good uptake, it was a solid release, but still we had this situation where it took a year for the contributed module space to really, really catch up with Drupal 7. So I want to point out a couple things about Drupal 8 that make it a much more usable release than we’ve ever seen before. Drupal 8 was completely and thoroughly tested from the first cutting of the branch, every single patch that was applied, so it’s really functionally solid. You can do a lot more with Core. Views is in Core, multilingual is in Core, and it’s very compact and like a ton of boilerplate has been taken out, you can make your own admin interface because it’s all Views. I mean it’s really very powerful and flexible and you can do great sites already now

But there’s also a lot more economic value hanging on the Drupal ecosystem for a major release than has ever happened before, and a lot of us--Acquia, community, shops, clients all around the world--don’t want to risk Drupal 8 failing as a release. We want this uptake to happen a lot faster. So I think the Lightning distribution coming from Acquia and being used in customer projects already being released on March 31st [and/or soon thereafter!] sends a signal, “Hey, this is ready to use.” One of my projects right now is writing the Drupal 8 Module of the Week series which is also - first of all, I want to celebrate the people who’ve done this work to bring this modules, to create these modules, to make them available for Drupal 8, but we also want to tell people, “Hey, use this thing. It’s awesome and you can do all these different things with it.”

Your other big project, you’re leading the Module Acceleration Program, Acquia’s Module Acceleration Program for Drupal 8. You’ve written some blog posts about it. You want to talk about what that is and how it addresses what I’ve just been going on and on and on about?

John Kennedy: Sure thing. So we did think a lot about Drupal 7’s adoption path and I was around and I made mistakes in 6 and I made mistakes in 7 and some of those mistakes were taking on modules early when they were not ready for release and I think lots of people being burnt by that process means they wait a little while before taking up a new major version.

What we wanted to do was take the great work of the community and bring modules to production readiness, giving people confidence to go out and implement them and use the functionality they’d expect in Drupal 7 in Drupal 8. So Views is in Core, so that’s less of an issue, but there are a whole range of modules that people use all the time and we have lots of experience and lots of channels to hear what people need through our clients, through our partners, through all of the community members we have in Acquia and we really ran a process where we looked at what were the top 50 modules that were going to make the biggest impact on Drupal 8’s usability?

We built that list; that was in conjunction with Angie and a range of other people at Acquia and we looked at how do we do that? How do we bring them to production? What we found was that all we needed to do was talk to people who already wanted to do them and give a little bit of funding so that they could take the time to do it. So that meant going to some of our partners like Palantir and like Lullabot and other shops who really wanted to give the time, and they gave us a really low community rate to go and do what they already wanted to do. We wanted them done, they wanted them done ...

jam: So you’re threading the needle somewhere between what a developer costs in a commercial setting, building a client project and the pure volunteerism which, for better or worse, most of our community wants to contribute, most of our community wants to make a difference, but when you’ve got paid work and volunteer work, the volunteer work happens whenever it can, right?

John Kennedy: Yes.

jam: So threading this needle, how did you figure out what a community development rate was that’s good enough to people to focus on helping the whole community by upgrading these functionalities?

John Kennedy: There was already a rate out there that we kind of went by as a rough approach, but there’re also people wanting to give their time and just said “Look, I can do it and this is what I can do it for.” A lot of that time, that was even better. So we brought that group of people together and it was made up of maybe 11 externals who were working on contract for us and maybe another five within Acquia that were doing work just as part of their jobs, and then maybe another 5 to 10 from the community who had parallel projects that had needs for these modules and actually just needed to get these done.

What we did then was we helped coordinate all those people. So we started running ... our internal scrum is daily, but then we had an external scrum which was weekly. Then we broke out and had a work flow scrum so that those people who are really interested in that could gather around that. In fact, we kicked a lot of this off at the summit that we had at BADCamp last year where we brought together one subgroup of the wider group which was around authoring. I was really interested in how Lightning was going to get done and a lot of other people were interested in how other parts of authoring were going to get done.

So we all got together at BADCamp and came up with the needs for this authoring experience and that fed into a lot of what we built as well, but it was really a lot of community engagement, a lot of going out and talking to the maintainers of modules and working out whether they had the time to actually do the ports themselves, or whether they could share maintainership, or whether they could support an effort, whether they could do patch reviews and things like this for our contractors and our contributors.

I think that was a lot of the goal in the project. It was great to get great rates from people and great to work with lots of the community, but that coordination effort meant we just steamed ahead in terms of getting modules done. I don’t want to take credit for ... I don’t think even the program should take credit for doing whole module ports. We took a lot of code that existed in porting, but what we did was we got it production-ready. We got it out there, we got it to the beta, we got it so that people use it in projects.

jam: One of the things that I’ve been really proud of over the years being part of Acquia is where the rubber really, really hits the road. Acquia has gotten open source right and especially on the contribution side, the first thing Dries did was hire Gábor to get Drupal 6 out the door in the best shape possible. We worked with really early versions of Drupal 7 when I was in Engineering building Drupal Gardens and we made some big mistakes around module upgrades at the time and we’ve learned from that and have apologized, I believe, along the way, but when the rubber hits the road, Acquia has done an awful lot of things that have helped the entire project and our community. I’m really, really proud of that as a Drupalist.

In this case, as you’ve written in the blog, Acquia’s invested US $500,000.00 in these upgrades, and frankly the fact that we get production-ready modules and our community gets some more rent paid, I think that’s a great outcome for everybody. How do you think this investment is going to - what’s the return on this investment going to be for us and for Drupal?

John Kennedy: We talked before about how a complete module ecosystem accelerates Drupal. I think the big return that we’ll see is people coming into Drupal 8 earlier, so existing Drupalists, but also people who are evaluating Drupal, seeing the functionality ready and getting onto Drupal, and we’ll see that curve. That exponential growth that we’ve seen in 7, we’re going to see that earlier. That’s the big return.

I think what keeps us all up at night is let’s see the big acceleration of Drupal 8. I think I’ve seen some early stats and there’re some really good news about what Drupal 8 is doing in terms of if we look at the path of Drupal 7, it looks like we’re actually double the amount of sites at the same stage. So that’s pretty good, but what we saw over a long period of time was this about 76% year-on-year growth of Drupal 7 and that brought us from a community that was already significant to one that ran 3% of the web. It was huge and the 35,000-odd active contributors now, a lot of them came onboard in that cycle and we really want to see that happen again. We’re going to bring back the excitement to Drupal.

I think Drupal is great at a lot of things and it’s still really relevant in this day and age where we are looking at more and more sophisticated use cases for authoring and for web applications, and if we want to create a world where regular people can create great experiences, not just developers, Drupal is a fantastic application for that.

jam: Well, thank you for coordinating all of this. Thank you for taking the time to talk with me today. Now get back to work and make Drupal 8 production-ready.

John Kennedy and I spoke about two exciting Drupal 8 projects he's running at Acquia in 2016. He's the Program Manager of the $500,000 US dollars Acquia is investing upgrading important modules from Drupal 7 to 8 as part of the Drupal 8 Module Acceleration Program. He's also the Product Manager of Acquia's Enterprise Authoring Drupal 8 distribution, Lightning.