Drupal Console

It is a suite of tools that you run on a command line interface (CLI) to generate boilerplate code and interact with a Drupal 8 installation.

How is it different from Drush? And are there overlapping features?

There are many similarities between these two tools, but the main difference is how it was built, using an object-oriented architecture and Symfony components.

Will we continue to use both? Or will Drupal Console replace Drush?

I think we will keep using both at least for now and maybe at some point we can merge both tools.

What are some of the things that you will keep using Drush for in Drupal 8?

Site alias (sync, backup), download modules, installing the site.

Are you planning to introduce those features into Drupal Console?

Yes we are actually working on site alias.

What are some things that Drupal Console can do that Drush can’t?

Who is the intended audience of Drupal Console?

You can use Drupal Console to help you developing faster and smarter with Drupal 8

Developers and SiteBuilders since you can Generate code and files required by a Drupal 8 module and Interacting with your Drupal installation (debugging services, routes) you can put your site in Maintenance mode or Switch system performance configuration.

But you can use this tool to Learn Drupal 8.

We are also working on a GUI so if you are afraid of CLIs we will provide a web site you will be able to use and select what you want to generate (module, controller, services, blocks, entities, etc…) and generate and download the generated code.

Would it be fair to say that right now, it’s most useful to developers who are actually writing code, while later, it will also be useful for sitebuilders who are used to using Drush to create site, install modules and themes, and perform routine tasks?

Use Cases

What are some things you can do with DrupalConsole?

Drupal Console help you developing faster and smarter with Drupal 8.

Generating the code and files required by a Drupal 8 module.

Interacting with your Drupal installation.

Learning Drupal 8.

Updates and Future

You were on the Drupalize.me podcast back in February talking about Drupal Console, what are some of the improvements that have been made since then? (Especially with Drupal 8 progressing the way it is.)

Update Automation

I’d like to talk a little bit about automation processes in general before we jump into Drop Guard, if that’s okay. What types of things are we talking about updating? Server configuration? Drupal projects? Deployment?

What are some of the technologies you were using before developing Drop Guard? Maybe the underlying pieces that make up the Drop Guard architecture.

Drop Guard

What is Drop Guard?

Simply put, Drop Guard is a service to automate Drupal updates with seamless integration into development and deployment processes. Drop Guard helps Drupal shops and other Drupal support and service providers to automate their update work. In case of critical security updates Drop Guard will update the site automatically within 1 hour. This makes the operation of a site more secure and reliable and makes Drupal updates a full part of the development process.

You said it’s “integration into development and deployment workflows.” What do you mean by that?

Drop Guard works simply as a dedicated team member that is responsible for applying updates in the development as well as in the maintenance and support life-cycle of a project. You can configure Drop Guard to work with any hosting provider and with any team workflow. Drop Guard can execute different Rules-Based commands to trigger deployment actions just as a real team member would do it on manual update work.

How granular can you get with updates? Security only? All updates?

How does Drop Guard actually work? Is there a module to install? Server setup?

What happens if a bug is introduced with an automatic update? Is there a process to notify the developer?

Who is Drop Guard designed to be used by?

Drop Guard is designed to help Drupal agencies and freelancers to deliver Drupal update services automatically. Every Drupal shop can use Drop Guard as a white label service to deliver update services to their clients as part of support contracts. For end users that don’t understand the processes behind deployment and developement deeply enough, the service is too complex but Drupal shops will definitely benefit from additional developer time that they can save for their project business.

What prompted you to start building the Drop Guard service? And when was that?

We started with the base technology in 2012 to build a system for our internal support contracts. We had the need to automate recurring things and ensure that our SLAs for security patches are processed reliably. When Drupalgeddon shocked the Drupal world and many sites had to be patched in a very short period of time, we already had the benefit of automated updates for our supported projects. At this point I realized that the system might have a benefit for other Drupal shops. So Drop Guard has its birthday with Drupalgeddon :-)

Do you have any insights of the roadmap of Drop Guard?

Sure! Currently we are in an internal Beta phase. That means we harden the service with some trusted users and we will add more beta users each week till the end of September.Then we will open Drop Guard for a public Beta version where everybody that is interested can start using the service with the help of our support team. I am sure that there are many usability issues we will face as the high flexibility results in a more complex configuration processes. But thanks to our current beta users we were able to address and fix many of them till now. Also the Feedback from Drupalcon Barcelona visitors will be an important milestone for us.

Does this work with all hosting providers? (VPS, Pantheon, Platform.sh, Acquia cloud etc.)

What does the pricing structure look like after the beta period?

You mentioned there’s an incentive for people to get involved with the beta now. Do you want to talk about that?

Automated Testing

Before we actually get started talking about Red Test, can we take a step back and talk about what automated testing is in general?

I am sure you are all aware of manual testing and QA. Once you build a site or a feature in the site, you go to the browser and test it out. As an example, you added a workflow to your blog post. You will go to the browser and test whether the blog you create gets the draft state. Then you will log in as an editor and you will move it to published state. Now you will log out and check whether the blog post is visible to anonymous users. What you are doing is manual testing. It works very well for smaller projects.

For bigger projects, and especially for projects with multiple developers, what starts happening is adding a feature may break an older feature. Let’s take the example above. Earlier all the blog posts used to be published by the writer directly as soon as he saved the form and an email used to go out to all your blog subscribers that a new blog has been created. Now after adding the functionality to save the blog post as a draft state, you want the email to go out only after the blog post has been published. Suppose the developer changed and tested the workflow functionality without changing the email sending rule and pushed the feature to live site. You will soon be flooded by complaints from your blog subscribers.

This is a very simple example but such problems increase exponentially when the software grows in size over time. Such problems can be prevented by having automation testing in place. For each feature in your software, you write test code and you run all these tests often, and especially after working on a feature. You push your new feature to production only after all the tests pass. This way, as long as you have ample test coverage, you minimize your chances of finding a bug on the production site.

Who generally does this testing, the developer? site-builder? site-owner?

There are different schools of thought here. In practice, I have seen two different approaches:

If you are following test-driven development, then the developer writes these automated tests. In fact, he writes the tests before he even writes the code for the feature and makes sure that the tests fail. Then he implements the feature and makes sure that all the tests pass. In our company, our developers write the tests but that’s usually done after he has implemented the feature. We are experimenting with test driven development as of now.

The other approach I have seen in having a separate QA team for testing. They write automated tests as well as do manual QA. The advantage of this approach is that you can hire QA person, who is usually much cheaper than a developer. The problem I have seen in this approach is miscommunication between the developer and tester. Because of the delay due to communication and also since tester writes the tests after developer has completed the task, as compared to in parallel, it takes more time for a feature to get out of the door.

What do these automated tests test for?

Ideally everything but practically that’s not possible. As everything in life, this follows pareto rule. 20% of tests will cover 80% of the functionality and you should definitely write these 20% tests. In addition to that, we follow a rule that no bug should appear on production twice. So if a bug is found on production, we write an automated test for it so that it doesn’t appear on production again. What these tests should cover is very dependent on your application. If you are building an information blog site with a beautiful theme and you are making changes mostly to the theme, then you should write regressions tests for your CSS and JS. On the other hand, if your site has a lot of custom business logic, access permissions and rules, then focus on testing the logic.

I always assumed tests were for functionality, can you give me an example of something you would test on the theme layer?

I have to admit that I haven’t ever written an automated test for any of the sites I’ve built. I did have an experience a little over a year ago where the videos on my site (that were supposed to be behind a pay-wall) weren’t because I made a change to panels that messed up the access control. I didn’t realize it for two months! So, if I had had tests in place to check the access to those videos, I would have been in better shape. So, even though my site is a small site in the grand scheme of things, even I can bennefit from writting appropriate tests.

RedTest

Red Test, as I understand it, is an open source integration testing framework for Drupal that you developed at Red Crackle. Can you tell us briefly what it does and why we should use it?

Red Test is an open-source integration testing framework for testing your Drupal applications. If you have multiple developers working on a big Drupal application stretching into months, you know that you need automated testing to keep adding more and more functionality without breaking old code. Red Test is ideal for testing:

Site structure and configuration

Custom business logic of your application

User access, roles and permissions

Rules

Drupal 7 supports Simpletest by default. Simpletest has unit testing and functional testing. Why do we need another automated testing solution? How is Red Test different from Simpletest and why should people use it?

You correctly mentioned that Drupal 7 supports Simpletest by default. In real life, when you are working on a big project, there are quite a few challenges when you test Drupal sites.

Drupal assumes that there is a persistent database connection available so any hook or function is free to access the database at will. This means that any function that accesses the database, unit testing is not possible. You can obviously refactor your code but that takes a long time. Also since Drupal stores all the configuration in the database, most of your custom code will need to access the database anyway.

For every test, simpletest creates a new test environment that has only minimal modules enabled. So if you want to test your current site with all the configuration, you have to first set all that configuration in code and then run the tests. That is too much of an overhead.

Functional testing in Simpletest is very slow because it uses a headless browser and every request needs to bootstrap Drupal. It’s not uncommon for a test suite on a large site to take multiple hours to finish.

Red Test alleviates these problems. It is an integration testing framework so it tests your site with the current configuration. It actually depends on the database so there is no need to refactor your code to make it work with Red Test. In fact, Red Test code is totally separate from Drupal code. We have created Red Test so that it runs in parallel on multiple processors of your machine and bootstraps Drupal only once at the start so it is 60 times faster than Simpletest.

A lot of Drupal developers have started using Behat which helps in testing your site functionally through a browser. With Behat gaining traction, is there still a need for Red Test?

Behat is an excellent tool and in fact, we use it in addition to Red Test. Since Red Test is an integration testing framework written in PHP and resides on the server, it can’t really test the javascript. So wherever JS functionality needs to be tested, Behat is the right tool. On the other hand, for testing all the other business logic in the application, we use Red Test. If you have used Behat on a big project, you will know that creating and especially maintaining Behat tests takes a long time. Every time an HTML id changes, Behat test needs to be changed.

Similarly, when you add a new required field in a form, Behat test needs to be updated to fill that field in the form. Red Test, on the other hand, knows about Drupal and its entities, nodes, people, roles and permissions. So it does a lot of tasks automatically in the background. As an example, if you added a new required field to a node form, Red Test will automatically fill suitable values in that field without you having to change anything in your tests. This makes it very easy for you to create and develop tests on Red Test.

In fact, we have done some measurement over the last couple of months and found that with the latest version of Red Test, creating and developing automated tests takes about 12% of total development time.

Is it easy to get started with writing automated tests using Red Test?

Yes, we are using all the standard PHP tools so it’s pretty easy for a developer to get started. Red Test uses Composer for installation and is based on PHPUnit. In fact, we measured how much time it takes a newbie to create the first integration test using Red Test and it comes to be a little less than 15 minutes. Below is the blog post you can follow to get started: http://redcrackle.com/red-test/documentation/first-integration-test

Use Cases

What are some concrete examples of what you should test?

Red Test is ideal for testing:

Site structure and configuration

Custom business logic of your application

User access, roles and permissions

Rules

If you are building just a basic informational site on Drupal without much functionality, you may not want to use Red Test. Use it only if you are building a large Drupal application.

What’s in the future for Red Test? Do you have improvement plans, or plans to integrate with D8?

Currently a developer needs to create tests using Red Test. We are considering whether enhancing it so that it works using Gherkin language makes sense. The next obvious step is to migrate it to Drupal 8.

So, Doug Vann emailed the two of us a while ago and said that I should have you on because

"I'm on the record as a huge fan of Tom's for his well educated and well rounded perspective on proprietary and Open Source software solutions. ... I'd be excited to hear Tom interviewed on the topic of how Drupal 8 will continue to erode into the proprietary market."

I thought that sounded good, so here we are!

Web Content Management

When you replied, you said that Drupal 8 is the most important product release in the history of the WCM market. Can you start out by explaining what WCM stands for and what qualifies software as a WCM product?

You also mentioned that the 2nd most important release was Day Software’s CQ5. What is that?

When I hear about Drupal’s competitors, I generally hear about Wordpress and Joomla. Why aren’t either of those number two?

Drupal’s Place in the WCM Market

How has Drupal faired in the WCM market so far?

What do you see Drupal 8 bringing to the table that sets it apart from other products?

Questions from Twitter

Jacob Redding

How does Drupal 8 change the comparison with AEM? Specifically what are the features with Drupal 8 that bring Drupal to a more level playing field with AEM? Is there a single specific feature that Drupal does hands down better than AEM?

Doug Vann

MY MOST IMPORTANT QUESTION

Drupal promotes an "Ownership Society" where Universities, Media Companies, Governments, etc. hire in ​Drupal talent and build sites inhouse. How does D8 impact that trend? Is D8 more for shops and agencies and less for DIYers or is that just F.U.D. talking?​

Any Drupaler would state that Drupal has been "disruptive" insofar as we have allowed highly visible sites to ditch their proprietary CMS in favour of Drupal.

To date, has that success been "truly disruptive" by your definition?

With the astounding advancements baked into D8, are you looking forward to an even more disruptive presence in the CMS playing field?

Shops

Is Drupal 8 ushering in a new era which will see a fundamental shift in how Drupal is delivered in the areas of customer procurement, engagement, and delivery?

To reword that. Are Adobe CQ5 and Sitecore shops operating significantly different than Drupal shops today AND are we going to see Drupal shops retooling and reshaping to a more enterprise looking organization?

In The past 18+ months, it seems that more people are willing to ​admit that Drupal 8 is moving Drupal "Up Market." Agencies are often the vendor of choice in those deep waters. Should we expect some more mergers and acquisitions which will ultimately empower agencies to deliver Drupal services inhouse?​

​The little guys

Where are the little guys in the D landscape?

Do you still see the $10K and the $45K range websites feeding the smaller end of the Drupal ecosystem?

Super exciting news this week! I've secured sponsors for the next two months to provide EVERY video on ModulesUnraveled.com absolutely FREE for you!

It's a bit of a long story, so if you don't care, just go learn some Drupal. But, I thought you might be interested in the back story, so here goes...

(P.S. If you're not a reader, I recorded this as an 8 minute podcast just for you.)

Why Modules Unraveled is now FREE for everyone

Late last year, there was a moment when I realized that it had been a long time since I had thought about why I started Modules Unraveled in the first place, and whether or not I was still on course with that original vision. I decided that I wasn't.

For reference, Modules Unraveled started with a video demonstrating how to setup Organic Groups in Drupal 7, back in 2011, when Amitai took over development, and did a complete re-write. Nothing was the same, and there were support requests all over the internet from people who didn't know how to use the new version.

I wanted to use it in a project I was starting at the time, so I went through the twisted, and difficult process of figuring out how the new module was supposed to be used. When I finally did, I recorded a video showing exactly what I had learned, so that others could skip the hours (and let's be honest, days!) that I had spent figuring it out, and just use the module the way it was intended.

Basically, I wanted to help everyone use Drupal to do amazing things.

At the time, I was teaching in the public schools full-time, but had the desire to do more web development and create videos like the one for Organic Groups. So, I started getting up at 3am... yes. 3:00 in the morning... and during that time I worked on videos until 6am when I would get ready to go to my full-time job.

Then, I started to charge for some videos. I figured, if I was saving people time, it was worth the $29 investment to learn how to use something like the Simplenews module, which has its own complexities. People apparently agreed with me, because the orders started coming in. Then I created more series' and put those up for sale, and everything was on an upward trajectory.

In the spring of 2012, I looked at what I was making between the hours of 3:00 and 6:00 in the morning, and extrapolated that to see what I could make full time. Based on that, I decided that it would make sense to quit my full-time teaching position, and focus on developing sites, and videos full time. (Too early as it turned out, but I'll explain that later.)

So, I was working for myself, from home, and everything seemed to be going great. The income was incredibly irregular though, so eventually, I switched the site from a pay-per-series model, to a subscription model. That was nice, because it provided a bit more predictable income, but though there were good months, on average, it still hadn't ramped up to what I was making as a full time teacher.

Because of this, I started to focus more and more on the money that I needed to make, instead of providing amazing quality videos that could help thousands of Drupal developers create their sites more quickly, and with less headache, like the original Organic Groups series did.

So, in December of of 2014, when I re-evaluated whether or not I was fulfilling my original mission, the answer was a resounding "No." I saw the thousands and thousands of site visitors in Google Analytics, and the meager handful that were actually signing up as a paying subscriber to access the videos. And the interesting thing is that it wasn't the lack of subscribers that hurt the most, it was that such a broad audience was coming to my site, because I had something they needed, but then immediately leaving because of the subscription requirement.

These were the exact people I had set out to help. And I was turning them away. I had reached a wide audience of people who didn't know what I was saying, because I was charging them to listen.

So, I wasn't making enough on the subscriptions alone to provide for my family, but I kept churning out new videos in the hope that eventually, the snow ball would roll in my direction, but it wasn't happening.

I eventually came to grips with that fact, and started trying to figure out other ways to produce an income, but still help the people that I had originally set out to help. I knew if I just got another job, I'd stop making videos. Because they take time. A LOT of time. You see this evidenced all over the web. YouTube is littered with Drupal tutorials teaching you how to do this or that, but there are rarely more than a handful done by any one individual. And, because video production, and teaching are not generally their areas of expertise, the quality is widely varying (to put it nicely.)

The obvious monetization strategy was advertising. I had never had third party ads on my site, and really didn't like the idea, but I had to try something new, because what I was doing wasn't working. (And one definition of insanity after all, is doing the same thing over and over, expecting a different result. I didn't want to be insane.)

So, I immediately started to contact businesses that I had some form of personal connection with. Some were ones that had products I used, some were ones where I knew someone(s) personally within the business, and others were ones that close friends of mine had personally recommended.

I wanted to see if anyone would be interested in advertising on my site. Some said yes, and some said no (one even suggested buying me outright, but after thorough discussion with my wife, and time in prayer, it didn't align with my life goals.)

The ones that said yes had a hard time nailing down a dollar amount (which is understandable, since I didn't have metrics to give them, having always been subscriber only), and for four or five months, I felt strung along, and didn't know if it was ever going to work out.

Then, I went to DrupalCon in LA. I wasn't planning to go, but then I won a conference ticket from the amazing people at Four Kitchens (thank you again!) and then just had to figure out a way to pay my room and flight. The only way I could convince my wife (who would be left alone with a two year old, and five month old) to let me go, was to convince her that I resolved to make something happen while I was there and that it would be a good investment.

I followed through. To give you an idea, I went to one (1) session the entire week, and spent the rest of my time in the expo hall, and hanging out with people who might be interested in becoming a sponsor (as well as a few old friends, that I don't get to see except at DrupalCons).

This was MUCH more fruitful than my previous, online-only, attempts. The net result was that six sponsors agreed to advertise, and combined, they would replace the existing membership income, and a little more. It wasn't much more, but was enough to let me run a two month trial to see how it would go, since I have absolutely no idea if it will be beneficial to anyone. My hope though, is that it will be beneficial for everyone, and I can sign additional sponsors, and/or charge higher rates.

One of those sponsors let me know that, after reviewing their budget, they would not be able to advertise, but the other five are still on board. You can see them, and (please!) thank them on the "Sponsors" page.

So, here we are! The site is completely free for anyone that wants to learn what I teach, and I get to find out if advertising will work on Modules Unraveled.