In Drupal 7, site deployments could be rather difficult onambitioussites. Some database level elements were worth programming out in hook_updates (turning on modules, reverting views, etc) and some usually weren't (block placement, contrib module configuration). I remember days where a deployment involved following a three page long Google doc of clicks that had to be carefully replicated. Ugh.

A New Hope

So if you've taken the dive intoDrupal 8, you'll quickly discover one of it's most prominent features -Configuration Management. Drupal 8's ability to manage configuration with yml files is absolutely amazing! It's nearly akin to watchingStar Warsand thinking "Hey, I can do anything with a lightsaber! Fight bad guys, cut holes in doors, remove my hand cuffs. Sweet!"

The Empire Strikes Back

Here's the rub. Managing Drupal 8 configuration in complex real world apps is akin to building a real world laser sword after watching Star Wars only to promptly burn your face off and lose two limbs as soon as you try to fight with it. "Ambitious digital experiences" essentially equates to "arduous development concerns" and even config management can't save the day simply by existing. You must use it for good.You must unlearn what you have learned. Iblogged a biton this shortly after Drupal 8 released, but oh how much I learned since then!

We've been doing Drupal 8 pretty heavy for about a year and a half here at Ashday and had both the fortune and misfortune of needing to manage a more complex set up which quickly revealed our deficiencies in understanding how to properly manage config.

Here's the scenario: A client needs a site that will become the model for many sites, but they don't want them to be a single site with multiple domains and they also don't want it to be costly or complicated to keep them mostly similar from a functional perspective. Given that our preferred hosting solution isPantheon, this quickly turned into an obvious Upstream project. And that means figuring out a new way to manage D8 config other than just import/export of the whole site.

If you aren't familiar, a Pantheon Upstream works nearly identical to their core updates - you have a remote repository that, upon code getting pushed to it, notifies you through the dashboard of your updates where you can apply them in the same way you do Pantheon core updates. It's pretty slick because it provides an easy way to have a big shared chunk of code and apply updates to many sites with a few clicks (well, except when nearly every update is major and requires hands-on management - but I'm not bitter).

The Phantom Menace

Our first try at this was to give theFeaturesmodule a go, but at the time the interface was just too buggy to give us enough confidence to rely on it, it auto-selected what we didn't want and didn't select what we did, and it didn't support some key things we needed like permissions. As a result we decided to home brew our own solution. We knew these sites were going to have a lot of config in common, and a lot of config unique, and we needed to deploy to many of them all in different states without tragedy striking. So to accomplish this, we concocted the following procedure that we would run at deployment time, all from a single drush command.

Export the current site's live config (using drush) to the config sync folder

Copy all config files (with uuids removed) in our cross-site custom module over top of the config sync directory

Copy all config files (with uuids removed) in our site-specific custom module over top of the config sync directory

Import all config.

What this allowed us was the ability to allow each upstream site to stray a bit as they needed to, but we could be assured the config we cared about was prioritized in the proper stacking order. The approach ultimately wasn't that different than the goal of Features, but we were in control of the process, it was all live and it was relatively quick. And you know what? It worked! For a while...

And then it didn't. You see, the method we used caused Drupal to see every config file we were tracking (upwards of 300) as changed simply because of the missing uuid. So if only 8 config files changed in a deployment, Drupal was attempting to import hundreds of config files every time. This meant that it started to slow significantly over time as the site grew in complexity and eventually, we started having timeout issues and long deployments. We also started to run into issues when there was a significant core update (ie: 8.3) because so much config was being imported unnecessarily that wasn't compatible in that moment with the new code because db updates hadn't run yet. Not good. It was time for something else.

Return of the Jedi

The Jedi in question here is again the Features module. Or maybe it'sMike Potter. At least it's not me anyways. AtDrupalCon Baltimore, I was set on speaking with Mike about how we were handling config because I simply knew there was a better way. If you don't know, Mike is one of the founders of the Features module and ran a greatBOF on config managementin Baltimore.

So I found this delightful man and laid out what we were doing and he reacted exactly as I had hoped. He didn't say that what we were doing was terribly wrong, but it made him visibly uneasy. After a chat, I discovered that Features had come a long way since we initially tried to use it and we should really give it another shot. He also explained some of the configuration of Features to help me better understand how to use it.

So we returned to Features now and are much happier for it. The thing is though that I don't think I would have really known how to manage it if we hadn't taken the deep dive into config and figured out how it needed to work. It all helped us a lot to decide how to incorporate Features properly for this particular situation so that I actually feel good about relying on it again. And that's how most good Drupal development goes. You really should know how something works before simply relying on a contrib module or someone else's code to take care of everything because otherwise you won't really know how to deal with problems - heck, you might not even know you have a problem! I personally don't prefer spending weeks writing code and then depending at a critical moment on a mysterious piece to make it all successfully roll out to production.

So as it all played out, we now understand what Drupal puts in config, what we care about and don't, what belongs in the upstream vs our site-specific modules vs no where, etc. Here is our current process after this 6 month long journey.

Revert the global base feature

If needed, revert the site specific feature

Run our previous script outlined above, but now on only the 5 or 6 role config files so we handle the permissions in the same fashion

So there you have it! For how long-winded this turned out to be, I'm glossing over a lot of details that are pretty critical to understanding Drupal 8 configuration (ex: blocks are a mix of config and content), but I recommend you do the same thing we did and really get your hands dirty and understand what's going on so that you don't get bit at rollout. After all of this, we feel even moreso that Configuration Management is an astoundingly useful component of Drupal 8 and now we find ourselves a bit sad when we update our Drupal 7 sites (a version we absolutely loved!) where we don't have this amazing tool.

So good luck and don't hesitate to drop us a note if you have any questions or thoughts on this stuff. I'll probably change my mind on all of it anyways tomorrow. That's why this job is awesome.

P.S. I apologize that I didn't find room to incorporate Attack of the Clones, Revenge of the Sith, The Force Awakens or Rogue One, but the reality is that I just didn't have time to modify our whole approach to configuration in order to make this blog post more cohesive.

With over 200 new features, Drupal 8 is officially here! Drupal is one of the world’s favorite open source content management platform.. and it just got even better. Here are some of the ways that Drupal 8 will benefit various groups of people.

Developers:

Configuration management – In prior versions of Drupal, most of the configuration was stored in the database. The problem with this is that it is very difficult to keep track of versions of the configuration when it changes. The only way to get configuration out of the database was to use a combination of modules such as strongarm and features to export things from the database into code. This was often time-consuming and error prone. Now with Drupal 8, configuration management is built-in so that carrying over configuration from development to production is a breeze.

Web services – Drupal 8 can now be used as a data source to output content as structured data such as XML or JSON. This means that Drupal 8 can strictly be used as a back-end while the front-end could be developed completely separate with a framework such as AngularJS or Ember. In other words “Headless Drupal” capabilities are now built-in instead of requiring various addon modules and lots of custom development.

Content Editors:

Bundled WYSIWYG editor – Drupal 8 is the first version of Drupal to come with a bundled WYSIWYG editor. Previously it was possible to add one of many different editors into Drupal but the setup was often time consuming and confusing. Additionally there were so many choices that some users felt lost about which one to choose. Over time CKEditor has become the most popular WYSIWYG editor for Drupal and now it is included out of the box.

In place editing – In addition to having CKEditor bundled in with Drupal 8, the Spark initiative is taking WYSIWYG concept a step further with true in place editing. This would give editors the ability to change content, menus, etc. directly from the front-end view of the site without having to navigate to an admin page on the back end. More info about the Spark initiative can be found here: http://buytaert.net/spark-update-in-line-editing-in-drupal

End Users:

Mobile First – Previous versions of Drupal allowed developers to create responsive themes. However some modules were not 100% compatible with responsive layouts. Now with Drupal 8 all themes are mobile first which means that all community modules will be compatible with responsive layouts. Additionally the default Drupal admin theme will be mobile friendly which should improve the experience for editors who want to author content from mobile devices.

Accessibility and Languages – Drupal 8 now has extensive support for accessibility standards including the adoption of many WAI-ARIA practices. This will make content structures easier to understand for people with disabilities. In addition to the accessibility improvements Drupal 8 now has multi-lingual support included. Drupal 8 has the capability to reach more users than any previous version of Drupal.

One of the biggest improvements in Drupal 8 is the new configuration management system. Config is now decoupled from code and the database. Unlike Drupal 6 and 7, developers no longer have to rely on the features module for moving configuration around.

Most large Drupal sites, and some smaller ones, require per environment configuration. Prior to Drupal 8 this was usually achieved using a combination of hard coding config variables and features. Drupal 8 still allows users to put config variables in the settings.php file, but putting config in code feels like a backward step given D8 emphasis on separating concerns.

For example we may have a custom module which calls a RESTful API of a backend service. There are dev, stage and production endpoints that we need to configure. We also keep our config out of docroot and use drush to import the config at deployment time. We have the following structure in our git repo:

When a developer needs to export the config for the site they run drush config-export --destination=/path/to/project/config/base. This exports all of the configuration to the specified path. To override the API endpoint for the dev environment, the developer would make the config change and then export just that piece of configuration. That can be done by runing drush config-get mymodule.endpoint > /path/to/project/config/dev/mymodule.endpoint.yml.

Drupal 8 and drush don't allow you to import the 2 config sets at the same time, so we need to run 2 drush commands to import our config. drush config-import --source=/path/to/project/config/base && drush config-import --partial --source=/path/to/project/config/dev. The first command imports the base config and the second applies any per environment overrides. The --partial flag prevents drush deleting any missing config. In most cases this is ok, but watch out if you delete a view or block placement.

Best practices are still emerging for managing configuration in Drupal 8. While I have this method working, I'm sure others have different approaches. Please leave a comment if you have an alternative method.

Update 11-Mar-2016:Removed --partial from base import. This prevented old configuration being removed during updates.

Until we can override the enabled module/theme/library list dynamically in Drupal 8 via configuration, we can bundle up our environment-specific development modules and Strongarm variables in a feature and enable the feature on a per-environment basis in settings.php (or even better, local.settings.php) via a project called Environment Modules. For example, one could create a feature for the development environment, enable the devel module, and set environment-specific Strongarm variables such as those that leave Drupal core caching disabled or environment-specific settings for Domain Access domains. Additionally, whenever refreshing the development, integration or staging environment databases from the production database as part of a release cycle, doing a simple Features revert reenables the correct environment-specific modules and sets the right environment-specific variables.

As the Environment Modules project states, $conf[‘environment_modules’] should not be set on your production site in its settings.php, however the module itself can still remain enabled on production without a performance hit — it just doesn’t do anything in that particular environment.

He's working through some pretty big architecture options, and the community has been awesome about stepping up and supporting this initiative - so huge thanks to you all for keeping so many balls in the air!

Best thing for new people wanting to get involved is to check out the list of issues and pick up what you think you can do (see important places to help section) core mentoring office hours can help as well (Wed. noon-2pm EST)!

He's working through some pretty big architecture options, and the community has been awesome about stepping up and supporting this initiative - so huge thanks to you all for keeping so many balls in the air!

Best thing for new people wanting to get involved is to check out the list of issues and pick up what you think you can do (see important places to help section) core mentoring office hours can help as well (Wed. noon-2pm EST)!

I've been getting inquiries in IRC and in the issue queue about a module I blogged about a few days ago. The blog post I wrote may have seemed that the module we are working on is duplicating the features module and that we should instead work on the features module. I want to clarify our intentions.

The configuration module isn't a replacement for features. The vision is that they could work together. Features currently serves two purposes, 1) to group configuration together to satisfy a certain use-case, and 2) the actual export and management of configuration into a "feature" module. Features module is an awesome module when using it for what it was built for, creating KIT compliant features. The reality is most people that use features probably haven't even read the KIT specification.

Hypothetically, let's use the features concept with the concept of a bakery. Lets say a baker has a feature called "Birthday Cake". This "birthday cake" feature has several ingredients to make a birthday cake and one of those ingredients is called flour. The baker has another system that manages ingredients called configuration. Configuration manages the flour ingredient and makes sure it always stays the same type of flour.

Flour can go in a lot of recipes, not just in birthday cakes, so this ingredient shouldn't be owned by birthday cakes, flour should be free to be in any food item without creating a dependency of needing to bake a birthday cake too. If the birthday cake feature managed the flour ingredient, the baker could never bake cookies with his "Cookies" feature, without also baking a "Birthday Cake". If flour is managed by the "Birthday Cake" feature and the baker decides to never bake birthday cakes again for his bakery, flour no longer exists, because you only have flour if you bake birthday cakes.

If one of the ingredients that configuration is managing ever changes the meaning of flour to mean "spelt flour" rather than what it was, "white flour", the recipes (features) would need to be notified by the configuration system saying that flour is different now and your recipes are now different. This is how they work together.

Ingredients shouldn't care what recipes use it, and recipes shouldn't manage and own individual ingredients exclusively. The ingredients need to be managed by a different system. Ingredients are configurations and recipes are features. Recipes use ingredients, and features could use configuration.

This module simply takes the configuration part out of features, and provides a new workflow for managing configuration, more along the lines of what CMI is going to do in Drupal 8. The vision is the features module could evolve to use this module once this module matures over the coming months. When Drupal 8 comes out, features will need to evolve to not do configuration management any longer anyways, all this will be built into core. Drupal 8 will probably not be released for a while, if it follows the same development cycle as D7, hence the reason why this module was written, to incorporate Drupal 8 configuration management ideas in Drupal 7. Since Features, in a sense, already does configuration management, we used a lot of features module code to build this module.

As CMI progresses and more code is committed for Drupal 8, it's possible that this module may start looking like a back port of CMI and less like a duplication of features module.

Either way, features module will need to evolve eventually.

Features module is a great module when using it for what is was built for, creating KIT compliant features. It's once you start to depart from it's intended use, that you begin to find its limits.

At ActiveLAMP, we have always been a big proponent for putting all configuration for the sites we work on into features. Much like everyone else in the Drupal community that uses features module, we figure out what configurations belong together, and create a feature to group these configurations together. Do these configurations together satisfy a certain use-case? Sure they do, for the particular site that we created it for, but for the most part, the feature really isn’t reusable on other sites unless we build another site that has the same exact requirements that this feature contains. In reality, we don’t really create reusable features that we can then use on other projects, because the projects we work on are just too different to be able to do this.

The features paradigm works great when you’re working on very similar sites, or even a distribution like Open Atrium, but not so much when working on many different sites that have nothing to do with each other, and many different requirements. When you really get down to it, we really use features module to manage configuration for the specific site we’re working on so that we can simplify deployment to dev, staging and production; we don’t use features to create configurations that satisfy a certain use-case that’s usable on other sites. In fact, I believe many of us in the Drupal community have become so accustomed to using features for configuration management and deployment, we just glaze over why Features module was really created -- to create a collection of Drupal entities which taken together satisfy a certain use-case (excerpt from features module page)

Don’t get me wrong, I love the features module, but I have to admit that I’ve run into my share of issues using features module for configuration management and deployment. Fortunately others in the community have ran into these issues too and have released modules such as features override, features plumber, and Features Tools. Not to mention entire workflows have been created around how to use Features to manage configuration for deployment that don’t even come close to creating KIT compliant features. Features module is really being misused, it’s not being used to create features, it’s being used to manage configuration and deployment.

Several weeks ago, after having multiple conversations with Alan Doucette (dragonwize), of RiotGames, this paradigm shift hit home for me. I had been using features module for configuration management and deployment so long, that I didn’t even think twice that I wasn’t using it for its intended purpose. I also realized that I had a tool belt full of work arounds to make features module kind of work for configuration management and deployment. There are a number of issues that you can run into using Features module for configuration management and deployment. In future blog posts I’ll elaborate on what specific issues we have run into.

After my discussions with Alan I was tasked to create a module just like the features module, except without the features part of it. We still think the features idea is a great idea -- to have a group of configuration to satisfy a certain use-case -- but we don’t think features module should be the tool to export and manage configuration. Our vision is that features evolves into using this configuration module to group configurations into a feature, but not actually own the configuration in a "feature" module.

If you want to checkout the module, you can download it from the sandbox for now (http://drupal.org/sandbox/tomfriedhof/1412412). Once we get the namespace we’ve requested, we’ll promote it to a full project. Try the module out, file issues, and help out. Alan and I will be giving a BoF at SandCamp this Saturday, for those of you in town. Come join us, and hear about our motivations for building this module, and give us feedback.

For those who haven't noticed yet .. I`m into devops .. I`m also a little bit into Drupal, (blame my last name..) , so one of the frustrations I've been having with Drupal (an much other software) is the automation of deployment and upgrades of Drupal sites ...

So for the past couple of days I've been trying to catch up to the ongoing discussion regarding the results of the configuration mgmt sprint , I've been looking at it mainly from a systems point of view , being with the use of Puppet/ Chef or similar tools in mind .. I know I`m late to the discussion but hey , some people take holidays in this season :) So below you can read a bunch of my comments ... and thoughts on the topic ..

First of all , to me JSON looks like a valid option.
Initially there was the plan to wrap the JSON in a PHP header for "security" reasons, but that seems to be gone even while nobody mentioned the problems that would have been caused for external configuration management tools.
When thinking about external tools that should be capable of mangling the file plenty of them support JSON but won't be able to recognize a JSON file with a weird header ( thinking e.g about Augeas (augeas.net) , I`m not talking about IDE's , GUI's etc here, I`m talking about system level tools and libraries that are designed to mangle standard files. For Augeas we could create a separate lens to manage these files , but other tools might have bigger problems with the concept.

As catch suggest a clean .htaccess should be capable of preventing people to access the .json files There's other methods to figure out if files have been tampered with , not sure if this even fits within Drupal (I`m thinking about reusing existing CA setups rather than having yet another security setup to manage) ,

In general to me tools such as puppet should be capable of modifying config files , and then activating that config with no human interaction required , obviously drush is a good candidate here to trigger the system after the config files have been change, but unlike some people think having to browse to a web page to confirm the changes is not an acceptable solution. Just think about having to do this on multiple environments ... manual actions are error prone..

Apart from that I also think the storing of the certificates should not be part of the file. What about a meta file with the appropriate checksums ? (Also if I`m using Puppet or any other tool to manage my config files then the security , preventing to tamper these files, is already covered by the configuration management tools, I do understand that people want to build Drupal in the most secure way possible, but I don't think this belongs in any web application.

When I look at other similar discussions that wanted to provide a similar secure setup they ran into a lot of end user problems with these kind of setups, an alternative approach is to make this configurable and or plugable. The default approach should be to have it enable, but the more experienced users should have the opportunity to disable this, or replace it with another framework. Making it plugable upfront solves a lot of hassle later.

Someone in the discussion noted :
"One simple suggestion for enhancing security might be to make it possible to omit the secret key file and require the user to enter the key into the UI or drush in order to load configuration from disk."

Requiring the user to enter a key in the UI or drush would be counterproductive in the goal one wants to achieve, the last thing you want as a requirement is manual/human interaction when automating setups. therefore a feature like this should never be implemented

Luckily there seems to be new idea around that doesn't plan on using a raped json fileinstead of storing the config files in a standard place, we store them in a directory that is named using a hash of your site's private key, like sites/default/config_723fd490de3fb7203c3a408abee8c0bf3c2d302392. The files in this directory would still be protected via .htaccess/web.config, but if that protection failed then the files would still be essentially impossible to find. This means we could store pure, native .json files everywhere instead, to still bring the benefits of JSON (human editable, syntax checkable, interoperability with external configuration management tools, native + speedy encoding/decoding functions), without the confusing and controversial PHP wrapper.

Figuring out the directory name for the configs from a configuration mgmt tool then could be done by something similar to

cd sites/default/conf/$(ls sites/default/conf|head -1)

In general I think the proposed setup looks acceptable , it definitely goes in the right direction of providing systems people with a way to automate the deployment of Drupal sites and applications at scale.

I`ll be keeping a eye on both the direction they are heading into and the evolution of the code !

I have been doing a lot of work related to easing the process of building a site from scratch on an individual machine and thus dabbling with configuration management and related topics. Since configuration management is one of the Drupal 8 key initiatives I figured I would share some thoughts I had on the outer fringes of the topic with more to come in the future.

Installation

One area of Drupal that has always seemed a bit odd to me has been the installation process/system. The process can play a key part in configuration management across machines and made it impossibly to implement a basic environment system during installation without hacking core. To remedy this issue I believe the installation system can be much improved, simplified, and made much more consistent with the rest of Drupal which will in tern make it easy to implement my environment system in contrib or core.

The installation system in Drupal 7 was re-factored/rewritten quite a bit and has thus been quite improved, but I think the direction of the installation system could be changed to make it much better. Currently the installer attempts to fake systems in core, like the cache, and actually duplicates a lot of code found elsewhere for module management and what not. Why not simply package a minimal database dump, similar to how the update tests used a Drupal 6 dump written in DBTNG, that can be installed to create an extremely minimal Drupal installation. At that point the installer can act like any other module and provide forms to complete the process. All modules in addition to the required modules can be installed through the standard process invoked from the modules page, but done in an automated fashion through the installer.

The reasons why this approach is beneficial are: 1) install.inc and install.core.inc could be virtually removed, 2) profiles would no longer be "hackish" during the install phase (solve the current issues related to dependency resolution and what not being the same for modules and profiles), 3) hooks like hook_system_info_alter() would work properly for profiles and modules during early installation phases, and 4) remove race conditions in general caused by maintain two sets of the same code.

Environments

In addition, my environment.module would work without hacking core. This concept is nothing new in the development world, but is something I feel would be a great candidate for Drupal 8.

The above code allows for the environment to be configured in the settings.php file for a site.

<?php$conf['environment'] = 'development';?>

During installation a profile (or module) can perform different tasks or conditional code based on the environment. For example, generated users can have a simple password on a development machine and complex ones on production or staging machines.

<?phpfunction my_profile_install() { if (environment_get() == 'development') {// Do cool stuff that only devs get to see.}}?>

Another cool feature that I have a use-case for on a site I am working on and seems to generally be useful is to enable modules based on the environment. Instead of having to do that in hook_install() or related it makes sense to have a way to specify that in a .info file. The above code allows for the following.

name = My profile

description = ....

version = 0.1

core = 7.x

dependencies[] = block

dependencies[] = dblog

dependencies[development][] = views_ui

dependencies[development][] = fields_ui

dependencies[development][] = devel

dependencies[production][] = integration_with_third_party

dependencies[staging][] = integration_with_third_party

Having this type of functionality in core would hopefully encourage better development practices and seems like a great feature to have. I have a number of scripts in combination with drush make, and the above environment utility that allow me to build out a fully functional site on a new box with a single command. I plan to cleanup the scripts, document them, and provide them in a followup post. As always I would love to hear your thoughts on this subject.