The PreviousNext team are sponsoring and helping to run the sprint day on Wednesday, December 5th 2018, and there are a few things you can do now to hit the ground running on the day.

What's a Sprint Day about anyway?

Contribution Sprints are a great opportunity to get involved in contributing to Drupal. Contributions don't have to be just about code. Issue triage, documentation, and manual testing are examples of non-code contributions.

If you are new to contributing, you can take a look at the New Contributor tasks on the Drupal.org Contributor Tasks page.

While there will be experienced contributors there on the day to help, keep in mind, this is not a training session. :-)

Set Up a Development Environment

There is more than one way to shear a sheep, and there is also more than one way to set up a local development environment for working on Drupal.

Find Issues to Work On

If you want to see what might be an interesting issue to work on, head over to the Drupal.org Issue Queue and look for issues tagged with 'DrupalSouth 2018'. These are issues that others have tagged.

You can also tag an issue yourself to be added to the list.

Being face-to-face with fellow contributors is a great opportunity to have discussions and put forward ideas. Don't feel like you need to come away from the day having completed lines and lines of code.

This has proved to be a useful tool for auditing our projects for unusual access scenarios as part of our standard go-live security checks or when opening sites up to additional mechanisms of content delivery, such as REST endpoints. Today this code has been released on Drupal.org: Entity Access Audit.

There are two primary interfaces for viewing access results, the overview screen and a detailed overview for each entity type. Here is a limited example of the whole-site overview showing a few of the entity types you might find in core or custom modules:

Here is a more detailed report for a single entity type:

The driving motivation behind these interfaces was being able to visually scan entity types and ensure that the access results align with our expectations. This has so far helped identify various bugs in custom and contributed code.

In order to conduct a thorough access test, the module uses a predefined set of dimensions and then uses a cartesian product of these dimensions to test every combination. The dimensions tested out of the box, where applicable to the given entity type are:

All bundles of an entity type.

If the current user is the entity owner or not.

The access operation: create, view, update, delete.

All the available roles.

It’s worth noting that these are only common factors used to determine access results, they are not comprehensive. If access was determined by other factors, there would be no visibility of this in the generated reports.

The module is certainly not a silver bullet for validating the security of Drupal 8 websites, but has proved to be a useful additional tool when conducting audits.

But if you still have those, you probably want to start there, because they won't be supported in Drupal 9.

Last but not the least if you think the is module is not ready yet to fix all the deprecation warning you can set suppress-deprecations: true.

As a contrib module maintainer or a contrib module consumer I encourage you to add this file to all the contrib modules you maintain or use, or at least create an issue in the module's issue queue so that at the time of Drupal 9 release all of your favourite modules will be ready. JSONAPI module added this file in https://www.drupal.org/node/2982964 which inspired me to add this to DER in https://www.drupal.org/node/3001640.

Comments

What Alex said. I did this for the JSON API module because we want it to land in Drupal core. But we are running into the problems explained in the issue linked by Alex.

Despite Drupal 9 being announced, Drupal Continuous Integration system is not yet ready for modules trying to keep current with all deprecations for Drupal 9, while remaining compatible with both simultaneously minors with security team coverage (current + previous, ATM 8.6 + 8.5) and the next minor (next, ATM 8.7). Hopefully we soon will be :)

Thanks for writing about this though, I do think it's important that more module maintainers get in this mindset!

DrupalCI always runs contrib tests against the latest core branch. As a contrib module maintainer if I have to make a compatibility change for core minor version then I create a new release and add that to releases notes after the stable release of core minor version e.g. 8.x-2.0-alpha8. I never have to create a new release for the core patch release at least till now but yes I don't know how would I name the new release if I ever have to do that but then again that's contrib semvar issue.

DrupalCI runs against whichever core branch the maintainer has configured it to run against.

If a contributed module wants to remove usages of deprecations, it should probably never do that against the "Development" branch, as there isnt a way for a contrib module to both remove those deprecations, *and* still be compatible with supported or security branches. The earliest that a contrib module should try to remove new deprecations is at the pre-release phase, as at that point we're unlikely to introduce new deprecations.

Pagination

Add new comment

Drupal 8.6 has shipped with the Media Library! It’s just one part of the latest round of improvements from the Media Initiative, but what a great improvement! Being brand new it’s still in the “experimental” module state but we’ve set it up on this website to test it out and are feeling pretty comfortable with its stability.

That said, I highly encourage you test it thoroughly on your own site before enabling any experimental module on a production site. Don’t just take my word for it :)

What it adds

The Media Library has two main parts to it...

Grid Listing

There’s the Grid Listing at /admin/content/media, which takes precedence over the usual table of media items (which is still available under the “Table” tab). The grid renders a new Media Library view mode showing the thumbnail and compact title, as well as the bulk edit checkbox.

Field Widget

Then there’s the field widget! The field widget can be set on the “Manage Form Display” page of any entity with a Media Reference Field. Once enabled, an editor can either browse existing media (by accessing the Grid Listing in a modal) or create a new media item (utilising the new Media Library form mode - which is easy to customise).

The widget is very similar to what the ‘Inline Entity Form’ module gave you, especially when paired with the Entity Browsers IEF submodule. But the final result is a much nicer display and in general feels like a nicer UX. Plus it’s in core so you don’t need to add extra modules!

The widget also supports bulk upload which is fantastic. It respects the Media Reference Fields cardinality, so limit it to one - and only file can be uploaded or selected from the browser. Allow more than one and upload or select up to that exact number. The field even tells you how many you can add and how many you have left. And yes, the field supports drag and drop :)

What is doesn’t add

WYSIWYG embedding

WYSIWYG embed support is now being worked on for a future release of Drupal 8 core, you can follow this Meta issue to keep track of the progress. It sounds like some version of Entity Embed (possibly limited to Media) will make it’s way in and some form of CKEditor plugin or button will be available to achieve something similar to what the Media Entity Browser, Entity Browser, Entity Embed and Embed module set provides currently.

Until then though, we’ve been working on integrating the Media Libraries Grid Listing into a submodule of Media Entity Browser to provide editors with the UX improvements that came with Media Library but keeping the same WYSIWYG embed process (and the contrib modules behind it) they’re currently used to (assuming they’re already using Media Entity Browser, of course). More on this submodule below.

This is essentially a temporary solution until the Media Initiative team and those who help out on their issue queue (all the way from UX through to dev) have the time and mental space to get it into core. It should hopefully have all same the bulk upload features the field widget has, it might even be able to support bulk embedding too!

View mode or image style selectors for editors

Site builders can set the view mode of the rendered media entity from the manage display page, which in turn allows you to set an image style for that view mode, but editors can’t change this per image (without needing multiple different Media reference fields).

There is work on supporting this idea for images uploaded via CKEditor directly, which has nothing to do with Media, but I think it would be a nice feature for Media embedding via WYSIWYG as well. Potentially also for Media Reference Fields. But by no means a deal breaker.

Advanced cropping

From what I can gather there are no plans to add any more advanced cropping capabilities into core. This is probably a good thing since cropping requirements can differ greatly and we don’t want core to get too big. So contrib will still be your goto for this. Image Widget Crop is my favourite for this, but there’s also the simpler Focal Point.

You can test out the submodule from the patch on this issue and let us know what you think! Once the patch is added, enable the submodule then edit your existing Entity Browsers and swap the View widget over to the “Media Entity Browser (Media Library)” view.

It shouldn’t matter if you’ve customised your entity browser. If you’ve added something like Dropzone for drag-and-drop support it *should* still work (if not, check the Dropzone or Entity Browser issue queues). If you’ve customised the view it uses however, you might need to redo those customisations on the new view.

I also like updating the Form Mode of the Entity Browsers IEF widget to use the new Media Library form display, which I always pair back to just the essential fields (who really needs to manually set the author and created time of uploaded media?).

You still can’t embed more than one media item at a time. But at least now you also can’t select more than one item when browsing so that’s definitely an improvement.

Plus editors will experience a fairly consistent UX between browsing and uploading media on fields as they do via the WYSIWYG.

Once setup and tested (ensuring you’ve updated any Media Reference Fields to use the new Media Library widget too) you can safely disable the base Media Entity Browser module and delete any unused configuration - it should just be the old “Media Entity Browser” view.

Please post any feedback on the issue itself so we can make sure it’s at its best before rolling another release of the module.

Happy days!

I hope you have as much fun setting up the Media Library as I did. If you want to contribute to the Media Initiative I’m sure they’ll be more than happy for the help! They’ve done a fantastic job so far but there’s still plenty left to do.

Adding classes onto a field element (for example a link or image tag - as opposed to the wrapper div) isn't always the easiest thing to do in Drupal. It requires preprocessing into the elements render array, using special Url::setOptions functions, or drilling down a combinations of objects and arrays in your Twig template.

The element_class_formatter module aims to make that process easier. At PreviousNext we love field formatters! We write custom ones where needed, and have been re-using a few generic ones for quite a while now. This module extends our generic ones into a complete set, to allow for full flexibility, sitebuilding efficiency and re-usability of code.

To use this module, add and enable it just like any other, then visit one of your Manage Display screens. The most widely available formatter is the Wrapper (with class) one, but the others follow a similar naming convention; "Formatter name (with class)". The majority of these formatters extend a core formatter, so all the normal formatter options should still be available.

The manage display page with new (with class) field formatters selected

Then we'll use our field formatters to add my-custom-class into the markup, and see where it ends up.

Element Class Formatter

As mentioned above, this module adds a class to a fields element. So the actual link of a link field, for example. The field template markup is untouched.

The class is set at the view mode (configuration) level, so content editors don't get to choose what class goes on the link. So for our node teaser with link field example, all nodes get the same class every time the teaser is displayed. The new markup would be:

Field Formatter Class

The Field Formatter Class module is similar to the Element Class Formatter module except that is adds the class to field template markup, not the link. Otherwise it works the same way, on a view mode level.

I haven't actually used this module before, as it's more natural for me to work in Twig templates, utilising embeds and includes. But if you like doing things via the UI then check it out.

Entity Class Formatter

Entity Class Formatter is a very nice module which lets you (site builder or content editor) add a class to the fields parent entity. It further differs from the above modules in that it's a combination of configuration and content. You (the site builder) define a set of classes that the content editor can choose from. Each node can have a different class from the pre-defined list.

Which is really handy for adding things like grid or variant classes to things. As you can see it does nothing to the link fields markup, it's actually it's own separate field, and just utilises a Field Formatter so you can define how the field should be used.

Summary

You can actually use all these modules together, as they target different parts of the markup. They're complimentary, not competitors. We definitely use the entity_class_formatter together with element_class_formatter and let Twig handle the middle part.

Conclusion

The question is which is the best solution to run Nightwatch and JTB/WTB tests using the same setup?

We had seen some memory issues with Selenium containers in the past but we haven't run into any issue recently so I prefer this and you can swap Selenium container to use different browsers for testing.

We have also seen some issues while running ChromeDriver in WebDriver mode. It just stops working mid-test runs.

I was unable to get Headless Chrome working with Nightwatch but it needs more investigation.

Headless ChromeDriver setup on DrupalCI is quite stable. For JTB this would mean that we could use anyone from \Drupal\FunctionalJavascriptTests\DrupalSelenium2Driver and DMore\ChromeDriver.

Pagination

Add new comment

Malicious users can intercept or monitor plaintext data transmitting across unencrypted networks, jeopardising the confidentiality of sensitive data in Drupal applications. This tutorial will show you how to mitigate this type of attack by encrypting your database queries in transit.

With attackers and data breaches becoming more sophisticated every day, it is imperative that we take as many steps as practical to protect sensitive data in our Drupal apps. PreviousNext use Amazon RDS for our MariaDB and MySQL database instances. RDS supports SSL encryption for data in transit, and it is extremely simple to configure your Drupal app to connect in this manner.

1. RDS PEM Bundle

The first step is ensuring your Drupal application has access to the RDS public certificate chain to initiate the handshake. How you achieve this will depend on your particular deployment methodology - we have opted to bake these certificates into our standard container images. Below are the lines we've added to our PHP Dockerfile.

If that looks OK, you can take it a step further and sniff the TCP connection between Drupal and the RDS server.

This requires root access to your server, and the tcpflow package installed - this tool will stream the data being transmitted over port 3306. You are wanting to see illegible garbled data - definitely not content that looks like a SQL queries or responses!

Run this command, and click around your site while logged in (to ensure minimal cache hits).

$ tcpflow -i any -C -g port 3306

This is the type of output which indicates the connection is encrypted.

Add new comment

In a previous article Using ES6 in your Drupal Components, we discussed writing our javascript using the modern ES6 methods and transpiling down for older browsers. It still used jQuery as an interim step to make the process of refactoring existing components a little easier. But let's go all the way now and pull out jQuery, leaving only modern, vanilla javascript.

Why should we do this?

jQuery was first introduced 12 years ago, with the intention of making javascript easier to write. It had support for older browsers baked into it and improved the developer experience a great deal. It also adds 87KB to a page.

Today, modern vanilla javascript looks so much like jQuery! It’s support in the evergreen browsers is great and it’s so much nicer to write than it was 12 years ago. There are still some things that jQuery wins on but in the world of javascript frameworks, understanding the foundation on which they are built makes learning them so much easier.

And those older browsers? We don’t need jQuery for that either. You can support older browsers with a couple of polyfills. The polyfills I needed for the examples in this post only amounted to a 2KB file.

Drupal 8 and jQuery

One of the selling points of Drupal 8 (for us front-enders at least) was that jQuery would be optional for a theme. You choose to add it as a dependency. A lot of work has gone into rewriting core JS to remove the reliance on jQuery. There are still some sections of core that need work - Ajax related stuff is a big one. But even if you have a complex site which uses features that add jQuery in, it's still only going to be on the pages that need it. Plus we can help! Create issues and write patches for core or contrib modules that have a dependency on jQuery.

So what does replacing jQuery look like?

In the Using ES6 blog post I had the following example for my header component.

We’ve replaced $.extend with Object.assign for our default/overridable options. We use context.querySelectorAll('.header'') instead of $('.header', context) to find all instances of .header. We’ve also moved the .each((i, obj) => {}) to the .drupal file as .forEach((obj) => {}) to simplify our called function. Overall not very different at all!

We could go further and convert our functions to Classes, but if you're just getting started with ES6 there's nothing wrong with taking baby steps! Classes are just fancy functions, so upgrading to them in the future would be a great way to learn how they work.

Everything I’ve mentioned here that’s linked to the MDN web docs required a polyfill for IE, which is available on their respective docs page.

If you’re refactoring existing JS it’s also a good time to make sure you have some Nightwatch JS tests written to make sure you’re not breaking anything :)

Polyfills and Babel

Babel is the JS transpiler we use and it can provide the polyfills itself (babel-polyfill), but due to the nature of our component library based approach, Babel would transpile the polyfills needed for each component into that components JS file. If you bundle everything into one file then obviously this won’t be an issue. But once we start having a couple of different components JS loaded on a page, all with similar polyfills in them you can imagine the amount of duplication and wasted KB.

I prefer to just put the polyfills I need into one file and load it separately. It means have full control over the quality of my polyfills (since not all polyfills are created equally). I can easily make sure I’m only polyfilling what I really need. I can easily pull them out when no longer needed, and I’m only loading that polyfill file to browsers that need it;

js/polyfill.min.js : { attributes: { nomodule: true, defer: true } }

This line is from my themes libraries.yml file, where I'm telling Drupal about the polyfill file. If I pass the nomodule attribute in browsers who DO support ES6 modules will ignore this file, but browsers like IE load it. We're also deferring the file so it's loading after everything else.

I should point out Babel is still needed. We can't polyfill everything (like Classes or Arrow functions) and we can't Transpile everything either. We need both, at least until IE stops requiring support.

Comments

Great article, as always!Wondering if you still use Rollup.js as a bundler or along the way you found out a better tool?(Or reverted to Webpack)Thanks!Gab

Thanks Gab, yeah we still use Rollup.js for the most part. Some of the more app-like projects are using Webpack, though I'm curious to try out Parcel.js one day too.

How to replace jquery.once when we're using vanilla js?

The addEventLister() function has an option you can set to ensure it only runs once, though it also requires a polyfill. Check out this post (which also shows the alternative approach of using removeEventLister().

Pagination

Add new comment

Automated accessibility tools are only one part of ensuring a website is accessible, but it is a very simple part that can catch a lot of really easy to fix issues. Issues that when found and corrected early in the development cycle, can go a long way to ensuring they don’t get compounded into much larger issues down the track.

I’m sure we all agree that the accessibility of ALL websites is important. Testing for accessibility (a11y) shouldn’t be limited to Government services. It shouldn’t be something we need to convince non-government clients to set aside extra budget for. It certainly shouldn’t be left as a pre-launch checklist item that only gets the proper attention if the allocated budget and timeframe hasn’t been swallowed up by some other feature.

Testing each new component or feature against an a11y checker, as it’s being developed, takes a small amount of time. Especially when compared to the budget required to check and correct an entire website before launch -- for the very first time. Remembering to run such tests after a components initial development is one thing. Remembering to re-check later down the line when a few changes and possible regressions have gone through is another. Our brains can only do so much, so why not let the nice, clever computer help out?

NightwatchJS

NightwatchJS is going to be included in Drupal 8.6.x, with some great Drupal specific commands to make functional javascript testing in Drupal super easy. It's early days so the documentation is still being formed. But we don't have to wait for 8.6.x to start using Nightwatch, especially when we can test interactions against out living Styleguide rather than booting up Drupal.

We're pointing to our theme and custom modules as the source of our JS tests as we like to keep the tests close to the original JS. Our test settings are largely based on the Docker setup described below, with the addition of the 'filter' setting which searches the source for .js files inside a tests directory.

A test could be as simple as checking for an attribute, like the following example;

Which launches the Styleguides table component, waits a beat for the JS to initiate then checks that our td elements have the data-label that our JS added. Or is could be much more complex.

aXe: the Accessibility engine

aXe is a really nice tool for doing basic accessibility checks, and the Nightwatch Accessibility node module integrates aXe with Nightwatch so we can include accessibility testing within our functional JS tests without needing to write out the rules ourself. Even if you don't write any component specific tests with your Nightwatch setup, including this one accessibility test will give you basic coverage.

$ npm install nightwatch-accessibility

Then we edit our nightwatch.json file to include the custom_commands_path and custom_assertions_path;

Here we're configuring aXe core to check for wcag2a and wcag2aa, for anything inside the .kss-modifier__example selector of our Styleguide. Running this will check all of our components and tell us if it's found any accessibility issues. It'll also fail a build, so when hooked up with something like CircleCI, we know our Pull Requests will fail.

If we want to exclude a selector, instead of the .kss-modifier__example selector, we pass an include/exclude object { include: ['.kss-modifier__example'], exclude: ['.hljs'] }.

If you only add one test add one like this. Hopefully once you get started writing Nightwatch tests you'll see how easy it is and eventually add more :)

You can include the accessibility test within another functional test too, for example a modal component. You'll want to test it opens and closes ok, but once it's open it might have some accessibility issues that the overall check couldn't test for. So we want to re-run the accessibility assertion once it's open;

If you're not using docker or any CI tools and just want to test this stuff locally, there's a node module for adding the selenium-webdriver but I haven't tested it out with Nightwatch.

Don’t forget the manual checks!

There’s a lot more to accessibility testing than just these kinds of automated tests. A layer of manual testing will always be required to ensure a website is truly accessible. But automating the grunt work of running a checklist against a page is one very nice step towards an accessible internet.

Add new comment

Back in the Drupal 6 days, I built the BOM Weather Drupal module to pull down weather data from the Australian Bureau of Meteorology (BOM) site, and display it to users.

We recently had a requirement for this in a new Drupal 8 site, so decided to take a more modern approach.

Not that kind of decoupled Drupal

We often hear the term Decoupled Drupal but I'm not talking about a Javascript front-end and Drupal Web Service API backend.

This kind of decoupling is removing the business logic away from Drupal concepts. Drupal then becomes a wrapper around the library to handle incoming web requests, configuration and display logic.

We can write the business logic as a standalone PHP package, with it's own domain models, and publish it to Packagist.org to be shared by both Drupal and non-Drupal projects.

The Bom Weather Library

We started by writing unit-testable code, that pulled in weather forecast data in an XML format, and produced a model in PHP classes that is much easier for consuming code to use. See the full BOM Weather code on GitHub

The library takes care of fetching the data, and the idiosyncrasies of a fairly crufty API (no offence intended!).

Unit Testing

We can have very high test coverage with our model. We can test the integration with mock data, and ensure a large percentage of the code is tested. As we are using PHPUnit tests, they are lightning fast, and are automated as part of a Pull Request workflow on CircleCI.

Any consuming Drupal code can focus on testing just the Drupal integration, and not need to worry about the library code.

Dependency Management

As this is a library, we need to be very careful not to introduce too many runtime dependencies. Also any versions of those dependencies need to be more flexible than what you would normally use for a project. If you make your dependency versions too high they can introduce incompatibilities when used a project level. Consumers will simply not be able to add your library via composer.

We took a strategy with the BOM Weather library of having high-low automated testing via CircleCI. This means you test using both:

composer update --prefer-lowest

and

composer update

The first will install the lowest possible versions of your dependencies as specified in your composer.json. The second will install the highest possible versions.

This ensures your version constraints are set correctly and your code should work with any versions in between.

Conclusion

At PreviousNext, we have been using the decoupled model approach on our projects for the last few years, and can certainly say it leads to more robust, clean and testable code. We have had projects migrate from Drupal 7 to Drupal 8 and as the library code does not need to change, the effort has been much less.

Comments

Thanks for writing this! It's great to see this approach gain traction in Drupal 8. We're doing the same thing with the Drupal 8 version of the media_mpx module (library at https://github.com/Lullabot/mpx-php). As you say, test coverage of the critical functionality is so much simpler when you aren't dealing with the testing difficulties of Drupal 8 entities.

We've had good success bridging Drupal services back into non-Drupal libraries. For example, we use the cache PSR's to allow the PHP library to save data to Drupal's cache. You might be interested in https://github.com/Lullabot/drupal-symfony-lock which does the same thing for locks.

In this post, I will show you how you can add your own processed fields to be output via the REST API.

The "processed" property mentioned above is what is known as a computed property on the textarea field.

The ability to make the computed properties available for the REST API like this can be very helpful. For example, when the user inputs the raw value and Drupal performs some complex logical operations on it before showing the output.

Drupal fieldable entities can also have computed properties and those properties can also be exposed via REST. I used the following solution to expose the data of an entity field which takes raw data from the users and perform some complex calculations on it.

The field we add is not a base field so we can't use \Drupal\Core\Field\BaseFieldDefinition. There is an open core issue to address that https://www.drupal.org/project/drupal/issues/2346347 but in tests there is a workaround using a copy of \Drupal\entity_test\FieldStorageDefinition:

Last but not least we need to announce our property definition to the entity system so that it can keep track of it. As it is an existing bundle we can write an update hook. Otherwise, we'd need to implement hook_entity_bundle_create.

Related Drupal core issues:

Add new comment

If you've ever patched Drupal core with Composer you may have noticed patched files can sometimes end up in the wrong place like core/core or core/b. Thankfully there's a quick fix to ensure the files make it into the correct location.

However in certain situations patches will get applied incorrectly. This can happen when the patch is only adding new files (not altering existing files), like in the patch above. The result is the patched files end up in a subfolder core/core. If the patch is adding new files and editing existing files the new files will end up in core/b. This is because composer-patches cycle through the -p levels trying to apply them; 1, 0, 2, then 4.

Thankfully there is an easy fix!

"extra": {
...
"patchLevel": {
"drupal/core": "-p2"
}
}

Setting the patch level to p2 ensures any patch for core will get applied correctly.

Note that until composer-patches has a 1.6.5 release, specifically this PR, you'll need to use the dev release like:

"require": {
...
"cweagans/composer-patches": "1.x-dev"
}

The 2.x branch of composer-patches also includes this feature.

Big thanks to cweagans for this great tool and jhedstrom for helping to get this into the 1.x branch.

Comments

thanks for the blog @Saul Willers .

Fantastic, thanks Cameron!

Thanks for the trick.

And just as an addendum about *creating* patches from a split core git repository: make sure to use "git diff --src-prefix=a/core/ --dst-prefix=b/core/".

Pagination

Add new comment

Several times in the past I've been caught out by Drupal's cron handler silently catching exceptions during tests.

Your test fails, and there is no clue as to why.

Read on to find out how to shine some light on this, by making your kernel tests fail on any exception during cron.

If you're running cron during a kernel test and expecting something to happen, but it doesn't - it can be hard to debug why.

Ordinarily an uncaught exception during a test will cause PHPUnit to fail, and you can pinpoint the issue.

However, if you're running cron in the test this may not be the case.

This is because, by default Drupal's cron handler catches all exceptions and silently logs them. This is colloquially known as Pokemon exception handling.

The act of logging an exception is not enough to fail a test.

So your test skips the exception and carries on, failing in other ways unexpectedly.

This is exacerbated by the fact that PHP Unit throws an exception for warnings. So the slightest issue in your code will cause it to halt execution. In an ordinary scenario, this exception causes the test to fail. But the pokemon catch block in the Cron class prevents that, and your test continues in a weird state.

This is the code in question in the cron handler

<?php
try {
$queue_worker->processItem($item->data);
$queue->deleteItem($item);
}
// ... catch (\Exception $e) {
// In case of any other kind of exception, log it and leave the item
// in the queue to be processed again later.
watchdog_exception('cron', $e);
}

So how do you make this fail your test? In the end, it's quite simple.

Firstly, you make your test a logger and use the handy trait to do the bulk of the work.

You only need to implement the log method, as the trait takes care of handling all other methods.

In this case, watchdog_exception logs exceptions as RfcLogLevel::ERROR. The log levels are integers, from most severe to least severe. So in this implementation we tell PHP Unit to fail the test with any messages logged where the severity is ERROR or worse.

Add new comment

In most of the projects we build, the HTML markup provided by core just gets in the way. There is way too many wrapper divs. This can cause issues when trying to create lean markup that matches what is produced in a generated styleguide.

In this post, I'll introduce you to the concept of bare templates, and how you can remove unnecessary markup from your Twig templates.

In Drupal 8, a couple of themes are shipped by default to serve a common set of end user needs.

Among them are:

Bartik: A flexible, recolourable theme with many regions and a responsive, mobile-first layout.

Seven: The default administration theme for Drupal 8 was designed with clean lines, simple blocks, and sans-serif font to emphasise the tools and tasks at hand.

Stark: An intentionally plain theme with almost no styling to demonstrate default Drupal’s HTML and CSS.

Stable: A base theme. Stable theme aggregates all of the CSS from Drupal core into a single theme. Theme markup and CSS will not change so any sub-theme of Stable will know that updates will not cause it to break.

Classy: A sub-theme of Stable. Theme designed with lot of markups for beginner themers.

But in an actual business scenario the requirements and expectations of a client towards the look and feel of the website is far more distinct than the themes that are provided in Drupal core.

When building your site based upon one of these themes it is common to face issues with templating during the frontend implementation phase. Quite often the default suggested templates for blocks, nodes, fields etc. contain HTML wrapper divs that your style guide doesn’t require.

Usually the most effective way is to build themes using the Stable theme. In Stable, the theme markup and CSS are fixed between any new Drupal core releases making any sub-theme to less likely to break on a Drupal core update. It also uses the verbose field template support for debugging.

Which leads us to use bare templates.

What is a bare template?

A bare template is a twig file that has the minimum number of HTML wrappers around actual content. It could be simple as a file with a single content output like {{content.name}}

Ease of maintenance: With minimum markup the complexity of the template is much lesser making it easy to maintain.

Cleaner Markup: The markup will only have the essential or relevant elements where as in traditional approach there are a lot of wrappers leading to a complex output.

Smaller page size: Less markup means less page size.

Avoids the need for markup removal modules: With bare markup method we do not need to use modules like fences or display suite. Which means less modules to maintain and less configuration to worry about.

Our Example

We need to create a bare template for type field and suggest it to render only field name and field_image of my_vocabulary taxonomy entity. This will avoid Drupal from suggesting this bare template for other fields belonging to different entities.

Field template

Let's have a look at field template which resides at app/core/themes/stable/templates/field/field.html.twig

As you see there is quite a lot of div wrappers used in the default template which makes it difficult to style components. If you are looking for simple output, this code is overkill. There is however, a lot of valuable information is provided in the comments of field.html.twig which we can use.

{#
/**
* @file
* Theme override for a field.
*
* To override output, copy the "field.html.twig" from the templates directory
* to your theme's directory and customize it, just like customizing other
* Drupal templates such as page.html.twig or node.html.twig.
*
* Instead of overriding the theming for all fields, you can also just override
* theming for a subset of fields using
* @link themeable Theme hook suggestions. @endlink For example,
* here are some theme hook suggestions that can be used for a field_foo field
* on an article node type:
* - field--node--field-foo--article.html.twig
* - field--node--field-foo.html.twig
* - field--node--article.html.twig
* - field--field-foo.html.twig
* - field--text-with-summary.html.twig
* - field.html.twig
*
* Available variables:
* - attributes: HTML attributes for the containing element.
* - label_hidden: Whether to show the field label or not.
* - title_attributes: HTML attributes for the title.
* - label: The label for the field.
* - multiple: TRUE if a field can contain multiple items.
* - items: List of all the field items. Each item contains:
* - attributes: List of HTML attributes for each item.
* - content: The field item's content.
* - entity_type: The entity type to which the field belongs.
* - field_name: The name of the field.
* - field_type: The type of the field.
* - label_display: The display settings for the label.
*
* @see template_preprocess_field()
*/
#}

The code

Building the hook.

We will be using hook_theme_suggestions_HOOK_alter() to suggest the two fields to use our bare template when rendering.

It is important to note that only these two fields will be using the bare template and the other fields (if any) in that entity will use the default field.html.twig template to render.

The hook key field__no_markup mentioned in the code corresponds to a twig file which must reside under app/themes/custom/my_theme/templates/field/field--no-markup.html.twig

Debugging Output

In order to see how this is working, we can fire up PHPStorm and walk the code in the debugger.

As you can see in the output below, the implode() creates the actual var structure from the second parameter. We will use this to compare with the $bare_hooks array we created fields specific to content entity types that we need to assign the bare template.

Note: As best practise make sure you pass a third argument TRUE to in_array(). Which will validate the data type as well.

Bare Template Markup

The following is the contents of our bare template file. Notice the lack of any HTML?

Bare templating can be used for other commonly used templates as well. To make it render a minimal amount of elements.

Conclusion

We can always use custom templating to avoid getting into complicated markups. And have the flexibility to maintain the templates to render for specific entities.

Resources

Comments

Great post! I love getting to the cleanest markup possible.

Since the field templates don't have the `attributes`, have you run into any issues with Contextual Links & Quick Edit working? I've run into this issue trying to achieve the same thing using different methods:

Pagination

Add new comment

Since the release of Drupal 8, it has become tricky to determine what and where override configuration is set.

Here are some of the options for a better user experience.

Drupal allows you to override configuration by setting variables in settings.php. This allows you to vary configuration by which environment your site are served. In Drupal 7, when overrides are set, the overridden value is immediately visible in administration UI. Though the true value is transparent, when a user attempts to change configuration, the changes appear to be ignored. The changes are saved and stored. But Drupal exposes the overridden value when a configuration form is (re)loaded.

With Drupal 8, the behaviour of overridden configuration has reversed. You are always presented with active configuration, usually set by site builders. When configuration is accessed by code, overrides are applied on top of active configuration seamlessly. This setup is great if you want to deploy the active configuration to other environments. But it can be confusing on sites with overrides, since its not immediately obvious what Drupal is using.

An example of this confusion is: is your configuration forms show PHP error messages are switched-on, but no messages are visible. Or, perhaps you are overriding Swiftmailer with environment specific email servers. But emails aren't going to the servers displayed on the form.

A Drupal core issue exists to address these concerns. However this post aims to introduce a stopgap. In the form of a contrib module, of course.

Introducing Configuration Override Inspector (COI). This module makes configuration-overrides completely transparent to site builders. It provides a few ways overridden values can be exposed to site builders.

The following examples show error settings set to OFF in active configuration, but ON in overridden configuration. (such as a local.settings.php override on your dev machine)

// settings.php
$config['system.logging']['error_level'] = 'verbose';

Hands-off: Allow users to modify active configuration, while optionally displaying a message with the true value. This is most like out-of-the-box Drupal 8 behaviour:

Expose and Disable: Choose whether to disable form fields with overrides display the true value as the field value:

Add new comment

In one of our recent projects, our client made a request to use LinkIt module to insert file links to content from the group module. However, with the added distinction of making sure that only content that is in the same group as the content they are editing is suggested in the matches.

Here’s how we did it.

The LinkIt module

First, let me give you a quick overview of the LinkIt module.

LinkIt is a tool that is commonly used to link internal or external artifacts. One of the main advantages of using it is because LinkIt maintains links by uuid which means no occurrence for broken links. And it can link any type of entity varying from core entities like nodes, users, taxonomy terms, files, comments and to custom entities created by developers.

Once you install the module, you need to set a Linkit profile which consists of information about which plugins to use. To set the profiles use /admin/config/content/linkit path. And the final step will be to enable the Linkit plugin on the text format you want to use. Formats are found at admin/config/content/formats. And you should see the link icon when editing content item.

Once you click on the LinkIt icon it will prompt a modal as shown below.

By default LinkIt ships with a UI to maintain profiles that enables you to manage matchers.

Matchers

Matchers are responsible for managing the autoload suggestion criteria for a particular LinkIt field. It provides bundle restrictions and bundle grouping settings

Proposed resolution

To solve the issue; we started off by creating a matcher for our particular entity type. Linkit has an EntityMatcher plugin that uses Drupal's Plugin Derivatives API to expose one plugin for each entity type. We started by adding the matcher that linkit module exposed for our custom group content entity type.

We left the bundle restrictions and bundle grouping sections un-ticked so that all existing bundles are allowed so the content of those bundles will be displayed.

Now that the content is ready we have to let the matcher know that we only need to load content that belongs to the particular group for which the user is editing or creating the page.

Using the deriver

In order to do that we have to create a new class in /modules/custom/your_plugin_name/src/Plugin/Linkit/Matcher/YourClassNameMatcher.php by extending existing EntityMatcher class which derives at /modules/contrib/linkit/src/Plugin/Linkit/Matcher/EntityMatcher.php.

Because Linkit module's plugin deriver exposes each entity-type plugin with and ID for the form entity:{entity_type_id} we simply need to create a new plugin with an ID that matches our entity type ID. This then takes precedence over the default derivative based plugin provided by Linkit module. We can then modify the logic in either the ::execute() or ::buildEntityQuery method.

Using LinkIt autocomplete request

But here comes the challenge, in that content edit page the LinkIt modal doesn’t know about the group of the content being edited, therefore we cannot easily filter the suggestions based on the content being edited. We need to take some fairly extreme measures to make that group ID available for our new class to filter the content once the modal is loaded and user starts typing in the field.

In this case the group id is available from the page uri.

So in order to pass this along, we can make use of the fact that the linkit autocomplete widget has a data attribute 'data-autocomplete-path' which is used by its JavaScript to perform the autocomplete request. We can add a process callback to the LinkIt element to extract the current page uri and pass it as a query parameter in the autocomplete path.

The code

To do so we need to implement hook_element_info_alter in our custom module. Here we will add a new process callback and in that callback we can add the current browser url as a query parameter to the data-autocomplete-path attribute of the modal.

Below is the code to add the process callback and alter the data-autocomplete-path element. We rely on the HTTP Referer header which Drupal sends in its AJAX request that is used to display the LinkIt modal, which in turn builds the LinkIt element

Conclusion

By re-implementing the execute() method of EntityMatcher class we are now able to make the LinkIt field to display only content from the same group as the content the user is editing/creating.

So next challenge here is to create some test coverage for this, as we're relying on a few random pieces of code - a plugin, some JavaScript in the LinkIt module, an element info alter hook and a process callback - any of which could change and render all of this non-functional. But that's a story for another post.

Comments

I am the maintainer for the Linkit module, and I really liked this post. Glad you found it (quite) easy to extend.

Hey there, I've got linkit 8.x-4.3 and the class EntitySuggestion and SuggestionCollection don't even seem to exist at all? So the use statements fail and everything after that. Is there some aspect you did not include in this description?

Pagination

Add new comment

Great to see this project a really good page builder is badly needed for Drupal - looks like a very good start, well done Lee.

Not sure if you are familiar with the layout builder and visual composer build by NikaDevs (a theme company) but you could do a lot worse then having a look at their approach, it's a very good page builder - which they ave on all their themes.

When optimising a site for performance, one of the options with the best effort-to-reward ratio is image optimisation. Crunching those images in your Front End workflow is easy, but how about author-uploaded images through the CMS?

Recently, a client of ours was looking for ways to reduce the size of uploaded images on their site without burdening the authors. To solve this, we used the module Image Optimize which allows you to use a number of compression tools, both local and 3rd party.

The tools it currently supports include:

Local

PNG

JPEG

3rd Party

We decided to avoid the use of 3rd party services, as processing the images on our servers could reduce processing time (no waiting for a third party to reply) and ensure reliability.

In order to pick the tools which best served our we picked an image that closely represented the type of image the authors often used. We picked an image featuring a person’s face with a complex background - one png and one jpeg, and ran it through each of the tools with a moderately aggressive compression level.

Using a combination of PngQuant and JpegOptim, we could save anywhere between 14% and 89% in file size, with larger images bringing greater percentage savings.

Setting up automated image compression in Drupal 8

The Image Optimize module allows us to set up optimisation pipelines and attach them to our image styles. This allows us to set both site-wide and per-image style optimisation.

After installing the Image Optimize module, head to the Image Optimize pipelines configuration (Configuration > Media > Image Optimize pipeline) and add a new optimization pipeline.

Now add the PngQuant and JpegOptim processors. If they have been installed to the server Image Optimize should pick up their location automatically, or you can manually set the location if using a standalone binary.

JpegOptim has some additional quality settings, I’m setting “Progressive” to always and “Quality” to a sweet spot of 60. 70 could also be used as a more conservative target.

The final pipeline looks like the following:

Back to the Image Optimize pipelines configuration page, we can now set the new pipeline as the sitewide default:

And boom! Automated sitewide image compression!

Overriding image compression for individual image styles

If the default compression pipeline is too aggressive (or conservative) for a particular image style, we can override it in the Image Styles configuration (Configuration > Media > Image styles). Edit the image style you’d like to override, and select your alternative pipeline:

Applying compression to existing images

Flushing the image cache will recreate existing images with compression the next time the image is loaded. This can be done with the drush command

drush image-flush --all

Conclusion

Setting up automated image optimisation is a relatively simple process, with potentially large impacts on site performance. If you have experience with image optimisation, I would love to hear about it in the comments.

Comments

Did you consider using mod_pagespeed at all? If you did what made you decide against it?

Good question, Drupal performs many of the optimisations that mod_pagespeed does but allows us more granular control. One of the benefits of this approach is being able to control compression levels per image style. As Drupal is resizing and reshaping images then anyway, I feel it makes sense to do the compression at the same time.

Hi Tony,Nice of you to post some details on this.

How does this integrate with Responsive Images and the Picture fields?

Can it crop and scale immediately after upload to get multiple files for multiple view ports?

Regards

Hi Saj, Responsive Images picks up its variants from the Image Styles so this works seamlessly. You can set your image dimensions and cropping in the image style, and the compression is applied after that.

Nice write up! I never knew about this Drupal module.

It'd be nice to compare the Original Image + Drupal Compression + Final Selected compression library output through some image samples.

Also might worth mentioning that PngQuant is a lossly image compression algorithm - and the others aren't (hence the big compression difference).

I'd recommend running optipng or pngcrush after pngquant to get an even more compressed image. Careful though, this can burn CPU cycles, especially with the module's obsessive parameter choices. Have a look at the $cmd entries in binaries/*.inc if you're curious.

Hi Christoph,

How do you define the order by which these compressions are applied?

Great article btw! The comparison metric is quite useful in knowing which tool is the best performer. I initially went for jpegtran but jpegoptim is producing way better results.

Thanks

I recommend one of the external services when you’re on a host where you can’t install extra server software (like Pantheon).

Hi! Nice post. Which version did you use with Drupal 8 ?

8.x-2.0-alpha3 ? No issues with alpha version ? Thanks

Hi! Did you try this version of the module with actual Drupal version?What result have you got?Thanks

I am glad to read such a huge stuff about the picture optimisation in Drupal. This looks really interesting to me and I would love to try out out after my bus tours from washington dc

Hi Tony,Nice explanation of use of imageAPI module. m a newbie in D8, working on a live site.. I had a question regarding, setting manual path of pngquant.. As understood in windows, php folder should have its dll file, but as i am working on server directly, i dont know how to proceed from that step. Please do help.

Tagged

All PreviousNext Drupal 8 projects are now managed using Composer. This is a powerful tool, and allows our projects to define both public and private modules or libraries, and their dependencies, and bring them all together.

However, a if you require public or private modules which are hosted on GitHub you may run into the API Rate Limits. In order to overcome this, it is recommended to add a GitHub personal access token to your composer configuration.

In this blog post, I'll show how you can do this in a secure and manageable way.

It's common practice when you encounter a Drupal project to see the following snippet in a composer.json file:

What this means is, everyone is sharing a single account's personal access token. While this may be convenient, it's also a major security risk should the token accidentally be made public, or a team member leaves the organisation, and still has read/write access to your repositories.

A better approach, is to have each team member have their own personal access token configure locally. This ensures that individuals can only access repositories they have read permissions for, and once they leave your organisation they can no longer access any private dependencies.

You can simply set this environment variable in your CI Environment (e.g. CircleCI, TravisCI, Jenkins) and have a personal access token specific to the CI environment.

Summary

By using Personal Access Tokens, you can now safely remove any tokens from the project's composer.json file, removing the risk this gets exposed. You can also know that by removing access for any ex-team members, they are no longer able to access your organisations repos using a token. Finally, in the event of a token being compromised, you have reduced the attack surface, and can more easily identify which user's token was used.

Comments

Pagination

Add new comment

Managing technical debt is important for the health of all software projects. One way to manage certain types of technical debt is to revisit code and decide if it’s still relevant to the project and to potentially remove it. Doing so can reducing complexity and the amount of code developers are required to maintain.

To address this we’ve been experimenting with adding simple annotations to code, which indicate an “expiry”. A nudge to developers to go and reevaluate if some bit of code will still be needed at some point in the future. This can be integrated into CI pipelines to fail builds which have outstanding expiry annotations.

Some scenarios where this has proved to be helpful have been:

Removing workarounds in CSS to address bugs in web browsers which have since been fixed.

Removing uninstalled modules, which were required only for hook_uninstall.

Removing code that exists for features which are gradually being superseded, like an organisation gradually migrating content from nodes into a new custom entity.

Here is an real snippet of code we were able to recently delete from a project, based on a bug which was fixed upstream in Firefox. I don’t believe without an explicit prompt to revisit the code, which was introduced many months earlier, we would have been able to confidently clean this up.

// @expire Jan 2018
// Fix a bug in firefox which causes all form elements to match the exact size
// specified in the "size" or "cols" attribute. Firefox probably will have
// fixed this bug by now. Test it by removing the following code and visiting
// the contact form at a small screen size. If the elements dont overflow the
// viewport, the bug is fixed.
.form-text__manual-size {
width: 529px;
@media (max-width: 598px) {
width: 100%;
}
}

The code we've integrated into our CI pipeline to check these expiry annotations simply greps the code base for strings matching the expiry pattern for the last n months worth of time:

Pagination

Add new comment

With the release of Drupal 8.4.x and its use of ES6 (Ecmascript 2015) in Drupal core we’ve started the task of updating our jQuery plugins/widgets to use the new syntax. This post will cover what we’ve learnt so far and what the benefits are of doing this.

If you’ve read my post about the Asset Library system you’ll know we’re big fans of the Component-Driven Design approach, and having a javascript file per component (where needed of course) is ideal. We also like to keep our JS widgets generic so that the entire component (entire styleguide for that matter) can be used outside of Drupal as well. Drupal behaviours and settings are still used but live in a different javascript file to the generic widget, and simply call it’s function, passing in Drupal settings as “options” as required.

Here is an example with an ES5 jQuery header component, with a breakpoint value set somewhere in Drupal:

Converting these files into a different language is relatively simple as you can do one at a time and slowly chip away at the full set. Since ES6 is used in the popular JS frameworks it’s a good starting point for slowly moving towards a “progressively decoupled” front-end.

Support for ES6

Before going too far I should mention support for this syntax isn’t quite widespread enough yet! No fear though, we just need to add a “transpiler” into our build tools. We use Babel, with the babel-preset-env, which will convert our JS for us back into ES5 so that the required older browsers can still understand it.

Our Gulp setup will transpile any .es6.js file and rename it (so we’re not replacing our working file), before passing the renamed file into out minifying Gulp task.

With the Babel ENV preset we can specify which browsers we actually need to support, so that we’re doing the absolute minimum transpilation (is that a word?) and keeping the output as small as possible. There’s no need to bloat your JS trying to support browsers you don’t need to!

On a side note, we’ll be outsourcing our entire Gulp workflow real soon. We’re just working through a few extra use cases for it, so keep an eye out!

Learning ES6

Reading about ES6 is one thing but I find getting into the code to be the best way for me to learn things. We like to follow Drupal coding standards so point our eslint config to extend what’s in Drupal core. Upgrading to 8.4.x obviously threw a LOT of new lint errors, and was usually disabled until time permitted their correction. But you can use these errors as a tailored ES6 guide. Tailored because it’s directly applicable to how you usually write JS (assuming you wrote the first code).

Working through each error, looking up the description, correcting it manually (as opposed to using the --fix flag) was a great way to learn it. It took some time, but once you understand a rule you can start skipping it, then use the --fix flag at the end for a bulk correction.

Of course you're also a Google away from a tonne of online resources and videos to help you learn if you prefer that approach!

ES6 with jQuery

Our original code is usually in jQuery, and I didn’t want to add removing jQuery into the refactor work, so currently we’re using both which works fine. Removing it from the mix entirely will be a future task.

The biggest gotcha was probably our use of this, once converted to arrow functions needed to be reviewed. Taking our header example from above:

return this.each(function () { var $header = $(this); }

Once converted into an arrow function, using this inside the loop is no longer scoped to the function. It doesn’t change at all - it’s not an individual element of the loop anymore, it’s still the same object we’re looping through. So clearly stating the obj as an argument of the .each() function lets us access the individual element again.

Since the exported ES6 module has to be a top level function, the jQuery wrapper was moved inside it, along with passing through the this object. There might be a nicer way to do this but I haven't worked it out yet! Everything inside the module is the same as I had in the jQuery plugin, just updated to the new syntax.

I also like to rename my modules when I export them so they’re name-spaced based on the project, which helps when using a mix of custom and vendor scripts. But that’s entirely optional.

Now that we have our generic JS using ES6 modules it’s even easier to share and reuse them. Remember our Drupal JS separation? We no longer need to load both files into our theme. We can import our ES6 module into our .drupal.js file then attach it as a Drupal behaviour.

So a few differences here, we're importing the myHeader function from our other file, we're destructuring our Drupal and drupalSettings arguments to simplify them, and using .call() on the function to pass in the object before setting its arguments. Now the header.drupal.js file is the only file we need to tell Drupal about.

Some other nice additions in ES6 that have less to do with jQuery are template literals (being able to say $(`.${opts.toggleClass}`) instead of $('.' + opts.toggleClass')) and the more obvious use of const and let instead of var , which are block-scoped.

Importing modules into different files requires an extra step in our build tools, though. Because browser support for ES6 modules is also a bit too low, we need to “bundle” the modules together into one file. The most popular bundler available is Webpack, so let’s look at that first.

Bundling with Webpack

Webpack is super powerful and was my first choice when I reached this step. But it’s not really designed for this component based approach. Few of them are truly... Bundlers are great for taking one entry JS file which has multiple ES6 modules imported into it. Those modules might be broken down into smaller ES6 modules and at some level are components much like ours, but ultimately they end up being bundled into ONE file.

But that’s not what I wanted! What I wanted, as it turned out, wasn’t very common. I wanted to add Webpack into my Gulp tasks much like our Sass compilation is, taking a “glob” of JS files from various folders (which I don’t really want to have to list), then to create a .bundle.js file for EACH component which included any ES6 modules I used in those components.

The each part was the real clincher. Getting multiple entry points into Webpack is one thing, but multiple destination points as well was certainly a challenge. The vinyl-named npm module was a lifesaver. This is what my Gulp talk looked like:

This worked. But Webpack has some boilerplate JS that it adds to its bundle output file, which it needs for module wrapping etc. This is totally fine when the output is a single file, but adding this (exact same) overhead to each of our component JS files, it starts to add up. Especially when we have multiple component JS files loading on the same page, duplicating that code.

It only made each component a couple of KB bigger (once minified, an unminified Webpack bundle is much bigger), but the site seemed so much slower. And it wasn’t just us, a whole bunch of our javascript tests started failing because the timeouts we’d set weren’t being met. Comparing the page speed to the non-webpack version showed a definite impact on performance.

So what are the alternatives? Browserify is probably the second most popular but didn’t have the same ES6 module import support. Rollup.js is kind of the new bundler on the block and was recommended to me as a possible solution. Looking into it, it did indeed sound like the lean bundler I needed. So I jumped ship!

Bundling with Rollup.js

The setup was very similar so it wasn’t hard to switch over. It had a similar problem about single entry/destination points but it was much easier to resolve with the ‘gulp-rollup-each’ npm module. My Gulp task now looks like:

Rollup.js does still add a little bit of boilerplate JS but it’s a much more acceptable amount. Our JS tests all passed so that was a great sign. Page speed tests showed the slight improvement I was expecting, having bundled a few files together. We're still keeping the original transpile Gulp task too for ES6 files that don't include any imports, since they don't need to go through Rollup.js at all.

Webpack might still be the better option for more advanced things that a decoupled frontend might need, like Hot Module Replacement. But for simple or only slightly decoupled components Rollup.js is my pick.

Next steps

Some modern browsers can already support ES6 module imports, so this whole bundle step is becoming somewhat redundant. Ideally the bundled file with it’s overhead and old fashioned code is only used on those older browsers that can’t handle the new and improved syntax, and the modern browsers use straight ES6...

Luckily this is possible with a couple of script attributes. Our .bundle.js file can be included with the nomodule attribute, alongside the source ES6 file with a type=”module” attribute. Older browsers ignore the type=module file entirely because modules aren’t supported and browsers that can support modules ignore the ‘nomodule’ file because it told them to. This article explains it more.

Then we'll start replacing the jQuery entirely, even look at introducing a Javascript framework like React or Glimmer.js to the more interactive components to progressively decouple our front-ends!

Comments

Is it absolutely necessary to bring all our jQuery plugins to ES6, or would they remain fine as it is?

They would be fine as is, you just won't get the full benefits of the ES6 module imports/exports. Being able to import a particular function (or all of them) from another file just means you can make things more reusable. You can be selective about what you convert too and just do the parts you know would benefit most from it.

Pagination

Add new comment

At DrupalSouth 2017, I presented a session on the new Workflows module, which just went stable in Drupal 8.4.0. Workflows was split out from content moderation as a separate module, and can be used independently to create custom workflows. In this presentation, I gave a demonstration of how to create a basic workflow for an issue tracker.

Since 2011 we have had access to a content moderation tool in Drupal 7 in the form of Workbench Moderation. This module introduced the concept of Draft ➡ Review ➡ Published workflows, with different user roles having specific permissions to move from one state to the next.

Unfortunately, the underlying Drupal core revision API was not designed to deal with this, and there were some pretty crazy workarounds.

Content moderation has long been a key feature request for Drupal, and so effort was made to port Workbench Moderation across to Drupal 8.

Content Moderation drove a lot of cleanup in Drupal core APIs, including proper support for forward revisions, and adding revision support to other content entities besides Content Types, such as Custom Blocks. More are on the way.

In Drupal 8.3, the Workflows module was split out of Content Moderation. Why you may ask? Well, because the Workflows module provides the state machine engine that Content Moderation relies on.

What is a State Machine?

A state machine defines a set of states and rules on how you can transition between those states.

A door state machine

In our simple example of a door, it can only be opened, closed or locked. However, you can't go directly from locked to open, you need to unlock it first.

Content Moderation Workflow Configuration

Content Moderation provides a set of Workflow states and transitions by default.

Content Moderation States
Content Moderation Transitions

If we were to put this together in a state machine diagram, it would look like the following:

Content Moderation State Machine

From the above diagram, it becomes clear what the allowed transitions are between states.

So now Workflows has been configured with our Content Moderation states and transitions, what is left for Content Moderation to do?

What Does Content Moderation Do?

It turns out quite a lot. Remember, that Workflows only provides the state machine. It in no way prescribes how you should manage the current state of a particular entity.

Content Moderation provides:

Default Workflows configuration

A fairly complex WorkflowType plugin which works with the Revision API.

Storage for individual states on content entities

Configuration of which entity bundles (Content types, etc.) should have Content Moderation

A number of admin forms for configuring the workflows and how they apply

Permissions

Building an Issue Tracker

We want to build a very simple issue tracker for our example. The state machine diagram is the following:

Issue Tracker State Machine

That's the simple bits out of the way. Now, in order to build an issue tracker, we will need to replicate the rest what Content Moderation does!

Fortunately there is a module that can do most of the heavy lifting for us.

Workflows Field

“This module provides a field which allows you to store states on any content entity and ensure the states change according to transitions defined by the core workflows module.”

Perfect! Let's download and install it.

Next we want to add a new Workflow. We can assign it a label of Issue Status and you'll see that we have a new Workflows Field option in the Workflow Type dropdown.

Add new workflow

We can then configure the desire Workflows states and transitions.

Issue States
Issue Transitions

Thats the our Workflows configured. Now we need to create a new Issue content type to attach our workflow to. It's assumed you know how to create a content type already. If not, check out the User Guide.

Next, we need to add our Workflows Field to our Issue content type. Follow the usual steps to add a field, and in the drop down choose Workflows as the field type, and our previously created Issue Status workflow.

Add workflows field

Test it out!

Now we can test our our workflow by creating a new Issue from the Content page. If everything was configured correctly, we should see a new field on the edit form for Status.

Issue status form

Given the transitions we defined in our workflow, we should only be allowed to see certain values in the drop-down, depending on the current state.

Testing workflow constraints

What next?

That's it for setting up and configuring a custom workflow using Workflows Field. Some next steps would be:

Add permissions for certain users (there's an issue for that #2904573 )

Add new comment

As seen in the recent Uber hack, storing secrets such as API tokens in your project repository can leave your organisation vulnerable to data breaches and extortion. This tutorial demonstrates a simple and effective way to mitigate this kind of threat by leveraging Key module to store API tokens in remote key storage.

Even tech giants like Uber are bitten by poor secret management in their applications. The snippet below describes how storing AWS keys in their repository resulted in a data breach, affecting 57 million customers and drivers.

Here’s how the hack went down: Two attackers accessed a private GitHub coding site used by Uber software engineers and then used login credentials they obtained there to access data stored on an Amazon Web Services account that handled computing tasks for the company. From there, the hackers discovered an archive of rider and driver information. Later, they emailed Uber asking for money, according to the company.

Uber could have avoided this breach by storing their API keys in a secret management system. In this tutorial, I'll show you how to do exactly this using the Key module in conjunction with the Lockr key management service.

This guide leverages a brand-new feature of Key module (as of 8.x-1.5) which allows overriding any configuration value with a secret. In this instance we will set up the MailChimp module using the this secure config override capability.

Service Set-Up

Before proceeding with the Drupal config, you will need a few accounts:

Note down this API key so we can configure in Drupal in the next step.

In your Drupal tab, go to /admin/config/system/keys and click Add Key

Create a new Key entity for your MailChimp token. The important values here are:

Key provider - ensure you select Lockr

Value - paste the API token you obtained from the MailChimp dashboard.

Now we need to set up the configuration overrides. Go to /admin/config/development/configuration/key-overrides and click Add Override

Fill out this form, the important values here are:

Configuration type: Simple configuration

Configuration name: mailchimp.settings

Configuration item: api_key

Key: The name of the key you created in the previous step.

... and it is that simple.

Result

The purpose of this exercise is to ensure the API token for our external services are not saved in Drupal's database or code repository - so lets see what those look like now.

MailChimp Config Export - Before

If you configured MailChimp in the standard way, you'd see a config export similar to this. As you can see, the api_key value is in plaintext - anyone with access to your codebase would have full access to your MailChimp account.

MailChimp Config Export - After

There are a few other relevant config export files - lets take a look at those.

Key Entity Export

This export is responsible for telling Drupal where Key module stored the API token. If you look at the key_provider and key_provider_settings values, you'll see that it is pointing to a value stored in Lockr. Still no API token in sight!

Conclusion

Hopefully this tutorial shows you how accessible these security-hardening techniques have become.

With this solution implemented, an attacker can not take control of your MailChimp account simply by gaining access to your repository or a database dump. Also remember that this exact technique can be applied to any module which uses the Configuration API to store API tokens.

Why? Here are a few examples of ways popular Drupal modules could harm your organisation if their configs were exposed (tell me about your own worst-case scenarios in the comments!).

s3fs - An attacker could leak or delete all of the data stored in your bucket. They could also ramp up your AWS bill by storing or transferring terabytes of data.

SMTP - An attacker could use your own SMTP server against you to send customers phishing emails from a legitimate email address. They could also leak any emails the compromised account has access to.

What other Drupal modules could be made more securing in this way? Post your ideas in the comments!

Step 1: Enable a display suite layout for the view mode

To use the chained fields functionality, you must enable a display suite layout for the view mode. Select a layout other than none and hit Save.

Enabling a layout

Step 2: Enable the entity reference fields you wish to chain

To keep the manage display list from being cluttered, you must manually enable the entity reference fields you wish to show chained fields from. For example, to show the author's picture, you might enable the 'Authored by' entity reference field, which points to the author. After you've enabled the required fields, press Save.

Enabling fields for chaining

Step 3: Configure the chained fields as required

Finally, just configure the chained fields as normal.

Configuring chained fields

That's it - let me know your thoughts in the comments or the the issue queue.

In my recent talk at DrupalSouth Auckland 2017 I took a hard look at the hyperbole of Drupal supposedly powering over a million websites. Where does Drupal really sit in relation to other CMS platforms, both open source and proprietary? What trends are emerging that will impact Drupal's market share? The talk looked outside the Drupal bubble and took a high level view of its market potential and approaches independent firms can take to capitalise on Drupal's strengths and buffer against its potential weaknesses.

But, Drupal powers over a million websites!

One of the key statistics that Drupalers hold onto is that it's powered over a million websites since mid 2014 when Drupal 7 was in ascendance. However, since Drupal 8 was released in late 2015, Drupal's overall use has stalled at around 1.2m websites, as seen circled in red on the Drupal Core usage statistics graph below.

The main reason for this stall in growth was that Drupal 8 was a major architectural re-write that wasn't essential or even affordable for many Drupal 7 sites to migrate to. For clients considering major new projects, many held off on committing to Drupal 8 until there were more successful case studies in the wild and didn't commission new Drupal 7 sites given that version was nearing a decade old. Anecdotally, 2016 was a tough year for many Drupal firms as they grappled with this pause in adoption.

Of course, Drupal 8 is now a well-proven platform and is experiencing steady uptake as circled in green on the usage graph above. This uptake corresponds with a down tick in Drupal 7 usage, but also indicates a softening of total Drupal usage. If we extrapolate these trend lines in a linear fashion, then we can see that Drupal 8 might surpass Drupal 7 usage around 2023.

Of course, technology adoption doesn't move in a straight line! Disruptive technologies emerge that rapidly change the playing field in a way that often can't be envisaged. The example that springs to mind is Nokia's market share was still growing when the iPhone 4 was released in 2010. By the time the iPhone 4s was released in 2011, Nokia's sales volumes had almost halved, leading to Microsoft's catastrophic purchase of the handset division in 2013 and subsequent re-sale for 5% of the purchase value in 2016. Oops!

Despite this downward trend in overall Drupal usage, we can take comfort that its use on larger scale sites is growing, powering 5.7% of the Top 10,000 websites according to Builtwith.com. However, its market share of the Top 100,000 (4.3%) and Top Million (3%) websites is waning, indicating that other CMS are gaining ground with smaller sites. It's also worth noting that Builtwith only counts ~680,000 Drupal websites, indicating that the other ~500,000 Drupal.org is detecting are likely to be development and staging sites.

So, where are these other sites moving to when they're choosing a new CMS?

Looking at the stats from W3Techs, it's clear to see that Wordpress accounts for almost all of the CMS growth, now sitting at around 30% of total market share.

Wordpress has been able to achieve this dominance by being a fantastic CMS for novice developers and smaller web agencies to build clients' websites with. This is reinforced by Wordpress having an exceptional editor experience and a hugely popular SAAS platform at Wordpress.com.

Drupal's place in the CMS market

The challenge Wordpress poses to other open-source CMS platforms, like Joomla, Typo3 and Plone, all with under 1% market share and falling, is their development communities are likely to look direct their efforts to other platforms. Drupal is able to hedge against this threat by having a large and highly engaged community around Drupal 8, but it's now abundantly clear that Drupal can't compete as a platform for building smaller brochure-ware style sites that Wordpress and SAAS CMS like Squarespace are dominating. We're also seeing SAAS platforms like Nationbuilder eat significantly into Drupal's previously strong share of the non-profit sector.

With all the hype around Headless or Decoupled CMS, Drupal 8 is well positioned to play a role as the backend for React or Angular Javascript front-ends. Competitors in this space are SAAS platforms like Contentful and Directus, with proprietary platforms like Kentico pivoting as a native cloud CMS service designed to power decoupled front-ends.

We often talk of Drupal as a CMS Framework, where it competes against frameworks like Ruby on Rails, .NET and Django to build rich web based applications. Drupal 8 is still well placed to serve this sector if the web applications are also relying on large scale content and user management features.

Which brings us to the Enterprise CMS sector, where Drupal competes head to head with proprietary platforms like Adobe Experience Manager, Sitecore and legacy products from Opentext, IBM and Oracle. The good news is that Drupal holds its own in this sector and has gained very strong market share with Government, Higher Education, Media and "Challenger" Enterprise clients.

This "Comfort zone" for Drupal usage is characterised by clients building large scale platforms with huge volumes of content and users, high scalability and integration with myriad third party products. Operationally, these clients often have well established internal web teams and varying degrees of self reliance. They're often using Agile delivery methods and place high value on speed to market and the cost savings associated with open-source software.

Where Drupal is gaining a competitive edge since the release of Drupal 8 is against the large proprietary platforms like Adobe Experience Manager and Sitecore. These companies market a platform of complementary products in a unified stack to their clients through long standing partnerships with major global digital agencies and system integrators. It's no surprise then that Acquia markets their own platform in a similar way to this sector where Drupal serves as the CMS component, complemented by subscription-based tools for content personalisation, customer segmentation and cloud based managed hosting. Acquia have actively courted global digital media agencies with this offering through global partnerships to give Drupal a toe hold in this sector.

This has meant Acquia has made significant headway into larger Enterprise clients through efforts like being recognised as a "Leader" in the Gartner Magic Quadrant for CMS, lending Drupal itself some profile and legitimacy as a result. This has driven Enterprise CIOs, CTOs and CMOs to push their vendors to offer Drupal services, who have looked to smaller Drupal firms to provide expertise where required. This is beneficial to independent Drupal services firms in the short term, but the large digital agencies will quickly internalise these skills if they see a long term market for Drupal with their global clients.

As one of those independent Drupal firms, PreviousNext have staked a bet that not all Enterprise customers will want to move to a monolithic platform where all components are provided by a single vendor's products. We're seeing sophisticated customers wanting to use Drupal 8 as the unifying hub for a range of best-of-breed SAAS platforms and cloud services.

This approach means that Enterprise customers can take advantage of the latest, greatest SAAS platforms whilst retaining control and consistency of their core CMS. It also allows for a high degree of flexibility to rapidly adapt to market changes.

What does this all mean for Drupal 8?

The outcome of our research and analysis has led to a few key conclusions about what the future looks like for Drupal 8:

The "comfort zone" of Government, Media, Higher Education and "Challenger" Enterprise clients will grow as many of these clients upgrade or switch to Drupal 8 from Drupal 7 or proprietary platforms.

Drupal will gain traction in the larger Enterprise as the global digital agencies and system integrators adopt Drupal 8 as a direct alternative to proprietary CMS products.

Independent Drupal services firms have a good opportunity to capitalise on these trends through partnerships with larger global agencies and specialisation in technologies that complement Drupal 8 as a CMS.

A culture of code contribution needs to grow within the larger clients and agencies moving to Drupal to ensure the burden of maintaining Drupal's development isn't shouldered by smaller independent firms and individual developers.

Despite the fact that we've probably already passed "Peak Drupal", we're firm believers that Drupal 8 is the right tool for large scale clients and that community has the cohesion to adapt to these existential challenges!

Add new comment

At PNX, style guide driven development is our bag. It’s what we love: building a living document that provides awesome reference for all our front end components. And Drupal 8, with its use of Twig, complements this methodology perfectly. The ability to create a single component, and then embed that component and its markup throughout a Drupal site in a variety of different ways without having to use any tricks or hacks is a thing of beauty.

Create a component

For this example we are going to use the much loved collapsible/accordion element. It’s a good example of a rich component because it uses CSS, JS, and Twig to provide an element that’s going to be used everywhere throughout a website.

The .scss file will end up compiling to a .css file, but we will be using SASS here because it’s fun. The widget.js file is a jQuery UI Widget Factory plugin that gives us some niceties - like state. The drupal.js file is a wrapper that adds our accordion widget as a drupal.behavior. The svg file provides some pretty graphics, and finally the twig file is where the magic starts.

This is a standard-ish BEM based component. It uses a js-* class to attach the widget functionality. We also have a {{ modifier_class }} variable, that can be used by kss-node to alter the default appearance of the collapsible (more on this later). There are two elements in this component title and content. They are expressed inside a twig block. What this means is we can take this twig file and embed it elsewhere. Because the component is structured this way, when it’s rendered in its default state by KSS we will have some default content, and the ability to show it's different appearances/styles using modifier_class.

Our twig file also uses the custom Drupal attach_library function which will bring in our components CSS and JS from the following theme.libraries.yml entry:

This is a pretty meaty component so it’s got some hefty javascript requirements. Not a problem in the end as it’s all going to get minified and aggregated by Drupal Cores library system.

And there we have it - a rich javascript component. It’s the building block for all the cool stuff we are about to do.

Use it in a field template override

As it stands we can throw this component as-is into KSS which is nice (although we must add our css and js to KSS manually, attach_library() won’t help us there sadly - yet), but we want drupal to take advantage of our twig file. This is where twigs embed comes in. Embed in twig is a mixture of the often used include, and the occasionally used extend. It’s a super powerful piece of kit that lets us do all the things.

Well these things anyway: include our twig templates contents, add variables to it, and add HTML do it.

Because this is an accordion, it’s quite likely we’ll want some field data inside it. The simplest way to get this happening is with a clunky old field template override. As an example I’ll use field--body.html.twig:

Here you can see the crux of what we are trying to achieve. The collapsible markup is specified in one place only, and other templates can include that base markup and then insert the content they need to use in the twig blocks. The beauty of this is any time this field is rendered on the page, all the markup, css and js will be included with it, and it all lives in our components directory. No longer are meaty pieces of markup left inside Drupal template directories - our template overrides are now embedding much richer components.

There is a trick above though, and it’s the glue that brings this together. See how we have a namespace in the embed path - all drupal themes/modules get a twig namespace automatically which is just @your_module_name or @your_theme_name - however it points to the theme or modules templates directory only. Because we are doing style guide driven development and we have given so much thought to creating a rich self-contained component our twig template lives in our components directory instead, so we need to use a custom twig namespace to point there. To do that, we use John Albins Component Libraries module. It lets us add a few lines to our theme.info.yml file so our themes namespace can see our component templates:

component-libraries:
pnx_project_theme:
paths:
- src
- templates

Now anything in /src or /templates inside our theme can be included with our namespace from any twig template in Drupal.

Use it in a field formatter

Now let’s get real because field template overrides are not the right way to do things. We were talking about making things DRY weren’t we?

Enter field formatters. At the simple end of this spectrum our formatter needs an accompanying hook_theme entry so the formatter can render to a twig template. We will need a module to give the field formatter somewhere to live.

Drupal magic is going to look for templates/collapsible-formatter.html.twig in our module directory automatically now. Our hook_theme template is going to end up looking pretty similar to our field template:

Now jump into the field display config of a text_long field, and you’ll be able to select the collapsible and it’s going to render our component markup combined with the field data perfectly, whilst attaching necessary CSS/JS.

Add settings to the field formatter

Let's take it a bit further. We are missing some configurability here. Our component has a modifier_class with a mini style (a cut down smaller version of the full accordion). You'll notice in the twig example above, we are using the with notation which works the same way for embed as it does for include to allow us to send an array of variables through to the parent template. In addition our hook_theme function has a style variable it can send through from the field formatter. Using field formatter settings we can make our field formatter far more useful to the site builders that are going to use it. Let's look at the full field formatter class after we add settings:

There's a few niceties there: It allows us to set a custom label (for the whole field), it automatically assigns the correct modifier_class, it links to the correct section in the style guide in the settings field description, and it adds a settings summary so site builders can see the current settings at a glance. These are all patterns you should repeat.

Let's sum up

We've created a rich interactive BEM component with its own template. The component has multiple styles and displays an interactive demo of itself using kss-node. We've combined its assets into a Drupal library and made the template - which lives inside the style guides component src folder - accessible to all of Drupal via the Component Libraries module. We've built a field formatter that allows us to configure the components appearance/style. Without having to replicate any HTML anywhere.

The component directory itself within the style guide will always be the canonical source for every version of the component that is rendered around our site.

Add new comment

Its extremely important to have default values that you can rely on for local Drupal development, one of those is "localhost". In this blog post we will explore what is required to make our local development environment appear as "localhost".

In our journey migrating to Docker for local dev we found ourselves running into issues with "discovery" of services eg. Solr/Mysql/Memcache.

In our first iteration we used linking, allowing our services to talk to each other, some downsides to this were:

Tricky to compose an advanced relationship, lets use PHP and PanthomJS as an example:

PHP needs to know where PhantomJS is running

PhantomJS needs to know the domain of the site that you are running locally

Wouldn't it be great if we could just use "localhost" for both of these configurations?

DNS entries only available within the containers themselves, cannot run utilities outside of the containers eg. Mysql admin tool

With this in mind, we hatched an idea.....

What if we could just use "localhost" for all interactions between all the containers.

If we wanted to access our local projects Apache, http://localhost (inside and outside of container)

If we wanted to access our local projects Mailhog, http://localhost:8025 (inside and outside of container)

Learn more

If you're interested in learning more about Chatbots and conversational UI with Drupal, I'm presenting a session on these topics at Drupal South 2017, the Southern Hemisphere's biggest Drupal Camp. October 31st is the deadline for getting your tickets at standard prices, so if you plan to attend, be sure to get yours this week to avoid the price hike.

Add new comment

In this week's Lightning talk, I go through a case study on an investigation into Deadlocks and Render caching and why cache contexts are so important to get right. Check out the video below to find out how we were able to withstand 10x the throughput with smarter caching.

Add new comment

Drupal 8.4 is stable! With 8.3 coming to end of life, it's important to update your projects to the latest and greatest. This blog will guide you through upgrading from Drupal core 8.3 to 8.4 while avoiding those nasty and confusing composer dependency errors.

The main issues with the upgrade to Drupal core 8.3 are dependency conflicts between Drush and Drupal core. The main conflict being that both Drush 8.1.x and Drupal 8.3 use the 2.x version of Symfony libraries, while Drupal 8.4 has been updated to use Symfony 3.x. This means that when using composer to update Drupal core alone, composer will complain about conflicts in dependencies, since Drush depends on Symfony 2.x

Updating your libraries

Note: If you are using Drush 8.1.15 you will not have these issues as it is now compatible with both Symfony 2.x and 3.x

However, if you are using Drush < 8.1.15 (which a lot of people will be on), running the following command will give you a dependency conflict:

composer update drupal/core --with-dependencies

Resulting in an error message, followed by a composer trace:

Your requirements could not be resolved to an installable set of packages.

The best way to fix this is to update both Drupal core and Drush at the same time. Drush 8.x is not compatible with Drupal 8.4 so you will need to update to Drush 9.x.

Some people have reported success with simply running a require on both updated versions of Drupal and Drush at the same time, but this did not work for me

composer require "drupal/core:~8.4" "drush/drush:~9.0"

What next?

Great, you're on the latest versions of both core and drush, but what's next? Well, that depends on a lot of things like what contributed and custom modules your project is running, how you're deploying your site, and what automated tests you are running. As I can't possibly cover all bases, I'll go through the main issues we encountered.

Next, you'll want to ensure that all of your deployment tools are still working. Here at PreviousNext our CI/CD tools call Make commands which are essentially just wrappers around one or more Drush commands.

For the most part, the core Drush commands (that is, the commands that ship with drush) continued working as expected, with a couple of small caveats:

1. You can no longer pipe a SQL dump into the drush sql-cli (sqlc) command.

Note: As of Drush 9.0-beta7 this has now been fixed, meaning the old version will work again!

2. The drush --root option no longer works with relative paths

Previously, our make commands all ran Drush with the --root (or -r) option relative to the repository root:./bin/drush -r ./app some-commandNow it must be an absolute path, or Drush will complain about not being able to find the Drupal settings:./bin/drush -r /path/to/app some-command

3. Custom Drush commands

For custom Drush commands, you will need to port them to use the new object oriented style approach and put the command into a dedicated module. Since version 9.0-beta5, Drush has dropped support for the old drush.inc style approach that could be used to add commands to a site without adding a new module.

For an example on this, take a look at our drush_cmi_tools library which provides some great extensions for importing and exporting config. This PR shows how we ported these commands to the new Drush 9 format.

Other gotchas

Following the Drush upgrades, your project will need various other updates based on the modules and libraries it uses. I'll detail some issues I faced when updating the Transport for NSW site below.

1. Stale bundles in the bundle field map key value collection

Added as part of this issue, views now throws warnings similar to "A non-existent config entity name returned by FieldStorageConfigInterface::getBundles(): field name: field_dates, bundle: page" for fields that are in the entity bundle field field map that no longer exist on the site. We had a handful of these fields which threw warnings on every cache clear. To fix this, simply add an update hook which clears out these stale fields from the entity.definitions.bundle_field_map keyvalue collection:

The menu_link_content module now has an entity_predelete hook that looks through an entities uri relationships and tries to find any menu links that link to that specific route, and if so deletes them. When the uri is external, an error is thrown when it tries to get the route name "External URLs do not have an internal route name.". See this issue for more information.

3. Tests that submit a modal dialog window will need to be altered

This is a very edge case issue, but will hopefully help someone! In older versions of jQuery UI, the buttons that were added to the bottom of the modal form for submission had an inner span tag which could be clicked as part of a test. For example, in Linkit's LinkitDialogTest. This span no longer exists, and attempting to "click" any other part of that button in a similar way will throw an error in PhantomJS. To get around that simply change your test to do something similar to the following:

Conclusion

Personally, I found the upgrade to be quite tedious for a minor version upgrade. Thankfully, our project has a large suite of functional/end-to-end tests which really helped tease out the issues and gave us greater confidence that the site was still functioning well post-upgrade. Let me know in the comments what issues you're facing!

Add new comment

Last week I was fortunate enough to attend and deliver a session at DrupalCon Vienna. The session was based around leveraging and getting productive with the automated testing tools we use in the Drupal community.

For the kind of large scale projects we work on, it's essential that automated testing is a priority and firmly embedded in our technical culture. Stability and maintainability of the code we're working on helps to build trusting relationships and happy technical teams. I have for a long time been engaged with the developments of automated testing in Drupal core and internally we've worked hard to adapt these processes into the projects we build and fill-in any blanks where required.

I was fortunate enough to be selected to share this at DrupalCon Vienna. Without further ado, I present, Test all the things! Get productive with automated testing in Drupal 8:

[embedded content]

Our current testing ethos is based around using the same tools for core and contrib for our bespoke Drupal project builds. Doing so allows us to context-switch between our own client work and contributed project or core work. To make this work we've addressed a few gaps in what's available to us out of the box.

Current State of Testing

I had some great conversations after the session with developers who were just starting to explore automated testing in Drupal. While the tools at our disposal are powerful, there is still lots of Drupal-specific knowledge required to become productive. My hope is the session helped to fill in some of the blanks in this regard.

E2E Testing

Because all of the test cases in core are isolated and individually setup environments/installations, end-to-end testing is tricky without some additional work. One of the touch points in the session was based around skipping the traditional set-up processes and using the existing test classes against pre-provisioned environments. Doing so replicates production-like environments in a test suite, which helps to provide a high-level of confidence tests are asserting behaviors of the whole system. Bringing this into core as a native capability is being discussed on drupal.org and was touched on in the session.

JS Unit Testing

One thing Drupal core has yet to address is JavaScript unit testing. For complex front-ends, testing JS application code with a browser is can become clumsy and hard to maintain. One approach we've used to address this is Jest. This nicely compliments front-ends where individual JavaScript modules can be isolated and individually tested.

Summing up, attending DrupalCon Vienna, presenting the session and meeting the members of the broader community was a great experience. I'm hopeful my session was able to contribute to the outstanding quality of sessions and technical discussions.

Symfony 3.x

If your project interacts Symfony directly at the lower level (rather than using Drupal core APIs that in turn use Symfony), you should be sure to review your code to make sure you're not using any of the APIs impacted by the BC breaks between 2.x and 3.x. Hopefully, your automated testing will reveal these regressions for you (you have automated testing right?). See the Symfony change list for the details of BC breaks.

One thing to note with the Symfony update is that whilst core dependencies were updated, your project may rely on other third-party PHP libraries that have dependencies on Symfony 2.x components. This may cause you issues with your update - and require you to update other dependencies at the same time - including drush - so testing sooner rather than later is recommended. If you find you're having issues with composer dependencies, we have another blog post dedicated to debugging them.

A recommended approach to auditing and tackling this is to add the jQuery migrate plugin to your project, and begin testing whilst watching the JavaScript console to detect deprecation notices thrown from the plugin.

So what's going on here?

We're using the Chatbot API module in conjunction with the API AI webook module to respond to intents. We're using API.ai for the natural language parsing and machine learning. And we're using the new Chatbot API entities sub module to push our Drupal entities to API.ai so it is able to identify Drupal entities in its language parsing.

A handful of custom Chatbot API intent plugin to wire up the webhook responses and that's it - as we create content, users and terms on our site - our chatbot automatically knows how to surface them. As we monitor the converstions in the API.ai training area, we can expand on our synonyms and suggestions to increase our matching rates.

So let's consider our team member Eric Goodwin. If I ask the chatbot about Eric, at first it doesn't recognise my question.

Eric isn't recognized as an entity

So I edit Eric's user account and add some synonyms

Adding synonyms to Eric's account

And then after running cron - I can see these show up in API.ai

Synonyms now available in API.ai

So I then ask the bot again 'Who is eric?'

Screenshot showing the default response

But again, nothing shows up. Now I recognise the response 'Sorry, can you say that again' as what our JavaScript shows if the response is empty. But just to be sure - I check the API.ai console to see that it parsed Eric as a staff member.

Intent is matched as Bio and Eric is identified as staff member

So I can see that the Bio Intent was matched and that Eric was correctly identifed as the Staff entity. So why was the response empty? Because I need to complete Eric's bio in his user account. So let's add some text (apologies Eric you can refine this later).

Adding a biography

Now I ask the bot again (note I've not reloaded or anything, this is all in real time).

A working response!

And just like that, the bot can answer questions about Eric.

What's next?

Well API.ai provides integrations with Google Assistant and Facebook messenger, so we plan to roll out those too. In our early testing we can use this to power an Actions on Google app with the flick of a switch in API.ai. Our next step is to expand on the intents to provide rich content tailored to those platforms instead of just plain-text that is required for chatbot and voice responses.

Credits

Thanks go to @gambry for the Chatbot API module and for being open to the feature addition to allow Drupal to push entities to the remote services.

And credit to the amazing Rikki Bochow for building the JavaScript and front-end components to incorporate this into our site so seamlessly.

Add new comment

We're starting up our Lightning talks again during our weekly developer meetings here at PreviousNext. This week was about wiring up a straight forward plugin.js and extending CKEditorPluginBase to create a custom CKEditor widget in Drupal 8.