Recently we’ve been looking at how we can improve aspects of the website support service that we provide for our clients.

A significant part of the support service is the maintenance of sites. Specifically, ensuring that all sites stay up to date with the correct core versions of Drupal and that their modules (or plugins, on Wordpress sites) are all upgraded and functioning properly too. This keeps a site working as expected, and keeps it safe and secure.

Up until now, the process has been manual. That means that we need to create a new environment, a specialist Drupal developer has to complete the upgrades, and then pass it back to Support to test. We (Support) look at it with our actual eyeballs and decide if everything seems to be looking good (i.e. like the live site) before it’s deployed along a line of other test environments. 

Testing website updates

Each of these requires testing of some sort as well, whether it’s visual and login checks or more in-depth functionality testing, depending on the scope of the update. When things break, they need to be fixed by a developer and then tested again. The process takes a while and often passes back and forth before staging and a final step to live.

We started looking into ways of making the initial steps more efficient and whether or not we could automate some of it. There are plenty of commercial services that provide this, but investing in those would put up our support prices, so we've started to work on some simple automations that save us time. 

Visual regression testing

That’s where visual regression testing (VRT) comes in. This is a method of testing that allows you to compare two versions of a site and identify visual changes in a quick and simple way. This is perfect for site security updates, which could minutely impact any page on the site. 

We’re using Backstop.js, an open source javascript framework built for this type of testing. Backstop.js allows you to view two screenshots of an application side-by-side, highlights any differences in bright colours, and you can even drag a bar from one side of the screen to the other to progressively view any changes.

You can view how a site looks on mobile and desktop using Backstop.js, too. While some failures flagged are false negatives – whether that’s down to a Twitter feed updating since we took our reference copy or carousels changing slides – it’s still an efficient method. In about eight minutes we can check 200 pages and confirm there’s nothing out of place; or discover that a component only used in a single place on the site has broken.

Soon our visual regression testing will be built into a script we run from the command line that deploys code changes. This allows us to focus our time on those things that need manual checks.

Embedding and automating process improvements

So far, our experience with visual regression testing has been positive. We’ve saved time, caught issues we might have missed previously, and it’s allowed us to focus our testing on functional stuff rather than visual changes. Setting things up and getting it to the point where it’s fit into our process seamlessly has taken a bit of time, but the investment seems worth it. 

We hope to be able to build this VRT in with other automated ways of testing, the first step of which will be bringing widespread automation to our security update process. Then we can start thinking about how we can make VRT a seamless part of our projects as well, so that we can expand the knowledge throughout the team, without simply limiting it to Support.