CivicTheme — setting up automated visual regression testing

Overview

CivicTheme’s challenge

CivicTheme has regular releases and was relying on manual visual regression testing to identify any defects. However, due to the large number of components and their variants, this process took an exorbitant amount of time.

CivicTheme’s transformation

CivicTheme now automatically generates pages with the different component variations whenever a new version of CivicTheme is released. Diffy takes screenshots and compares them to earlier versions of the same component to find any visual differences, and then a QA engineer reviews the report.

The outcomes

• Rigorous visual regression testing to catch any defects
• A much faster testing process through automation
• Significant cost-savings

Detailed case study

Below is detailed information on the challenge, transformation and final outcomes/benefits.

CivicTheme’s challenge — frequent release cycles

As a growing design system and Drupual theme, CivicTheme has regular releases that include both minor changes, and feature enhancements. The original process in place for visual regression testing was manual: a QA tester would need to manually create a variety of pages with all the components on them, and then manually compare them to previous versions. Obviously this is a large investment in terms of time, resources, and therefore money. The CivicTheme team wanted to automate the visual regression testing.

CivicTheme’s transformation — automated visual regression testing

The CivicTheme team started with an open source Drupal module called Generated Content. This module automatically generates pages with all of the different component variations whenever a new version of CivicTheme is released. See the screenshot above for an example or go to the generated component pages.

The automatically generated pages are then run through Diffy, which takes screenshots and compares them to earlier versions of the same component to find any visual differences. Then, a QA engineer can go through the Diffy list and assess if the differences are intentional (e.g. component updates) or if a regression defect was introduced.

The outcomes — a robust release process

• Rigorous visual regression testing to catch any defects
• A much faster testing process through automation
• Significant cost-savings for testing
• A robust, solid support base for CivicTheme releases
• A way for developers, content editors and site owners to see how different components look (in addition to Storybook)

About CivicTheme

CivicTheme is a design system, component library and Drupal theme. CivicTheme was created so organisations can build Drupal websites faster and at a lower cost. Salsa maintains the open source project, which can be used by any vendor or organisation to build their own Drupal website.