Survey of screenshot-based CSS testing tools

css_regression_testing.md

Currently considering https://github.com/webdriverio/webdrivercss

---

Core Goals:

• Can test in up-to-date versions of all major browsers
• Can test on up-to-date versions of all major OSes
• Can test in IE9 (because Bootstrap v4 will support IE9+)
• Don't want to have to setup/maintain our own cluster of VMs running all the necessary OSes (and all the versions of Windows)
• Workflow for management of reference/baseline/norm screenshots

Nice-to-haves:

• Any necessary third-party services being low-cost or free, since this is for an open-source project (namely, Bootstrap)
• Ability to adjust viewport/screen size for testing of "responsive" features
• GitHub integration

Non-requirements:

• Testing interactions with JavaScript widgets
  • A nice-to-have, but not worth it if it overcomplicates things or requires compromises on the main Requirements

---

Prior art (and boy, is there a lot of it!)

• Mogotest
  • Went out of business
• SiteEffect
  • No public source code is yet available; GitHub repo is a stub
• Bugsnag's unnamed Selenium-based tool
  • Not currently publicly available at all
• Huxley
  • Has been abandoned by Facebook; is no longer maintained
  • Was geared towards testing interactions with JavaScript widgets, as opposed to testing of static webpages
• Hardy
  • Offers "CSS property X of element Y has value Z" testing, rather than screenshot comparison
  • Writing/updating this type of test seems prohibitively tedious
• DalekJS
  • Seems geared more towards functional testing
  • Doesn't seem to have any integrated workflow for management of reference/baseline/norm screenshots
• CSS Critic
  • Supports Firefox and PhantomJS
  • Has a review workflow
• Needle
  • Somewhat promising. Offers unit tests with screenshot equality assertions.
  • Supports remote Selenium.
  • Supports setting viewport size.
  • Has some tooling around setting reference/baseline/norm screenshots
    • Not clear whether it supports updating a single reference screenshot at a time, rather than everything at once.
    • Reference/baseline/norm screenshot update workflow doesn't seem quite suited to Bootstrap's particular situation
  • Visual diffs require Perceptual Image Diff tool whose maintenance status is unclear
• Browsershots
  • Uses virtual cluster of volunteers' computers to take screenshots in various browsers/OSes
  • Throughput/capacity level is unclear, and we'd be asking for a *lot* of screenshots
  • No workflow or higher-level tooling, but it exposes an API which might allow building such workflow tooling
• CrossBrowserTesting.com
  • Hosted solution with integrated review workflow
  • Pay per minute of test
• Depicted / dpxdt
  • PhantomJS-based, but also offers "Diff My Images" option to import images taken from other tools
  • Offers web-based test result viewer and reference/baseline/norm management workflow
  • No explicit cross-browser support in its data model
• Applitools Eyes
  • Smart visual diff algorithm
  • You provide it the images yourself
• WebdriverCSS
  • Supports Selenium
  • Supports setting viewport dimensions
  • WebdriverCSS Adminpanel
    • Web UI to view test results and manage reference/baseline/norm screenshots
• Gemini
  • Supports Selenium
  • Offers experimental CSS code coverage feature
  • Gemini GUI
    • Web UI to view test results (not sure if it allows updating of norms)
• Wraith-Selenium
  • A Wraith fork that supports Selenium instead of CasperJS
• Screenbeacon
  • Paid service
  • Has GitHub integration
  • Supports responsive testing
  • No cross-browser support. Doesn't even say which browser they're using. Probably Phantom or Chrome.
• Percy
  • Paid service
  • Currently Firefox-only
  • Currently no responsive testing support
  • They plan to implement these in the future
• Gatling
  • Supports Selenium
  • Supports cross-browser testing
• Shoov
  • Supports Selenium
  • Supports cross-browser testing
  • Offers a web UI (in PHP...)
  • Not sure whether this supports responsive testing
• PhantomJS-related tools
  • Included for completeness, but these fail our cross-browser + cross-OS requirements
  • CasperJS
    • Wrapper for PhantomJS & SlimerJS
    • No higher level tooling around screenshots, just a "take a screenshot" API
  • SlimerJS
    • Firefox/Gecko equivalent of PhantomJS
  • TrifleJS
    • Internet Explorer equivalent of PhantomJS
  • PhantomCSS
    • CasperJS-based CSS regression testing
  • Wraith
    • CasperJS-based CSS regression testing
  • BackstopJS
    • CasperJS-based CSS regression testing of "responsive" webpages
    • Includes web app to review test results (and manage reference/baseline/norm screenshots?)
  • grunt-photobox
    • PhantomJS-based screenshot-taker and web-based visual diff review tool

Web UI to view test results (not sure if it allows updating of norms)

It does.

Currently considering https://github.com/webdriverio/webdrivercss

Why?

Gemini is a way more better and robust. It has slim API for writing test. It implemented CIEDE2000 color comparison algorithm. It manages caret (input fields) difference for you and many other things. It is open for contributions also.

Gemini is already used in production at Yandex by the team of over 70 front-end developers to test UI components library, yandex.ru/search, yandex.ru/images, yandex.ru/video and many other web services for more that 2 years.

Gemini is not a prior art, it appeared nearly at the same time as Webdriver.io and WebdriverCSS.

PS: I'm leading the team who are developing Gemini and Gemini GUI, so I'm biased ;)

Hi, if the question is still relevant, I would suggest trying Screenster, a visual testing tool. It is able to perform CSS testing. Does not require coding to record tests. You can check it out at free Demo server and decide if it is good enough for your specific case.

What do you guys think about lineup

Howdy @cvrebert - Percy supports responsive testing now, at any widths you specify. It also has CI and GitHub integration, and discounts for open source projects.

Happo is a screenshot testing tool focused on unit testing (e.g. individual React components). Mostly meant for use in a CI, posts to your code review tool when it detects a diff.

Galen: https://github.com/galenframework/galen
Tests elements size and relative position to each other
Uses Selenium

WebdriverCSS seems to be obsolete and abandoned in some way
https://github.com/webdriverio/webdrivercss

**Note: WebdriverCSS isn't yet compatible with WebdriverIO v3.0 and is not longer maintained. ** If anyone wants to take ownershipt of this project I am happy to assign push permissions.

Awesome list of Visual Regression Testing

http://arguseyes.io

does a good job for me for static and RESTfull sites

to make screenshot of dynamic sites/webapps, I use puppeteer since yesterday, to patch this missing feature
I just capture the component state after a click event with puppeteer to the same folder as argus-eyes uses and take the argus-eyes compare then as normal

https://www.browserstack.com/support?tag=screenshots isn't free, but will take screenshots across browsers (including mobile). They have a screenshot testing API, or you can use Selenium.

I'm surprised no one has listed the open source imagemagick. I did some experiments with it; was easy to use and had some useful sophisticated comparison capabilities. If you need to "roll your own" image capture, I'd look at this.

Differencify is a UI visual regression testing tool built on top of Puppeteer

Really useful list, thank you for putting this together :)

Just want to note here that Wraith now supports using headless Chrome

Percy now supports Chrome as well.

Take a look at Axcept

Hello,
unfortunately siteeffect.io is no longer available.

Together with a colleague I am working on Warhol, which analyses Pattern Libraries and uses the data to compare it against the implementation.
Warhol does not use Screenshots but runs in a real browser which makes its error reporting faster and much more reliable than a lot of other tools which exist.

Take a look at https://diffy.website. We use puppeteer on AWS Lambda (very performant) and also have custom images comparison algorithm that produce way less false positives than standard imagemagick.

I'm looking at lightweight libraries that would interface with Selenium (not come with Puppeteer or Selenium or anything), just accept screenshots and provide a dashboard. https://github.com/reg-viz/reg-cli and https://github.com/reg-viz/reg-suit look good for accomplishing that.

@Christopher-Hayes

I'm looking at lightweight libraries that would interface with Selenium (not come with Puppeteer or Selenium or anything), just accept screenshots and provide a dashboard. https://github.com/reg-viz/reg-cli and https://github.com/reg-viz/reg-suit look good for accomplishing that.

https://screenshotbot.io is just that. We don't try to solve the problem of generating screenshots, or running them. There are many fantastic libraries and CI systems to do that just fine. We just have a simple dashboard that integrates with your Jira/Asana/Github Issues, and will work with Android, iOS, Web, or for whichever platform where there's already a screenshot testing library. I built the screenshot testing service at Facebook, so I have pretty good experience running something like this at scale, and understand what it takes to keep developers happy (or annoyed).