A testing system for catching visual regressions in Web applications.

Related tags

Testinghuxley
Overview

Huxley

Watches you browse, takes screenshots, tells you when they change

Huxley is a test-like system for catching visual regressions in Web applications. It was built by Pete Hunt with input from Maykel Loomans at Instagram.

Archived Repo

This is an archived project and is no longer supported or updated by Facebook or Instagram. Please do not file issues or pull-requests against this repo. If you wish to continue to develop this code yourself, we recommend you fork it.

What is the problem?

  • UI tests are hard to write and are usually fragile.
  • Automated testing can't tell you if something doesn't look right, so UI regressions may go undetected.
  • It can be difficult for designers to participate in the code review process even though reviewing the way the UI looks is just as important as reviewing the code that creates it.

How does Huxley help me?

Huxley runs in two modes:

Record mode

Using Selenium WebDriver, Huxley opens a page and records your actions. When you press enter in the Huxley terminal, Huxley will save a screenshot.

Testing a new flow is as simple as manually testing it once. Huxley will remember and re-run your "manual" test plan for you automatically.

Playback mode

You should run Huxley in playback mode before submitting code for review and in continuous integration. Huxley will open the page and re-run your actions with WebDriver. It will take screenshots and compare them with the original screenshots. If they have changed, it will save the new screenshots and warn you that they have changed.

When screenshots have changed, those screenshot changes will show up in your commit. A designer can review them to be sure they're OK. And your continuous integration system can alert you if you forgot to run Huxley.

By default, Huxley will overwrite the old screenshots with new ones. That means you don't have to rewrite anything when your UI changes like you would with a traditional WebDriver test -- Huxley will just take a new screenshot for you and when it's checked in your test is updated!

Installation

pip install huxley

Tutorial

In examples/ you'll find two simple completed Huxley tests. To start from scratch, simply remove toggle.huxley, type.huxley and Huxleyfile.

Motivation

In examples/webroot/toggle.html you'll find a very simple JavaScript application that implements a toggle button. The goal of Huxley is to make creating an integration for this component effortless, and to make it easy to update the test when the UI changes.

Step 1: host your app somewhere

For our example, simply cd to examples/webroot and run python -m SimpleHTTPServer to start a basic server for our demo. In your app you may need to start up whatever framework you're using.

Step 2: create a Huxleyfile

A Huxleyfile describes your test. Create one that looks like this:

[toggle]
url=http://localhost:8000/toggle.html

This creates a test named toggle that tests the URL http://localhost:8000/toggle.html.

Step 2: record the test

Huxley makes writing tests easy because it simply records your browser session -- specifically mouse clicks and key presses on a single page -- and can replay them in an automated way. To do this you need to install Selenium Server and start it. It's as easy as java -jar selenium-server-standalone-XXX.jar.

Then, run Huxley in record mode: huxley --record. Huxley will bring up a browser using Selenium. Press enter in the Huxley console to take a screen shot of the initial page load. Then toggle the button in the browser a few times. After every click, switch back to the Huxley console to take a screen shot. When you've tested all of the functionality you want to test, simply type q and then enter in the Huxley console to exit.

After confirming, Huxley will automatically record the test for you and save it to disk as toggle.huxley. Be sure to commit the Huxleyfile as well as toggle.huxley into your repository so you can track changes to them.

Step 3: playback

Simply run the huxley command in the same directory as the Huxleyfile to be sure that your app still works.

Step 4: update the test with new screen shots

You'll likely update the UI of the component a lot without changing its core functionality. Huxley can take new screen shots for you when this happens. Tweak the UI of the component in toggle.html somehow (maybe change the button color or something) and re-run huxley. It will warn you that the UI has changed and will automatically write new screen shots for you. If you run huxley again, the test will pass since the screen shots were updated.

The best part is, since the screen shots are checked into the repository, you can review the changes to the UI as part of the code review process if you'd like. At Instagram we have frontend engineers reviewing the JavaScript and designers reviewing the screenshots to ensure that they're pixel perfect.

Step 5: run in CI mode

If you're using a continuous integration solution like Jenkins you probably don't want to automatically rerecord screen shots on failure. Simply run huxley --playback-only to do this.

Additionally, you may find that you're dissatisfied with Huxley replaying your browsing session in real-time. You can speed it up (or slow it down) by editing your Huxleyfile to read:

[toggle]
url=http://localhost:8000/toggle.html
sleepfactor=0.5

This edit should cut the execution time in half.

Best practices

Integration tests sometimes get a bad rap for testing too much at once. We've found that if you use integration tests correctly they can be just as effective and accurate as unit tests. Simply follow a few best practices:

  • Don't test a live app. Use mocking to make your components reliable instead. If you hit your live app, failures in any number of places could trigger false failures in your UI tests. Instead of hitting a real URL in your app, create a dedicated test URL for Huxley to hit that uses mocking (and perhaps dependency injection) to isolate your UI component as much as possible. Huxley is completely unopinionated; use whatever tools you want to do this.
  • Test a small unit of functionality. You should try to isolate your UI into modular components and test each one individually. Additionally, try to test one interaction per Huxley test so that when it fails, it's easy to tell exactly which interaction is problematic and it's faster to re-run it.

Technical FAQ

Why does Huxley stop recording when I navigate away from the page?

Huxley is designed for testing JavaScript UI components at this time. We've found that you can test multiple pages by creating a new Huxley test for each URL. This is valuable even if you don't use the interactive features of Huxley because it will ensure your static pages stay pixel perfect.

I can't tell what changed!

It's usually best if you use an image comparison tool like Kaleidoscope to tell what changed. But Huxley includes a simple image diff tool; simply run huxley with the --save-diff option to output a diff.png which will show you the pixels that changed.

How do I use a remote webdriver server?

You can set the HUXLEY_WEBDRIVER_LOCAL environment variable to tell Huxley which webdriver URL to use for --record mode. You can set the HUXLEY_WEBDRIVER_REMOTE environment variable to tell Huxley which webdriver URL to use for screenshots and playback. Usually you only need to use this when working in a team setting such that everyone's screenshots are taken on the same machine configuration (otherwise they'll change depending on who ran them last).

Can I test responsive design?

Of course! Simply add a screensize setting to your Huxleyfile. The default is screensize=1024x768.

Philosophical FAQ

Why would you use this instead of unit testing?

First of all, if you sufficiently componentize your UI, Huxley can be used as a unit testing tool.

With that said, unit tests have two shortcomings today.

  • They usually take a long time to write. Instagram on the web had a single engineer and a designer working on a ton of things in parallel, and we didn't have time to write beautifully isolated tests with elegant dependency injection and comprehensive assertions. We just had to make sure that we didn't cause bugs when we were frantically shipping code. A lot of small web teams can probably identify with this.
  • They do not test the look of the UI. Huxley does pixel-by-pixel comparisons of the UI. Traditional UI test systems inspect the DOM but do not look at how it actually renders. We once had a bug where a CSS rule made the height of all image components 0px; without a pixel-by-pixel comparison it's unlikely we would have ever written an explicit test for this.

What's the best way to use Huxley?

Use it however you want! But we generally shell out to it from within an existing test runner (i.e. Django or Rails). This lets us programmatically start a test server for Huxley to hit.

If you're using Python, you can use Huxley directly in a test (see huxley.integration.HuxleyTestCase) or browse the source to see its core APIs.

If you're on a team I recommend setting up webdriver on a shared server and changing the HUXLEY_WEBDRIVER_REMOTE environment variable such that everyone's screenshots are pixel perfect (see the technical FAQ above).

Why is it called Huxley?

Lots of test frameworks and methodologies are very opinionated about how your code should be structured or how you should write tests. Some tools are so opinionated that they're almost religious about their view of testing! We wanted a tool that got out of our way and let us fight regressions as quickly and easily as possible without being opinionated about it. So we named it after the guy who coined the term "agnostic", Thomas Henry Huxley.

Owner
Facebook Archive
These projects have been archived and are generally unsupported, but are still available to view and use
Facebook Archive
API Test Automation with Requests and Pytest

api-testing-requests-pytest Install Make sure you have Python 3 installed on your machine. Then: 1.Install pipenv sudo apt-get install pipenv 2.Go to

Sulaiman Haque 2 Nov 21, 2021
A library for generating fake data and populating database tables.

Knockoff Factory A library for generating mock data and creating database fixtures that can be used for unit testing. Table of content Installation Ch

Nike Inc. 30 Sep 23, 2022
XSSearch - A comprehensive reflected XSS tool built on selenium framework in python

XSSearch A Comprehensive Reflected XSS Scanner XSSearch is a comprehensive refle

Sathyaprakash Sahoo 49 Oct 18, 2022
Plugin for generating HTML reports for pytest results

pytest-html pytest-html is a plugin for pytest that generates a HTML report for test results. Resources Documentation Release Notes Issue Tracker Code

pytest-dev 548 Dec 28, 2022
frwk_51pwn is an open-sourced remote vulnerability testing and proof-of-concept development framework

frwk_51pwn Legal Disclaimer Usage of frwk_51pwn for attacking targets without prior mutual consent is illegal. frwk_51pwn is for security testing purp

51pwn 4 Apr 24, 2022
A pytest plugin, that enables you to test your code that relies on a running PostgreSQL Database

This is a pytest plugin, that enables you to test your code that relies on a running PostgreSQL Database. It allows you to specify fixtures for PostgreSQL process and client.

Clearcode 252 Dec 21, 2022
Show, Edit and Tell: A Framework for Editing Image Captions, CVPR 2020

Show, Edit and Tell: A Framework for Editing Image Captions | arXiv This contains the source code for Show, Edit and Tell: A Framework for Editing Ima

Fawaz Sammani 76 Nov 25, 2022
pytest plugin for testing mypy types, stubs, and plugins

pytest plugin for testing mypy types, stubs, and plugins Installation This package is available on PyPI pip install pytest-mypy-plugins and conda-forg

TypedDjango 74 Dec 31, 2022
Automated Penetration Testing Framework

Automated Penetration Testing Framework

OWASP 2.1k Jan 01, 2023
A cross-platform GUI automation Python module for human beings. Used to programmatically control the mouse & keyboard.

PyAutoGUI PyAutoGUI is a cross-platform GUI automation Python module for human beings. Used to programmatically control the mouse & keyboard. pip inst

Al Sweigart 7.5k Dec 31, 2022
Mypy static type checker plugin for Pytest

pytest-mypy Mypy static type checker plugin for pytest Features Runs the mypy static type checker on your source files as part of your pytest test run

Dan Bader 218 Jan 03, 2023
Asyncio http mocking. Similar to the responses library used for 'requests'

aresponses an asyncio testing server for mocking external services Features Fast mocks using actual network connections allows mocking some types of n

93 Nov 16, 2022
Make Selenium work on Github Actions

Make Selenium work on Github Actions Scraping with BeautifulSoup on GitHub Actions is easy-peasy. But what about Selenium?? After you jump through som

Jonathan Soma 33 Dec 27, 2022
The Social-Engineer Toolkit (SET) repository from TrustedSec - All new versions of SET will be deployed here.

💼 The Social-Engineer Toolkit (SET) 💼 Copyright 2020 The Social-Engineer Toolkit (SET) Written by: David Kennedy (ReL1K) @HackingDave Company: Trust

trustedsec 8.4k Dec 31, 2022
A simple tool to test internet stability.

pingtest Description A personal project for testing internet stability, intended for use in Linux and Windows.

chris 0 Oct 17, 2021
An improbable web debugger through WebSockets

wdb - Web Debugger Description wdb is a full featured web debugger based on a client-server architecture. The wdb server which is responsible of manag

Kozea 1.6k Dec 09, 2022
A utility for mocking out the Python Requests library.

Responses A utility library for mocking out the requests Python library. Note Responses requires Python 2.7 or newer, and requests = 2.0 Installing p

Sentry 3.8k Jan 03, 2023
Donors data of Tamil Nadu Chief Ministers Relief Fund scrapped from https://ereceipt.tn.gov.in/cmprf/Interface/CMPRF/MonthWiseReport

Tamil Nadu Chief Minister's Relief Fund Donors Scrapped data from https://ereceipt.tn.gov.in/cmprf/Interface/CMPRF/MonthWiseReport Scrapper scrapper.p

Arunmozhi 5 May 18, 2021
A test fixtures replacement for Python

factory_boy factory_boy is a fixtures replacement based on thoughtbot's factory_bot. As a fixtures replacement tool, it aims to replace static, hard t

FactoryBoy project 3k Jan 05, 2023
A library to make concurrent selenium tests that automatically download and setup webdrivers

AutoParaSelenium A library to make parallel selenium tests that automatically download and setup webdrivers Usage Installation pip install autoparasel

Ronak Badhe 8 Mar 13, 2022