A testing system for catching visual regressions in Web applications.

Related tags

Testinghuxley
Overview

Huxley

Watches you browse, takes screenshots, tells you when they change

Huxley is a test-like system for catching visual regressions in Web applications. It was built by Pete Hunt with input from Maykel Loomans at Instagram.

Archived Repo

This is an archived project and is no longer supported or updated by Facebook or Instagram. Please do not file issues or pull-requests against this repo. If you wish to continue to develop this code yourself, we recommend you fork it.

What is the problem?

  • UI tests are hard to write and are usually fragile.
  • Automated testing can't tell you if something doesn't look right, so UI regressions may go undetected.
  • It can be difficult for designers to participate in the code review process even though reviewing the way the UI looks is just as important as reviewing the code that creates it.

How does Huxley help me?

Huxley runs in two modes:

Record mode

Using Selenium WebDriver, Huxley opens a page and records your actions. When you press enter in the Huxley terminal, Huxley will save a screenshot.

Testing a new flow is as simple as manually testing it once. Huxley will remember and re-run your "manual" test plan for you automatically.

Playback mode

You should run Huxley in playback mode before submitting code for review and in continuous integration. Huxley will open the page and re-run your actions with WebDriver. It will take screenshots and compare them with the original screenshots. If they have changed, it will save the new screenshots and warn you that they have changed.

When screenshots have changed, those screenshot changes will show up in your commit. A designer can review them to be sure they're OK. And your continuous integration system can alert you if you forgot to run Huxley.

By default, Huxley will overwrite the old screenshots with new ones. That means you don't have to rewrite anything when your UI changes like you would with a traditional WebDriver test -- Huxley will just take a new screenshot for you and when it's checked in your test is updated!

Installation

pip install huxley

Tutorial

In examples/ you'll find two simple completed Huxley tests. To start from scratch, simply remove toggle.huxley, type.huxley and Huxleyfile.

Motivation

In examples/webroot/toggle.html you'll find a very simple JavaScript application that implements a toggle button. The goal of Huxley is to make creating an integration for this component effortless, and to make it easy to update the test when the UI changes.

Step 1: host your app somewhere

For our example, simply cd to examples/webroot and run python -m SimpleHTTPServer to start a basic server for our demo. In your app you may need to start up whatever framework you're using.

Step 2: create a Huxleyfile

A Huxleyfile describes your test. Create one that looks like this:

[toggle]
url=http://localhost:8000/toggle.html

This creates a test named toggle that tests the URL http://localhost:8000/toggle.html.

Step 2: record the test

Huxley makes writing tests easy because it simply records your browser session -- specifically mouse clicks and key presses on a single page -- and can replay them in an automated way. To do this you need to install Selenium Server and start it. It's as easy as java -jar selenium-server-standalone-XXX.jar.

Then, run Huxley in record mode: huxley --record. Huxley will bring up a browser using Selenium. Press enter in the Huxley console to take a screen shot of the initial page load. Then toggle the button in the browser a few times. After every click, switch back to the Huxley console to take a screen shot. When you've tested all of the functionality you want to test, simply type q and then enter in the Huxley console to exit.

After confirming, Huxley will automatically record the test for you and save it to disk as toggle.huxley. Be sure to commit the Huxleyfile as well as toggle.huxley into your repository so you can track changes to them.

Step 3: playback

Simply run the huxley command in the same directory as the Huxleyfile to be sure that your app still works.

Step 4: update the test with new screen shots

You'll likely update the UI of the component a lot without changing its core functionality. Huxley can take new screen shots for you when this happens. Tweak the UI of the component in toggle.html somehow (maybe change the button color or something) and re-run huxley. It will warn you that the UI has changed and will automatically write new screen shots for you. If you run huxley again, the test will pass since the screen shots were updated.

The best part is, since the screen shots are checked into the repository, you can review the changes to the UI as part of the code review process if you'd like. At Instagram we have frontend engineers reviewing the JavaScript and designers reviewing the screenshots to ensure that they're pixel perfect.

Step 5: run in CI mode

If you're using a continuous integration solution like Jenkins you probably don't want to automatically rerecord screen shots on failure. Simply run huxley --playback-only to do this.

Additionally, you may find that you're dissatisfied with Huxley replaying your browsing session in real-time. You can speed it up (or slow it down) by editing your Huxleyfile to read:

[toggle]
url=http://localhost:8000/toggle.html
sleepfactor=0.5

This edit should cut the execution time in half.

Best practices

Integration tests sometimes get a bad rap for testing too much at once. We've found that if you use integration tests correctly they can be just as effective and accurate as unit tests. Simply follow a few best practices:

  • Don't test a live app. Use mocking to make your components reliable instead. If you hit your live app, failures in any number of places could trigger false failures in your UI tests. Instead of hitting a real URL in your app, create a dedicated test URL for Huxley to hit that uses mocking (and perhaps dependency injection) to isolate your UI component as much as possible. Huxley is completely unopinionated; use whatever tools you want to do this.
  • Test a small unit of functionality. You should try to isolate your UI into modular components and test each one individually. Additionally, try to test one interaction per Huxley test so that when it fails, it's easy to tell exactly which interaction is problematic and it's faster to re-run it.

Technical FAQ

Why does Huxley stop recording when I navigate away from the page?

Huxley is designed for testing JavaScript UI components at this time. We've found that you can test multiple pages by creating a new Huxley test for each URL. This is valuable even if you don't use the interactive features of Huxley because it will ensure your static pages stay pixel perfect.

I can't tell what changed!

It's usually best if you use an image comparison tool like Kaleidoscope to tell what changed. But Huxley includes a simple image diff tool; simply run huxley with the --save-diff option to output a diff.png which will show you the pixels that changed.

How do I use a remote webdriver server?

You can set the HUXLEY_WEBDRIVER_LOCAL environment variable to tell Huxley which webdriver URL to use for --record mode. You can set the HUXLEY_WEBDRIVER_REMOTE environment variable to tell Huxley which webdriver URL to use for screenshots and playback. Usually you only need to use this when working in a team setting such that everyone's screenshots are taken on the same machine configuration (otherwise they'll change depending on who ran them last).

Can I test responsive design?

Of course! Simply add a screensize setting to your Huxleyfile. The default is screensize=1024x768.

Philosophical FAQ

Why would you use this instead of unit testing?

First of all, if you sufficiently componentize your UI, Huxley can be used as a unit testing tool.

With that said, unit tests have two shortcomings today.

  • They usually take a long time to write. Instagram on the web had a single engineer and a designer working on a ton of things in parallel, and we didn't have time to write beautifully isolated tests with elegant dependency injection and comprehensive assertions. We just had to make sure that we didn't cause bugs when we were frantically shipping code. A lot of small web teams can probably identify with this.
  • They do not test the look of the UI. Huxley does pixel-by-pixel comparisons of the UI. Traditional UI test systems inspect the DOM but do not look at how it actually renders. We once had a bug where a CSS rule made the height of all image components 0px; without a pixel-by-pixel comparison it's unlikely we would have ever written an explicit test for this.

What's the best way to use Huxley?

Use it however you want! But we generally shell out to it from within an existing test runner (i.e. Django or Rails). This lets us programmatically start a test server for Huxley to hit.

If you're using Python, you can use Huxley directly in a test (see huxley.integration.HuxleyTestCase) or browse the source to see its core APIs.

If you're on a team I recommend setting up webdriver on a shared server and changing the HUXLEY_WEBDRIVER_REMOTE environment variable such that everyone's screenshots are pixel perfect (see the technical FAQ above).

Why is it called Huxley?

Lots of test frameworks and methodologies are very opinionated about how your code should be structured or how you should write tests. Some tools are so opinionated that they're almost religious about their view of testing! We wanted a tool that got out of our way and let us fight regressions as quickly and easily as possible without being opinionated about it. So we named it after the guy who coined the term "agnostic", Thomas Henry Huxley.

Owner
Facebook Archive
These projects have been archived and are generally unsupported, but are still available to view and use
Facebook Archive
A simple tool to test internet stability.

pingtest Description A personal project for testing internet stability, intended for use in Linux and Windows.

chris 0 Oct 17, 2021
This is a bot that can type without any assistance and have incredible speed.

BulldozerType This is a bot that can type without any assistance and have incredible speed. This bot currently only works on the site https://onlinety

1 Jan 03, 2022
자동 건강상태 자가진단 메크로 서버전용

Auto-Self-Diagnosis-for-server 자동 자가진단 메크로 서버전용 이 프로그램은 SaidBySolo님의 auto-self-diagnosis를 참고하여 제작하였습니다. 개인 사용 목적으로 제작하였기 때문에 추후 업데이트는 진행하지 않습니다. 의존성 G

JJooni 3 Dec 04, 2021
Whatsapp messages bulk sender using Python Selenium.

Whatsapp Sender Whatsapp Sender automates sending of messages via Whatsapp Web. The tool allows you to send whatsapp messages in bulk. This program re

Yap Yee Qiang 3 Jan 23, 2022
create custom test databases that are populated with fake data

About Generate fake but valid data filled databases for test purposes using most popular patterns(AFAIK). Current support is sqlite, mysql, postgresql

Emir Ozer 2.2k Jan 04, 2023
A pytest plugin, that enables you to test your code that relies on a running PostgreSQL Database

This is a pytest plugin, that enables you to test your code that relies on a running PostgreSQL Database. It allows you to specify fixtures for PostgreSQL process and client.

Clearcode 252 Dec 21, 2022
A wrapper for webdriver that is a jumping off point for web automation.

Webdriver Automation Plus ===================================== Description: Tests the user can save messages then find them in search and Saved items

1 Nov 08, 2021
RAT-el is an open source penetration test tool that allows you to take control of a windows machine.

To prevent RATel from being detected by antivirus, please do not upload the payload to TOTAL VIRUS. Each month I will test myself if the payload gets detected by antivirus. So you’ll have a photo eve

218 Dec 16, 2022
A pytest plugin that enables you to test your code that relies on a running Elasticsearch search engine

pytest-elasticsearch What is this? This is a pytest plugin that enables you to test your code that relies on a running Elasticsearch search engine. It

Clearcode 65 Nov 10, 2022
Browser reload with uvicorn

uvicorn-browser This project is inspired by autoreload. Installation pip install uvicorn-browser Usage Run uvicorn-browser --help to see all options.

Marcelo Trylesinski 64 Dec 17, 2022
Testing - Instrumenting Sanic framework with Opentelemetry

sanic-otel-splunk Testing - Instrumenting Sanic framework with Opentelemetry Test with python 3.8.10, sanic 20.12.2 Step to instrument pip install -r

Donler 1 Nov 26, 2021
Mockoon is the easiest and quickest way to run mock APIs locally. No remote deployment, no account required, open source.

Mockoon Mockoon is the easiest and quickest way to run mock APIs locally. No remote deployment, no account required, open source. It has been built wi

mockoon 4.4k Dec 30, 2022
Find index entries in $INDEX_ALLOCATION attributes

INDXRipper Find index entries in $INDEX_ALLOCATION attributes Timeline created using mactime.pl on the combined output of INDXRipper and fls. See: sle

32 Nov 05, 2022
Free cleverbot without headless browser

Cleverbot Scraper Simple free cleverbot library that doesn't require running a heavy ram wasting headless web browser to actually chat with the bot, a

Matheus Fillipe 3 Sep 25, 2022
Screenplay pattern base for Python automated UI test suites.

ScreenPy TITLE CARD: "ScreenPy" TITLE DISAPPEARS.

Perry Goy 39 Nov 15, 2022
One-stop solution for HTTP(S) testing.

HttpRunner HttpRunner is a simple & elegant, yet powerful HTTP(S) testing framework. Enjoy! ✨ 🚀 ✨ Design Philosophy Convention over configuration ROI

HttpRunner 3.5k Jan 04, 2023
To automate the generation and validation tests of COSE/CBOR Codes and it's base45/2D Code representations

To automate the generation and validation tests of COSE/CBOR Codes and it's base45/2D Code representations, a lot of data has to be collected to ensure the variance of the tests. This respository was

160 Jul 25, 2022
This project demonstrates selenium's ability to extract files from a website.

This project demonstrates selenium's ability to extract files from a website. I've added the challenge of connecting over TOR. This package also includes a personal archive site built in NodeJS and A

2 Jan 16, 2022
A complete test automation tool

Golem - Test Automation Golem is a test framework and a complete tool for browser automation. Tests can be written with code in Python, codeless using

486 Dec 30, 2022
PyAutoEasy is a extension / wrapper around the famous PyAutoGUI, a cross-platform GUI automation tool to replace your boooring repetitive tasks.

PyAutoEasy PyAutoEasy is a extension / wrapper around the famous PyAutoGUI, a cross-platform GUI automation tool to replace your boooring repetitive t

Dingu Sagar 7 Oct 27, 2022