Fast scatter density plots for Matplotlib

Overview

Azure Status Coverage Status

About

Plotting millions of points can be slow. Real slow... 😴

So why not use density maps?

The mpl-scatter-density mini-package provides functionality to make it easy to make your own scatter density maps, both for interactive and non-interactive use. Fast. The following animation shows real-time interactive use with 10 million points, but interactive performance is still good even with 100 million points (and more if you have enough RAM).

Demo of mpl-scatter-density with NY taxi data

When panning, the density map is shown at a lower resolution to keep things responsive (though this is customizable).

To install, simply do:

pip install mpl-scatter-density

This package requires Numpy, Matplotlib, and fast-histogram - these will be installed by pip if they are missing. Both Python 2.7 and Python 3.x are supported, and the package should work correctly on Linux, MacOS X, and Windows.

Usage

There are two main ways to use mpl-scatter-density, both of which are explained below.

scatter_density method

The easiest way to use this package is to simply import mpl_scatter_density, then create Matplotlib axes as usual but adding a projection='scatter_density' option (if your reaction is 'wait, what?', see here). This will return a ScatterDensityAxes instance that has a scatter_density method in addition to all the usual methods (scatter, plot, etc.).

import numpy as np
import mpl_scatter_density
import matplotlib.pyplot as plt

# Generate fake data

N = 10000000
x = np.random.normal(4, 2, N)
y = np.random.normal(3, 1, N)

# Make the plot - note that for the projection option to work, the
# mpl_scatter_density module has to be imported above.

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1, projection='scatter_density')
ax.scatter_density(x, y)
ax.set_xlim(-5, 10)
ax.set_ylim(-5, 10)
fig.savefig('gaussian.png')

Which gives:

Result from the example script

The scatter_density method takes the same options as imshow (for example cmap, alpha, norm, etc.), but also takes the following optional arguments:

  • dpi: this is an integer that is used to determine the resolution of the density map. By default, this is 72, but you can change it as needed, or set it to None to use the default for the Matplotlib backend you are using.
  • downres_factor: this is an integer that is used to determine how much to downsample the density map when panning in interactive mode. Set this to 1 if you don't want any downsampling.
  • color: this can be set to any valid matplotlib color, and will be used to automatically make a monochromatic colormap based on this color. The colormap will fade to transparent, which means that this mode is ideal when showing multiple density maps together.

Here is an example of using the color option:

import numpy as np
import matplotlib.pyplot as plt
import mpl_scatter_density  # noqa

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1, projection='scatter_density')

n = 10000000

x = np.random.normal(0.5, 0.3, n)
y = np.random.normal(0.5, 0.3, n)

ax.scatter_density(x, y, color='red')

x = np.random.normal(1.0, 0.2, n)
y = np.random.normal(0.6, 0.2, n)

ax.scatter_density(x, y, color='blue')

ax.set_xlim(-0.5, 1.5)
ax.set_ylim(-0.5, 1.5)

fig.savefig('double.png')

Which produces the following output:

Result from the example script

ScatterDensityArtist

If you are a more experienced Matplotlib user, you might want to use the ScatterDensityArtist directly (this is used behind the scenes in the above example). To use this, initialize the ScatterDensityArtist with the axes as first argument, followed by any arguments you would have passed to scatter_density above (you can also take a look at the docstring for ScatterDensityArtist). You should then add the artist to the axes:

from mpl_scatter_density import ScatterDensityArtist
a = ScatterDensityArtist(ax, x, y)
ax.add_artist(a)

Advanced

Non-linear stretches for high dynamic range plots

In some cases, your density map might have a high dynamic range, and you might therefore want to show the log of the counts rather than the counts. You can do this by passing a matplotlib.colors.Normalize object to the norm argument in the same wasy as for imshow. For example, the astropy package includes a nice framework for making such a Normalize object for different functions. The following example shows how to show the density map on a log scale:

import numpy as np
import mpl_scatter_density
import matplotlib.pyplot as plt

# Make the norm object to define the image stretch
from astropy.visualization import LogStretch
from astropy.visualization.mpl_normalize import ImageNormalize
norm = ImageNormalize(vmin=0., vmax=1000, stretch=LogStretch())

N = 10000000
x = np.random.normal(4, 2, N)
y = np.random.normal(3, 1, N)

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1, projection='scatter_density')
ax.scatter_density(x, y, norm=norm)
ax.set_xlim(-5, 10)
ax.set_ylim(-5, 10)
fig.savefig('gaussian_log.png')

Which produces the following output:

Result from the example script

Adding a colorbar

You can show a colorbar in the same way as you would for an image - the following example shows how to do it:

import numpy as np
import mpl_scatter_density
import matplotlib.pyplot as plt

N = 10000000
x = np.random.normal(4, 2, N)
y = np.random.normal(3, 1, N)

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1, projection='scatter_density')
density = ax.scatter_density(x, y)
ax.set_xlim(-5, 10)
ax.set_ylim(-5, 10)
fig.colorbar(density, label='Number of points per pixel')
fig.savefig('gaussian_colorbar.png')

Which produces the following output:

Result from the example script

Color-coding 'markers' with individual values

In the same way that a 1-D array of values can be passed to Matplotlib's scatter function/method, a 1-D array of values can be passed to scatter_density using the c= argument:

import numpy as np
import mpl_scatter_density
import matplotlib.pyplot as plt

N = 10000000
x = np.random.normal(4, 2, N)
y = np.random.normal(3, 1, N)
c = x - y + np.random.normal(0, 5, N)

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1, projection='scatter_density')
ax.scatter_density(x, y, c=c, vmin=-10, vmax=+10, cmap=plt.cm.RdYlBu)
ax.set_xlim(-5, 13)
ax.set_ylim(-5, 11)
fig.savefig('gaussian_color_coded.png')

Which produces the following output:

Result from the example script

Note that to keep performance as good as possible, the values from the c attribute are averaged inside each pixel of the density map, then the colormap is applied. This is a little different to what scatter would converge to in the limit of many points (since in that case it would apply the color to all the markers than average the colors).

Q&A

Isn't this basically the same as datashader?

This follows the same ideas as datashader, but the aim of mpl-scatter-density is specifically to bring datashader-like functionality to Matplotlib users. Furthermore, mpl-scatter-density is intended to be very easy to install - for example it can be installed with pip. But if you have datashader installed and regularly use bokeh, mpl-scatter-density won't do much for you. Note that if you are interested in datashader and Matplotlib together, there is a work in progress (pull request) by @tacaswell to create a Matplotlib artist similar to that in this package but powered by datashader.

What about vaex?

Vaex is a powerful package to visualize large datasets on N-dimensional grids, and therefore has some functionality that overlaps with what is here. However, the aim of mpl-scatter-density is just to provide a lightweight solution to make it easy for users already using Matplotlib to add scatter density maps to their plots rather than provide a complete environment for data visualization. I highly recommend that you take a look at Vaex and determine which approach is right for you!

Why on earth have you defined scatter_density as a projection?

If you are a Matplotlib developer: I truly am sorry for distorting the intended purpose of projection 😊 . But you have to admit that it's a pretty convenient way to have users get a custom Axes sub-class even if it has nothing to do with actual projection!

Where do you see this going?

There are a number of things we could add to this package, for example a way to plot density maps as contours, or a way to color code each point by a third quantity and have that reflected in the density map. If you have ideas, please open issues, and even better contribute a pull request! 😄

Can I contribute?

I'm glad you asked - of course you are very welcome to contribute! If you have some ideas, you can open issues or create a pull request directly. Even if you don't have time to contribute actual code changes, I would love to hear from you if you are having issues using this package.

[![Build Status](https://dev.azure.com/thomasrobitaille/mpl-scatter-density/_apis/build/status/astrofrog.mpl-scatter-density?branchName=master)](https://dev.azure.com/thomasrobitaille/mpl-scatter-density/_build/latest?definitionId=17&branchName=master)

Running tests

To run the tests, you will need pytest and the pytest-mpl plugin. You can then run the tests with:

pytest mpl_scatter_density --mpl
Owner
Thomas Robitaille
Thomas Robitaille
A GUI for Pandas DataFrames

PandasGUI A GUI for analyzing Pandas DataFrames. Demo Installation Install latest release from PyPi: pip install pandasgui Install directly from Githu

Adam 2.8k Jan 03, 2023
Fast data visualization and GUI tools for scientific / engineering applications

PyQtGraph A pure-Python graphics library for PyQt5/PyQt6/PySide2/PySide6 Copyright 2020 Luke Campagnola, University of North Carolina at Chapel Hill h

pyqtgraph 3.1k Jan 08, 2023
A Bokeh project developed for learning and teaching Bokeh interactive plotting!

Bokeh-Python-Visualization A Bokeh project developed for learning and teaching Bokeh interactive plotting! See my medium blog posts about making bokeh

Will Koehrsen 350 Dec 05, 2022
1900-2016 Olympic Data Analysis in Python by plotting different graphs

🔥 Olympics Data Analysis 🔥 In Data Science field, there is a big topic before creating a model for future prediction is Data Analysis. We can find o

Sayan Roy 1 Feb 06, 2022
Simple addon for snapping active object to mesh ground

Snap to Ground Simple addon for snapping active object to mesh ground How to install: install the Python file as an addon use shortcut "D" in 3D view

Iyad Ahmed 12 Nov 07, 2022
Simple python implementation with matplotlib to manually fit MIST isochrones to Gaia DR2 color-magnitude diagrams

Simple python implementation with matplotlib to manually fit MIST isochrones to Gaia DR2 color-magnitude diagrams

Karl Jaehnig 7 Oct 22, 2022
eoplatform is a Python package that aims to simplify Remote Sensing Earth Observation by providing actionable information on a wide swath of RS platforms and provide a simple API for downloading and visualizing RS imagery

An Earth Observation Platform Earth Observation made easy. Report Bug | Request Feature About eoplatform is a Python package that aims to simplify Rem

Matthew Tralka 4 Aug 11, 2022
nvitop, an interactive NVIDIA-GPU process viewer, the one-stop solution for GPU process management

An interactive NVIDIA-GPU process viewer, the one-stop solution for GPU process management.

Xuehai Pan 1.3k Jan 02, 2023
A blender import/export system for Defold

defold-blender-export A Blender export system for the Defold game engine. Setup Notes There are no exhaustive documents for this tool yet. Its just no

David Lannan 27 Dec 30, 2022
Render Jupyter notebook in the terminal

jut - JUpyter notebook Terminal viewer. The command line tool view the IPython/Jupyter notebook in the terminal. Install pip install jut Usage $jut --

Kracekumar 169 Dec 27, 2022
🌀❄️🌩️ This repository contains some examples for creating 2d and 3d weather plots using matplotlib and cartopy libraries in python3.

Weather-Plotting 🌀 ❄️ 🌩️ This repository contains some examples for creating 2d and 3d weather plots using matplotlib and cartopy libraries in pytho

Giannis Dravilas 21 Dec 10, 2022
股票行情实时数据接口-A股,完全免费的沪深证券股票数据-中国股市,python最简封装的API接口

股票行情实时数据接口-A股,完全免费的沪深证券股票数据-中国股市,python最简封装的API接口,包含日线,历史K线,分时线,分钟线,全部实时采集,系统包括新浪腾讯双数据核心采集获取,自动故障切换,STOCK数据格式成DataFrame格式,可用来查询研究量化分析,股票程序自动化交易系统.为量化研究者在数据获取方面极大地减轻工作量,更加专注于策略和模型的研究与实现。

dev 572 Jan 08, 2023
Visualise top-rated GitHub repositories in a barchart by keyword

This python script was written for simple purpose -- to visualise top-rated GitHub repositories in a barchart by keyword. Script generates html-page with barchart and information about repository own

Cur1iosity 2 Feb 07, 2022
Gesture controlled media player

Media Player Gesture Control Gesture controller for media player with MediaPipe, VLC and OpenCV. Contents About Setup About A tool for using gestures

Atharva Joshi 2 Dec 22, 2021
An application that allows you to design and test your own stock trading algorithms in an attempt to beat the market.

StockBot is a Python application for designing and testing your own daily stock trading algorithms. Installation Use the

Ryan Cullen 280 Dec 19, 2022
Fastest Gephi's ForceAtlas2 graph layout algorithm implemented for Python and NetworkX

ForceAtlas2 for Python A port of Gephi's Force Atlas 2 layout algorithm to Python 2 and Python 3 (with a wrapper for NetworkX and igraph). This is the

Bhargav Chippada 227 Jan 05, 2023
Minimalistic tool to visualize how the routes to a given target domain change over time, feat. Python 3.10 & mermaid.js

Minimalistic tool to visualize how the routes to a given target domain change over time, feat. Python 3.10 & mermaid.js

Péter Ferenc Gyarmati 1 Jan 17, 2022
📊 Charts with pure python

A zero-dependency python package that prints basic charts to a Jupyter output Charts supported: Bar graphs Scatter plots Histograms 🍑 📊 👏 Examples

Max Humber 54 Oct 04, 2022
Draw datasets from within Jupyter.

drawdata This small python app allows you to draw a dataset in a jupyter notebook. This should be very useful when teaching machine learning algorithm

vincent d warmerdam 505 Nov 27, 2022
HW 2: Visualizing interesting datasets

HW 2: Visualizing interesting datasets Check out the project instructions here! Mean Earnings per Hour for Males and Females My first graph uses data

7 Oct 27, 2021