An extension to pandas dataframes describe function.

Last update: Dec 30, 2022

Overview

pandas_summary

An extension to pandas dataframes describe function.

The module contains DataFrameSummary object that extend describe() with:

properties
- dfs.columns_stats: counts, uniques, missing, missing_perc, and type per column
- dsf.columns_types: a count of the types of columns
- dfs[column]: more in depth summary of the column
function
- summary(): extends the describe() function with the values with columns_stats

Installation

The module can be easily installed with pip:

> pip install pandas-summary

This module depends on numpy and pandas. Optionally you can get also some nice visualisations if you have matplotlib installed.

Tests

To run the tests, execute the command python setup.py test

Usage

The module contains one class:

DataFrameSummary

The DataFrameSummary expect a pandas DataFrame to summarise.

from pandas_summary import DataFrameSummary

dfs = DataFrameSummary(df)

getting the columns types

dfs.columns_types


numeric     9
bool        3
categorical 2
unique      1
date        1
constant    1
dtype: int64

getting the columns stats

dfs.columns_stats


                      A            B        C              D              E 
counts             5802         5794     5781           5781           4617   
uniques            5802            3     5771            128            121   
missing               0            8       21             21           1185   
missing_perc         0%        0.14%    0.36%          0.36%         20.42%   
types            unique  categorical  numeric        numeric        numeric

getting a single column summary, e.g. numerical column

# we can also access the column using numbers A[1]
dfs['A']

std                                                                 0.2827146
max                                                                  1.072792
min                                                                         0
variance                                                           0.07992753
mean                                                                0.5548516
5%                                                                  0.1603367
25%                                                                 0.3199776
50%                                                                 0.4968588
75%                                                                 0.8274732
95%                                                                  1.011255
iqr                                                                 0.5074956
kurtosis                                                            -1.208469
skewness                                                            0.2679559
sum                                                                  3207.597
mad                                                                 0.2459508
cv                                                                  0.5095319
zeros_num                                                                  11
zeros_perc                                                               0,1%
deviating_of_mean                                                          21
deviating_of_mean_perc                                                  0.36%
deviating_of_median                                                        21
deviating_of_median_perc                                                0.36%
top_correlations                         {u'D': 0.702240243124, u'E': -0.663}
counts                                                                   5781
uniques                                                                  5771
missing                                                                    21
missing_perc                                                            0.36%
types                                                                 numeric
Name: A, dtype: object

Future development

Summary analysis between columns, i.e. dfs[[1, 2]]

An extension to pandas dataframes describe function.

Related tags

Overview

pandas_summary

Installation

Tests

Usage

DataFrameSummary

Future development

Owner

Mourad

PyNHD is a part of HyRiver software stack that is designed to aid in watershed analysis through web services.

Data collection, enhancement, and metrics calculation.

Statistical Rethinking course winter 2022

track your GitHub statistics

API>local_db>AWS_RDS - Disclaimer! All data used is for educational purposes only.

This is an analysis and prediction project for house prices in King County, USA based on certain features of the house

Evaluation of a Monocular Eye Tracking Set-Up

Finding project directories in Python (data science) projects, just like there R rprojroot and here packages

Picka: A Python module for data generation and randomization.

A Python package for modular causal inference analysis and model evaluations

CINECA molecular dynamics tutorial set

Analyse the limit order book in seconds. Zoom to tick level or get yourself an overview of the trading day.

signac-flow - manage workflows with signac

An ETL Pipeline of a large data set from a fictitious music streaming service named Sparkify.

BinTuner is a cost-efficient auto-tuning framework, which can deliver a near-optimal binary code that reveals much more differences than -Ox settings.

An Indexer that works out-of-the-box when you have less than 100K stored Documents

Integrate bus data from a variety of sources (batch processing and real time processing).

Hatchet is a Python-based library that allows Pandas dataframes to be indexed by structured tree and graph data.

Create HTML profiling reports from pandas DataFrame objects

A project consists in a set of assignements corresponding to a BI process: data integration, construction of an OLAP cube, qurying of a OPLAP cube and reporting.