Build a better understanding of your data in PostgreSQL.

Overview

Data Fluent for PostgreSQL

license

Build a better understanding of your data in PostgreSQL.

The following shows an example report generated by this tool. It gives the numbers of rows, columns, bytes as well as human-friendly size counts for each table within a given PostgreSQL database.

The Metrics Report

The following shows the row count for every column that represents a date grouped by year and month.

The Time Distribution Report

Installation

On Ubuntu 20:

$ wget -qO- \
    https://www.postgresql.org/media/keys/ACCC4CF8.asc \
        | sudo apt-key add -
$ echo "deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main" \
    | sudo tee /etc/apt/sources.list.d/pgdg.list

$ sudo apt update
$ sudo apt install \
    git \
    python3-pip \
    python3-virtualenv \
    postgresql-13 \
    postgresql-client-13 \
    postgresql-contrib

On macOS:

$ brew install \
    git \
    postgresql \
    virtualenv

Then, regardless of platform, setup a virtual environment and install the following Python-based dependencies.

$ virtualenv ~/.fluency
$ source ~/.fluency/bin/activate
$ python3 -m pip install \
    csvkit \
    humanfriendly \
    ipython \
    openpyxl \
    Pandas \
    psycopg2-binary \
    typer \
    xlsxwriter

Then clone this repo as it contains the datafluent_pg.py script:

$ git clone https://github.com/marklit/datafluent_pg.git ~/datafluent_pg
$ cd ~/datafluent_pg

Example Analysis

Clone fivethirtyeight's data repo. It has a large number of CSV-formatted datasets.

$ git clone https://github.com/fivethirtyeight/data.git ~/538data

Make sure you can access a PostgreSQL database on your machine. Here I'm creating an intel database for the mark user on my Ubuntu 20 machine.

$ sudo -u postgres \
    bash -c "psql -c \"CREATE USER mark
                       WITH PASSWORD 'test'
                       SUPERUSER;\""

With PostgreSQL access setup, create a database called intel.

$ createdb intel

I'll import one of the datasets within fivethirtyeight's repo. Note, because the dates within this dataset are not formatted in YYYY-MM-DD format, I needed to override the format so that the MM/DD/YYYY format would be read properly.

$ csvsql --db postgresql:///intel \
         --insert ~/538data/congress-generic-ballot/generic_topline_historical.csv \
         --datetime-format="%m/%d/%Y"

I'll run the Excel Report Generator:

$ python datafluent_pg.py

This will result in a fluency.xlsx file being produced with two worksheets: Metrics and Time Distributions.

If you need to override any parameters, please refer to the documentation:

$ python datafluent_pg.py --help
Usage: datafluent_pg.py [OPTIONS]

Options:
  --pg-dns TEXT                   [default: postgresql://localhost:5432/intel]
  --output TEXT                   [default: fluency.xlsx]
  --install-completion [bash|zsh|fish|powershell|pwsh]
                                  Install completion for the specified shell.
  --show-completion [bash|zsh|fish|powershell|pwsh]
                                  Show completion for the specified shell, to
                                  copy it or customize the installation.

  --help                          Show this message and exit.
Die wichtigsten APIs Deutschlands in einem Python Paket.

Deutschland A python package that gives you easy access to the most valuable datasets of Germany. Installation pip install deutschland Geographic data

Bundesstelle für Open Data 921 Jan 08, 2023
Compulsory join Telegram Bot

mussjoin About Compulsory join Telegram Bot this Telegram Bot Application can be added users to Telegram Channel or Group compulsorily. in addition wh

Hamed Mohammadvand 4 Dec 03, 2021
multi-purpose discord bot

virus multi-purpose discord bot ⚠️ WARNING This project is incomplete and may not work as expected. Download & Run Install Python =3.10 Clone the sou

miten 2 Jan 17, 2022
Aplicação dos metodos de classificação em 3 diferentes banco de dados. Usando...

Machine Learning - Métodos de classificação Base de Dados utilizadas: Dados de crédito Dados do Census Métodos de classificação aplicados: Naive Bayes

1 Jan 18, 2022
Python API Client for Twitter API v2

🐍 Python Client For Twitter API v2 🚀 Why Twitter Stream ? Twitter-Stream.py a python API client for Twitter API v2 now supports FilteredStream, Samp

Twitivity 31 Nov 19, 2022
Yes, it's true :revolving_hearts: This repository has 301 stars.

Yes, it's true! Inspired by a similar repository from @RealPeha, but implemented using a webhook on AWS Lambda and API Gateway, so it's serv

511 Dec 30, 2022
This is a script to forward forward large number of documents to another telegram channel.

ChannelForward 😇 This is a Script to Forward Large Number of Documents to Another Telegram Channel. If You Try to Forward Very Large Number of Files

Anjana Madushanka 10 Jun 08, 2021
Fully asynchronous trace.moe API wrapper

AioMoe Fully asynchronous trace.moe API wrapper Installation You can install the stable version from PyPI: $ pip install aiomoe Or get it from github

2 Jun 26, 2022
C Y B Ξ R UserBot is a project that simplifies the use of Telegram. All rights reserved.

C Y B Ξ R USΞRBOT 🇦🇿 C Y B Ξ R UserBot is a project that simplifies the use of Telegram. All rights reserved. Automatic Setup Android: open Termux p

C Y B Ξ R 0 Sep 20, 2022
A webhook API for Discord.

Webhook API A webhook API for Discord. Requirements requests Usage

1 Feb 08, 2022
Detects members having unicode names. Public bot: @scarletwitchprobot

✨ Scarletwitch bot ✨ Detects unicode names members in a tg chat & provides a option to take action on that user ! Public bot: @scarletwitchprobot Supp

ÁÑÑÍHÌLÅTØR SPÄRK 18 Nov 12, 2022
This app is providing you to track some online products' prices via GMAIL.

Price Tracking App variables and descriptions of that code is in Turkish language. but we're working on translate them into English. This app is provi

Abdullah Aslan 1 Dec 11, 2021
Ethereum transactions and wallet information for people you follow on Twitter.

ethFollowing Ethereum transactions and wallet information for people you follow on Twitter. Set up Setup python environment (requires python 3.8): vir

Brian Donohue 2 Dec 28, 2021
Cogs for Red-DiscordBot

matcha-cogs Cogs for Red-DiscordBot. Installation [p]repo add matcha-cogs

MatchaTeaLeaf 2 Aug 27, 2022
Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering

Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering

Alexandr Savinov 326 Jan 03, 2023
Source code of u/pekofy_bot from reddit.

pekofy-bot Source code of u/pekofy_bot from reddit. Get more info about the bot here: https://www.reddit.com/user/pekofy_bot/comments/krxxol/pekofy_bo

32 Dec 25, 2022
Easy-apply-bot - A LinkedIn Easy Apply bot to help with my job search.

easy-apply-bot A LinkedIn Easy Apply bot to help with my job search. Getting Started First, clone the repository somewhere onto your computer, or down

Matthew Alunni 5 Dec 09, 2022
DSAIL repos - DSAIL Repository Template

DSAIL Repository Template DSAIL @ KAIST . ├── configs ('--F', help='for configur

yunhak 2 Feb 14, 2022
The official Python library for Shodan

shodan: The official Python library and CLI for Shodan Shodan is a search engine for Internet-connected devices. Google lets you search for websites,

John Matherly 2.1k Dec 31, 2022
A results generator and automatic token checker for Yandex Contest

Yandex contest Python checking tools A results generator and automatic token checker for Yandex Contest. Версия на русском языке Installation Clone th

Nikolay Chechulin 9 Dec 14, 2022