Ballcone is a fast and lightweight server-side Web analytics solution.

Overview

Ballcone

Ballcone is a fast and lightweight server-side Web analytics solution. It requires no JavaScript on your website.

GitHub Tests Docker Hub

Screenshots

Ballcone

Ballcone: petrovich

Design Goals

  • Simplicity. Ballcone requires almost zero set-up as it prefers convention over configuration
  • Efficiency. Ballcone performs lightning-fast analytic queries over data thanks to the underlying columnar database
  • Specificity. Ballcone aims at providing visual insights on the HTTP access logs with no bloat

Features

  • No JavaScript snippets required
  • GeoIP mapping with the GeoLite2 database
  • Extraction of platform and browser information from User-Agent

Architecture

Ballcone captures the access_log entries exported in JSON by nginx via the bundled syslog logger (65140/udp). These entries are stored in the embedded MonetDBLite database. Ballcone uses it to perform data manipulation and analytic queries. Also, Ballcone provides a convenient Web interface (8080/tcp) for accessing and observing the gathered data.

          +-----------+            +------------+
   HTTP   |           |   syslog   |            |   HTTP
<-------->+   nginx   +----------->+  Ballcone  +<-------->
          |           |    JSON    |            |
          +-----------+            +------------+
                                   |MonetDB-Lite|
                                   +------------+

For better performance, Ballcone inserts data in batches, committing them to MonetDBLite every few seconds (five seconds by default).

Requirements

Demo

This repository contains an example configuration of nginx and Ballcone. Just run the container from Docker Hub or build it locally. nginx will be available at http://127.0.0.1:8888/ and Ballcone will be available at http://127.0.0.1:8080/.

docker-compose up
# or
docker run --rm -p '127.0.0.1:8888:80' -p '127.0.0.1:8080:8080' dustalov/ballcone:demo

Naming and Meaning

Ballcone has two meanings.

First, it is the romanization of the Russian word балкон that means a balcony. You go to the balcony to breath some fresh air and look down at the things outside.

Second, if a ball is inscribed in a cone, it resembles the all-seeing eye (help wanted: dustalov/ballcone#8).

Regardless of the meaning you prefer, Ballcone helps you to watch your websites.

Installation

The simplest way to get started is to run make pipenv after cloning the repository. Just make sure Pipenv is installed.

Getting Ballcone

Running the Docker image is the simplest way to get started. Docker Hub performs automated builds of the Ballcone source code from GitHub: https://hub.docker.com/r/dustalov/ballcone. The following command runs Ballcone on 127.0.0.1: the syslog protocol will be available via 65140/udp, the Web interface will be available via 8080/tcp, and the data will be stored in the /var/lib/ballcone directory on the host machine.

docker run -p '127.0.0.1:8080:8080' -p '127.0.0.1:65140:65140/udp' -v '/var/lib/ballcone/monetdb:/usr/src/app/monetdb' --restart=unless-stopped dustalov/ballcone ballcone -sh '0.0.0.0' -wh '0.0.0.0'

However, Docker is not the only option. Alternatively, Ballcone can be packaged into a standalone executable using PyInstaller and runned as a systemd service (see ballcone.service as an example):

make pyinstaller
sudo make install-systemd
sudo systemctl start ballcone

Finally, Ballcone can be installed directly on the host machine for manual runs:

pip3 install -e git+https://github.com/dustalov/[email protected]#egg=ballcone

Note that ballcone without arguments creates the monetdb directory inside the current directory.

Configuring nginx

You need to define the JSON-compatible log format for your service in the nginx configuration file. Let us call it ballcone_json_example. This format is similar to the one used in Matomo (see matomo-log-analytics). It should be put before the server context.

log_format ballcone_json_example escape=json
    '{'
    '"service": "example", '
    '"ip": "$remote_addr", '
    '"host": "$host", '
    '"path": "$request_uri", '
    '"status": "$status", '
    '"referrer": "$http_referer", '
    '"user_agent": "$http_user_agent", '
    '"length": $bytes_sent, '
    '"generation_time_milli": $request_time, '
    '"date": "$time_iso8601"'
    '}';

Then, you should put this access_log directive inside the server context to transfer logs via the syslog protocol.

access_log syslog:server=127.0.0.1:65140 ballcone_json_example;

Please look at the complete example of nginx configuration in demo/nginx.conf.

Roadmap

Roadmap is available at https://github.com/dustalov/ballcone/issues.

Alternatives

Copyright

Copyright © 2020 Dmitry Ustalov. See LICENSE for details.

Owner
Dmitry Ustalov
Head of Research at @Toloka, https://toloka.ai/research
Dmitry Ustalov
We are building an open database of COVID-19 cases with chest X-ray or CT images.

🛑 Note: please do not claim diagnostic performance of a model without a clinical study! This is not a kaggle competition dataset. Please read this pa

Joseph Paul Cohen 2.9k Dec 30, 2022
PyScaffold is a project generator for bootstrapping high quality Python packages

PyScaffold is a project generator for bootstrapping high quality Python packages, ready to be shared on PyPI and installable via pip. It is easy to use and encourages the adoption of the best tools a

PyScaffold 1.7k Jan 03, 2023
The Official interpreter for the Pix programming language.

The official interpreter for the Pix programming language. Pix Pix is a programming language dedicated to readable syntax and usability Q) Is Pix the

Pix 6 Sep 25, 2022
A web app that is written entirely in Python

University Project About I made this web app to finish a project assigned by my teacher. It is written entirely in Python, thanks to streamlit to make

15 Nov 27, 2022
A modern Python build backend

trampolim A modern Python build backend. Features Task system, allowing to run arbitrary Python code during the build process (Planned) Easy to use CL

Filipe Laíns 39 Nov 08, 2022
A web-based analysis toolkit for the System Usability Scale providing calculation, plotting, interpretation and contextualization utility

System Usability Scale Analysis Toolkit The System Usability Scale (SUS) Analysis Toolkit is a web-based python application that provides a compilatio

Jonas Blattgerste 3 Oct 27, 2022
Click2call for asterisk with python

Click2call para Asterisk com Python Este projeto disponibiliza uma API construíd

Benedito Marques 1 Jan 17, 2022
Interfaces between napari and pymeshlab library to allow import, export and construction of surfaces.

napari-pymeshlab Interfaces between napari and the pymeshlab library to allow import, export and construction of surfaces. This is a WIP and feature r

Zach Marin 4 Oct 12, 2022
Originally used during Marketplace.tf's open period, this program was used to get the profit of items bought with keys and sold for dollars.

Originally used during Marketplace.tf's open period, this program was used to get the profit of items bought with keys and sold for dollars. Practically useless for me now, but can be used as an exam

BoggoTV 1 Dec 11, 2021
An implementation of an interpreter for the Brainfuck esoteric language in Python

Brainfuck Interpreter in Python An implementation of an interpreter for the Brainfuck esoteric language in Python. 🧠 The Brainfuck Language Created i

Carlos Santos 0 Feb 01, 2022
奇遇淘客服务器端

奇遇淘客 APP 服务器端 警告 正在使用 v0.2.0 版本的用户,请尽快升级到 v0.2.1。 v0.2.0 版本的 Docker 镜像中包含了有问题的 aiohttp。 奇遇淘客代码库 奇遇淘客 iOS APP 奇遇淘客 Android APP 奇遇淘客文档 服务器端文档 Docker 使用

奇遇科技 92 Nov 09, 2022
A project to find out all the words in a crossword.

A project to find out all the words in a crossword.

Kalpesh Dhoundiyal 1 Feb 06, 2022
Example of my qtile config using the gruvbox colorscheme.

QTILE config Example of my qtile config using the gruvbox colorscheme. unicodes.py unicodes.py returns a widget.TextBox with a unicode. Currently it c

Imanuel Febie 31 Jan 02, 2023
A GUI love Calculator which saves all the User Data in text file(sql based script will be uploaded soon). Interative GUI. Even For Admin Panel

Love-Calculator A GUI love Calculator which saves all the User Data in text file(sql based script will be uploaded soon). Interative GUI, even For Adm

Adithya Krishnan 1 Mar 22, 2022
PyDateWaiter helps waiting special day & calculating remain days till that day with Python code.

PyDateWaiter (v.Beta) PyDateWaiter helps waiting special day(aniversary) & calculating remain days till that day with Python code. Made by wallga gith

wallga 1 Jan 14, 2022
Analisador de strings feito em Python // String parser made in Python

Este é um analisador feito em Python, neste programa, estou estudando funções e a sua junção com "if's" e dados colocados pelo usuário. Neste código,

Dev Nasser 1 Nov 03, 2021
ICEtool - ICEtool plugin for QGIS

ICEtool ICEtool is an all in one QGIS plugin to easily compute ground temperatur

Arthur Evrard 13 Dec 16, 2022
The fastest way to copy to (not from) high speed flash storage.

FastestCopy The fastest way to copy to (not from) high speed flash storage. This is about 3-6x faster than file copy on explorer.exe to usb flash driv

Derek Frombach 0 Nov 03, 2021
TickerRain is an open-source web app that stores and analysis Reddit posts in a transparent and semi-interactive manner.

TickerRain is an open-source web app that stores and analysis Reddit posts in a transparent and semi-interactive manner

GonVas 180 Oct 08, 2022
Module-based cryptographic tool

Cryptosploit A decryption/decoding/cracking tool using various modules. To use it, you need to have basic knowledge of cryptography. Table of Contents

/SNESE_AR\ 33 Nov 27, 2022