Ballcone is a fast and lightweight server-side Web analytics solution.

Overview

Ballcone

Ballcone is a fast and lightweight server-side Web analytics solution. It requires no JavaScript on your website.

GitHub Tests Docker Hub

Screenshots

Ballcone

Ballcone: petrovich

Design Goals

  • Simplicity. Ballcone requires almost zero set-up as it prefers convention over configuration
  • Efficiency. Ballcone performs lightning-fast analytic queries over data thanks to the underlying columnar database
  • Specificity. Ballcone aims at providing visual insights on the HTTP access logs with no bloat

Features

  • No JavaScript snippets required
  • GeoIP mapping with the GeoLite2 database
  • Extraction of platform and browser information from User-Agent

Architecture

Ballcone captures the access_log entries exported in JSON by nginx via the bundled syslog logger (65140/udp). These entries are stored in the embedded MonetDBLite database. Ballcone uses it to perform data manipulation and analytic queries. Also, Ballcone provides a convenient Web interface (8080/tcp) for accessing and observing the gathered data.

          +-----------+            +------------+
   HTTP   |           |   syslog   |            |   HTTP
<-------->+   nginx   +----------->+  Ballcone  +<-------->
          |           |    JSON    |            |
          +-----------+            +------------+
                                   |MonetDB-Lite|
                                   +------------+

For better performance, Ballcone inserts data in batches, committing them to MonetDBLite every few seconds (five seconds by default).

Requirements

Demo

This repository contains an example configuration of nginx and Ballcone. Just run the container from Docker Hub or build it locally. nginx will be available at http://127.0.0.1:8888/ and Ballcone will be available at http://127.0.0.1:8080/.

docker-compose up
# or
docker run --rm -p '127.0.0.1:8888:80' -p '127.0.0.1:8080:8080' dustalov/ballcone:demo

Naming and Meaning

Ballcone has two meanings.

First, it is the romanization of the Russian word балкон that means a balcony. You go to the balcony to breath some fresh air and look down at the things outside.

Second, if a ball is inscribed in a cone, it resembles the all-seeing eye (help wanted: dustalov/ballcone#8).

Regardless of the meaning you prefer, Ballcone helps you to watch your websites.

Installation

The simplest way to get started is to run make pipenv after cloning the repository. Just make sure Pipenv is installed.

Getting Ballcone

Running the Docker image is the simplest way to get started. Docker Hub performs automated builds of the Ballcone source code from GitHub: https://hub.docker.com/r/dustalov/ballcone. The following command runs Ballcone on 127.0.0.1: the syslog protocol will be available via 65140/udp, the Web interface will be available via 8080/tcp, and the data will be stored in the /var/lib/ballcone directory on the host machine.

docker run -p '127.0.0.1:8080:8080' -p '127.0.0.1:65140:65140/udp' -v '/var/lib/ballcone/monetdb:/usr/src/app/monetdb' --restart=unless-stopped dustalov/ballcone ballcone -sh '0.0.0.0' -wh '0.0.0.0'

However, Docker is not the only option. Alternatively, Ballcone can be packaged into a standalone executable using PyInstaller and runned as a systemd service (see ballcone.service as an example):

make pyinstaller
sudo make install-systemd
sudo systemctl start ballcone

Finally, Ballcone can be installed directly on the host machine for manual runs:

pip3 install -e git+https://github.com/dustalov/[email protected]#egg=ballcone

Note that ballcone without arguments creates the monetdb directory inside the current directory.

Configuring nginx

You need to define the JSON-compatible log format for your service in the nginx configuration file. Let us call it ballcone_json_example. This format is similar to the one used in Matomo (see matomo-log-analytics). It should be put before the server context.

log_format ballcone_json_example escape=json
    '{'
    '"service": "example", '
    '"ip": "$remote_addr", '
    '"host": "$host", '
    '"path": "$request_uri", '
    '"status": "$status", '
    '"referrer": "$http_referer", '
    '"user_agent": "$http_user_agent", '
    '"length": $bytes_sent, '
    '"generation_time_milli": $request_time, '
    '"date": "$time_iso8601"'
    '}';

Then, you should put this access_log directive inside the server context to transfer logs via the syslog protocol.

access_log syslog:server=127.0.0.1:65140 ballcone_json_example;

Please look at the complete example of nginx configuration in demo/nginx.conf.

Roadmap

Roadmap is available at https://github.com/dustalov/ballcone/issues.

Alternatives

Copyright

Copyright © 2020 Dmitry Ustalov. See LICENSE for details.

Owner
Dmitry Ustalov
Head of Research at @Toloka, https://toloka.ai/research
Dmitry Ustalov
LinkScope allows you to perform online investigations by representing information as discrete pieces of data, called Entities.

LinkScope Client Description This is the repository for the LinkScope Client Online Investigation software. LinkScope allows you to perform online inv

108 Jan 04, 2023
A simple script for generating screenshots with Vapoursynth

Vapoursynth-Screenshots A simple script for generating screenshots with Vapoursynth. About I'm lazy, and hate changing variables for each batch of scr

7 Dec 31, 2022
Scrapper For Paste.pics

PrntScScrapper Scrapper for Paste.pics If you are bored you can find some random screenshots from prnt.sc Features Saving screenshots Open in Browser

Fareusz 1 Dec 29, 2021
Objetivo: de forma colaborativa pasar de nodos de Dynamo a Python.

ITTI_Ed01_De-nodos-a-python ITTI. EXPERT TRAINING EN AUTOMATIZACIÓN DE PROCESOS BIM: OFFICIAL DE AUTODESK. Edición 1 Enlace al Master Enunciado: Traba

1 Jun 06, 2022
Repo Home WPDrawBot - (Repo, Home, WP) A powerful programmatic 2D drawing application for MacOS X which generates graphics from Python scripts. (graphics, dev, mac)

DrawBot DrawBot is a powerful, free application for macOS that invites you to write Python scripts to generate two-dimensional graphics. The built-in

Frederik Berlaen 342 Dec 27, 2022
Pengenalan para anggota KOMPETEGRAM

Pengenalan Anggota KOMPETEGRAM Apa isi repositori ini ? 💬 Repositori ini berisi pengenalan nama anggota KOMPETEGRAM dari seluruh angkatan atau Batch.

Repositori KOMPETEGRAM 7 Sep 17, 2022
Web站点选优工具 - 优化GitHub的打开速度、高效Clone

QWebSiteOptimizer - Web站点速度选优工具 在访问GitHub等网站时,DNS解析到的IP地址可能并不是最快,过慢的节点会严重影响我们的访问情况,故制作出这样的工具来进一步优化网络质量。 由于该方案并非为VPN等方式进行的速度优化,以下几点需要您注意: 后续访问对应网站时仍可能需

QPT Family 15 May 01, 2022
Procedurally generated Oblique Strategies for writing your own Oblique Strategies

Procedurally generated Oblique Strategies for writing your own Oblique Strategies.

Gordon Brander 13 Aug 17, 2022
ChronoRace is a tool to accurately perform timed race conditions to circumvent application business logic.

ChronoRace is a tool to accurately perform timed race conditions to circumvent application business logic. I've found in my research that w

Tanner 64 Aug 04, 2022
A tool that bootstraps your dotfiles ⚡️

Dotbot Dotbot makes installing your dotfiles as easy as git clone $url && cd dotfiles && ./install, even on a freshly installed system! Rationale Gett

Anish Athalye 5.9k Jan 07, 2023
A short course on Julia and open-source software development

Advanced Scientific Computing: producing better code This course is taught as a 6-session "nanocourse" at Washington University in St. Louis. See the

Tim Holy 230 Jan 07, 2023
Free Data Engineering course!

Data Engineering Zoomcamp Register in DataTalks.Club's Slack Join the #course-data-engineering channel The videos are published to DataTalks.Club's Yo

DataTalksClub 7.3k Dec 30, 2022
An open-source Python project series where beginners can contribute and practice coding.

Python Mini Projects A collection of easy Python small projects to help you improve your programming skills. Table Of Contents Aim Of The Project Cont

Leah Nguyen 491 Jan 04, 2023
Module-based cryptographic tool

Cryptosploit A decryption/decoding/cracking tool using various modules. To use it, you need to have basic knowledge of cryptography. Table of Contents

/SNESE_AR\ 33 Nov 27, 2022
Simple macOS StatusBar app to remind you to unplug your laptop when sufficiently charged

ChargeMon Simple macOS StatusBar app to monitor battery charge status and remind you to unplug your Mac when the battery is sufficiently charged Overv

Rhet Turnbull 5 Jan 25, 2022
Your self-hosted bookmark archive. Free and open source.

Your self-hosted bookmark archive. Free and open source. Contents About LinkAce Support Setup Contribution About LinkAce LinkAce is a self-hosted arch

Kevin Woblick 1.7k Jan 03, 2023
Python module to work with Magneto Database directly without using broken Magento 2 core

Python module to work with Magneto Database directly without using broken Magento 2 core

Egor Shitikov 13 Nov 10, 2022
Extend the maya channel box with searchability and colour

channel-box-plus will add search-ability over its attributes, and it will colour user defined attributes, making them easier to distinguish.

Robert Joosten 12 Jun 08, 2022
BOHB tune library template (included example)

BOHB-template 실행 방법 python main.py 2021-10-10 기준 tf keras 버전 (tunecallback 방식) 완료 tf gradienttape 버전 (train_iteration 방식) 완료 pytorch 버전은 구현 준비중 방법 소개

Seungwoo Han 5 Mar 24, 2022
Larvamatch - Find your larva or punk match.

LarvaMatch Find your larva or punk match. UI TBD API (not started) The API will allow you to specify a punk by token id to find a larva match, and vic

1 Jan 02, 2022