A python app which aggregates and splits costs from multiple public cloud providers into a csv

Overview

Cloud Billing

This project aggregates the costs public cloud resources by accounts, services and tags by importing the invoices from public cloud providers and exporting their aggregates in CSV (or another format). It can be used for internal service brokers and allows for aggregating account / subscription / project costs for aws, azure and gcp and additionally allows for tag based aggregation.

It can run as a container and can schedule jobs through kubernetes cronjobs for monthly imports and exports.

It is not a graphical tool to get a drill down view over all costs (such as Azure Cost Management) but rather glue code between cloud provider invoices and internal invoicing systems (such as csv files based system, although other formats could added).

Sample Use Case

A team hosts a kubernetes service and an managed database on a public cloud provider, where multiple application teams run their apps. With this tool, cluster usage and databases can be imported and costs split between the app teams. Each app will be its own service and will receive its fair share of costs. Additionally, the platform team also receives an overview of their total costs. Billing setup is flexible and allows for fixed prices (e.g. each database price is X) and also flexible pricing based on tags and percentages (e.g. the cluster is invoiced for a resource with a tag plus a 10% fee for support).

Features

  • Multiple import options, amongst them azure, gcp and aws
  • Hashicorp Vault Integration, read secrets from Hashicorp Vault
  • Flexible billing with fixed prices and percentage-based prices
  • Allows invoicing of projects without tags
  • Informs service customers about service costs with a monthly email

Architecture

Architecture

Install & Run

Requirements

  • python3
  • pip
  • virtualenv
  • sqlite (or mysql for production)
  • mariadb-client

Install Requirements on Ubuntu

Some libraries for the DB connectors are required (mssql and mysql): sudo apt-get install mariadb-client libmariadbclient-dev unixodbc unixodbc-dev

Install Python Packages Requirements

pip install -r requirements.txt
pip install -r requirements-test.txt

Run locally with sqlite

# Import Sample Data from Azure from ./sample_files
python run.py -e env/dev.env import AzureImport -n azure

# Update Prices and Percentages
# Run command below and navigate to: http://localhost:5000/account/
python run.py -e ./env/dev.env run

# Export Accounts to local CSV and check the Azure.csv afterwards on the local filesystem
python run.py -e env/dev.env bill Azure -n bill_azure

# Update Services and attach them to a service customer
# Run below command and navigate to: http://localhost:5000/service/
# Set a a service customer in the service customer column and set a random string in the column left to it
python run.py -e ./env/dev.env run

# Now create an invoice csv file for the managed services and check the ManagedCloudServices.csv file on the local filesystem:
python run.py -e env/dev.env bill ManagedCloudServices -n bill_msc

Run Tests and Coverage

Tests are run with pytest and pymock. See /src/tests and /conftest.py for setup.

pytest
coverage run --source ./src -m pytest
coverage report

Configuration

Configuration is done within the env files in /env or through environment variables. Database settings are loaded in /src/settings.py.

Deployment

Ideally in a kubernetes environment where cronjobs can be created automatically from the interface. A Dockerfile can be found in the repository. Environment vars in ./env can also be added from configmaps within kubernetes.

See /k8s_manifests folder for sample deployment manifests. Setup is divided into:

  • _ns (Namespace)
  • _secret (Secrets for Connection to the Cloud Providers and the Mysql database)
  • _configmap (Configuration Details)
  • _deployment (Deployment of the app and ingresses and services)
Spacegit is a .git exposed finder

Spacegit Spacegit is a basic .git exposed finder Usage: You need python3 installed to run spacegit use: python3 spacegit.py (url) Disclaimer: **This i

2 Nov 30, 2021
A python module for extract domains

A python module for extract domains

Fayas Noushad 4 Aug 10, 2022
RapidFuzz is a fast string matching library for Python and C++

RapidFuzz is a fast string matching library for Python and C++, which is using the string similarity calculations from FuzzyWuzzy

Max Bachmann 1.7k Jan 04, 2023
A multipurpose python module

pysherlock pysherlock is a Python library for dealing with web scraping using images, it's a Python application of the rendertron headless browser API

Sachit 2 Nov 11, 2021
Python Yeelight YLKG07YL/YLKG08YL dimmer handler

With this class you can receive, decrypt and handle Yeelight YLKG07YL/YLKG08YL dimmer bluetooth notifications in your python code.

12 Dec 26, 2022
DUQ is a python package for working with physical Dimensions, Units, and Quantities.

DUQ is a python package for working with physical Dimensions, Units, and Quantities.

2 Nov 02, 2022
A module for account creation with python

A module for account creation with python

Fayas Noushad 3 Dec 01, 2021
Script to autocompound 3commas BO:SO based on user provided risk factor

3commas_compounder Script to autocompound 3commas BO:SO based on user provided risk factor Setup Step 1 git clone this repo into your working director

0 Feb 24, 2022
Python Libraries with functions and constants related to electrical engineering.

ElectricPy Electrical-Engineering-for-Python Python Libraries with functions and constants related to electrical engineering. The functions and consta

Joe Stanley 39 Dec 23, 2022
A Python utility belt containing simple tools, a stdlib like feel, and extra batteries. Hashing, Caching, Timing, Progress, and more made easy!

Ubelt is a small library of robust, tested, documented, and simple functions that extend the Python standard library. It has a flat API that all behav

Jon Crall 638 Dec 13, 2022
JavaScript to Python Translator & JavaScript interpreter written in 100% pure Python🚀

Pure Python JavaScript Translator/Interpreter Everything is done in 100% pure Python so it's extremely easy to install and use. Supports Python 2 & 3.

Piotr Dabkowski 2.1k Dec 30, 2022
Helpful functions for use alongside the rich Python library.

🔧 Rich Tools A python package with helpful functions for use alongside with the rich python library. 󠀠󠀠 The current features are: Convert a Pandas

Avi Perl 14 Oct 14, 2022
A python mathematics module

A python mathematics module

Fayas Noushad 4 Nov 28, 2021
Dynamic key remapper for Wayland Window System, especially for Sway

wayremap Dynamic keyboard remapper for Wayland. It works on both X Window Manager and Wayland, but focused on Wayland as it intercepts evdev input and

Kay Gosho 50 Nov 29, 2022
Simple integer-valued time series bit packing

Smahat allows to encode a sequence of integer values using a fixed (for all values) number of bits but minimal with regards to the data range. For example: for a series of boolean values only one bit

Ghiles Meddour 7 Aug 27, 2021
Standard implementations of FedLab and its provided benchmarks.

FedLab-benchmarks This repo contains standard implementations of FedLab and its provided benchmarks. Currently, following algorithms or benchrmarks ar

SMILELab-FL 104 Dec 05, 2022
Simple python module to get the information regarding battery in python.

Battery Stats A python3 module created for easily reading the current parameters of Battery in realtime. It reads battery stats from /sys/class/power_

Shreyas Ashtamkar 5 Oct 21, 2022
Check the basic quality of any dataset

Data Quality Checker in Python Check the basic quality of any dataset. Sneak Peek Read full tutorial at Medium. Explore the app Requirements python 3.

MalaDeep 8 Feb 23, 2022
Runes - Simple Cookies You Can Extend (similar to Macaroons)

Runes - Simple Cookies You Can Extend (similar to Macaroons) is a paper called "Macaroons: Cookies with Context

Rusty Russell 22 Dec 11, 2022
Lark is a parsing toolkit for Python, built with a focus on ergonomics, performance and modularity.

Lark is a parsing toolkit for Python, built with a focus on ergonomics, performance and modularity.

Lark - Parsing Library & Toolkit 3.5k Jan 05, 2023