sqlconfig: manage your config files with sqlite

Overview

sqlconfig: manage your config files with sqlite

The problem

Your app probably has a lot of configuration in git. Storing it as files in a git repo has a lot of advantages, including:

  • Diffing
  • Rollbacks
  • Blame
  • Branching

However, flat files in a repo can get unwieldy:

  • Different files need to be kept in sync with each other
  • Bulk operations are challenging
  • It's easy to add invalid data
  • They're disorganized

sqlconfig gives you all the advantages of config files stored in version control, with the power, flexibility and safety of SQL.

The solution

Model your config as a SQLite database. sqlconfig can deterministically turn that database into diffable flat files that live in the repo and back again.

Tutorial

Install sqlconfig. You need Python 3.

$ pip install sqlconfig

Next, design your config in SQLite. For this example, we'll use Twitch's unfairly ridiculued spam system.

create table keyword_categories (id integer primary key, name text not null unique); sqlite> create table keywords (id integer primary key, category_id integer not null references keyword_categories(id), keyword text unique); sqlite> insert into keyword_categories (name) values ("spam"),("hate"),("false_positives"); sqlite> insert into keywords (category_id, keyword) values (1, "viagra"),(1, "nigerian prince"),(2, "suck"),(2, "jerk"),(3, "fanny hands lane"); ">
$ sqlconfig --shell --dir example_config --overwrite
Running shell in read-write mode.
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> create table keyword_categories (id integer primary key, name text not null unique);
sqlite> create table keywords (id integer primary key, category_id integer not null references keyword_categories(id), keyword text unique);
sqlite> insert into keyword_categories (name) values ("spam"),("hate"),("false_positives");
sqlite> insert into keywords (category_id, keyword) values (1, "viagra"),(1, "nigerian prince"),(2, "suck"),(2, "jerk"),(3, "fanny hands lane");

We can see that sqlconfig has created files on disk:

$ ls example_config/
keyword_categories.json  keywords.json  schema.sql

Note that the JSON files are printed deterministically and in a format that's easy for git diff to work with.

Note one of the advantages of using SQL is that it's harder to write a bad config. For example, if we delete a keyword category without deleting all the keywords in it, we get an error.

delete from keyword_categories where id=1; sqlite> error: 2 rows failed foreign key integrity checks. Run "pragma foreign_key_check" in the sqlite shell for more information. ">
$ sqlconfig --shell --dir example_config --overwrite
Running shell in read-write mode.
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> delete from keyword_categories where id=1;
sqlite>
error: 2 rows failed foreign key integrity checks. Run "pragma foreign_key_check" in the sqlite shell for more information.

Reading your config

It's recommended to put your schema.sql and .json files in a git repository and to review changes to them as part of your normal code review or pull request process. Through CI, you can publish this repo to your configuration management system of choice.

If you don't want to read the files directly, you can also read them with SQL. Either via the command line:

$ sqlconfig --shell --dir example_config -- -header -csv 'select keyword from keywords where category_id=1'
Running shell in read-only mode. Pass --overwrite to save your changes
keyword
viagra
"nigerian prince"

or by exporting a SQLite database to disk with --load and reading it from your application or another tool.

$ sqlconfig --load --db config.sqlite --dir example_config
$ sqlite3 config.sqlite -header -csv 'select keyword from keywords where category_id=1'
keyword
viagra
"nigerian prince"

Best practices

It's best to put the schema.sql and

.jsonsqlconfig

$ sqlconfig --check --dir example_config/
OK

API

You can use sqlconfig.lib.load(db, dir) and sqlconfig.lib.dump(db, dir) to programmatically perform these operations. Read the code if you have any questions :)

files in a git repo, and use a CI job to push the config to whatever system should serve it. Before doing so, it's a good idea to use to validate the configuration:
Owner
Pete Hunt
CEO @ Smyte, ex-React team at Facebook
Pete Hunt
A compact library for Python 3.10x that allows users to configure their SimPads real-time

SimpadLib v1.0.6 What is this? This is a python library programmed by Ashe Muller that allows users to interface directly with their SimPad devices, a

Ashe Muller 2 Jan 08, 2022
A set of Python scripts and notebooks to help administer and configure Workforce projects.

Workforce Scripts A set of Python scripts and notebooks to help administer and configure Workforce projects. Notebooks Several example Jupyter noteboo

Esri 75 Sep 09, 2022
Dynamic Django settings.

Constance - Dynamic Django settings A Django app for storing dynamic settings in pluggable backends (Redis and Django model backend built in) with an

Jazzband 1.5k Jan 04, 2023
environs is a Python library for parsing environment variables.

environs: simplified environment variable parsing environs is a Python library for parsing environment variables. It allows you to store configuration

Steven Loria 920 Jan 04, 2023
Python YAML Environment (ymlenv) by Problem Fighter Library

In the name of God, the Most Gracious, the Most Merciful. PF-PY-YMLEnv Documentation Install and update using pip: pip install -U PF-PY-YMLEnv Please

Problem Fighter 2 Jan 20, 2022
Python Marlin Configurator to make valid configuration files to be used to compile Marlin with.

marlin-configurator Concept originally imagined by The-EG using PowerShell Build Script for Marlin Configurations The purpose of this project is to pa

DevPeeps 2 Oct 09, 2021
Flexible Python configuration system. The last one you will ever need.

OmegaConf Description Project Code quality Docs and support OmegaConf is a hierarchical configuration system, with support for merging configurations

Omry Yadan 1.4k Jan 02, 2023
Hydra is a framework for elegantly configuring complex applications

Python Decouple: Strict separation of settings from code Decouple helps you to organize your settings so that you can change parameters without having

Facebook Research 6.6k Jan 04, 2023
sqlconfig: manage your config files with sqlite

sqlconfig: manage your config files with sqlite The problem Your app probably has a lot of configuration in git. Storing it as files in a git repo has

Pete Hunt 4 Feb 21, 2022
Kubernates Config Manager

Kubernates Config Manager Sometimes we need manage more than one kubernates cluster at the same time. Switch cluster configs is a dangerous and troubl

周文阳 3 Jan 10, 2022
A modern simfile parsing & editing library for Python 3

A modern simfile parsing & editing library for Python 3

ash garcia 38 Nov 01, 2022
Python 3+ compatible port of the configobj library

configobj Python 3+ compatible port of the configobj library. Documentation You can find a full manual on how to use ConfigObj at readthedocs. If you

Differently Sized Kittens 288 Dec 14, 2022
A slightly opinionated template for iPython configuration for interactive development

A slightly opinionated template for iPython configuration for interactive development. Auto-reload and no imports for packages and modules in the project.

Seva Zhidkov 24 Feb 16, 2022
This Ivy plugin adds support for TOML file headers.

This Ivy plugin adds support for TOML file headers as an alternative to YAML.

Darren Mulholland 1 Nov 09, 2021
Simple dataclasses configuration management for Python with hocon/json/yaml/properties/env-vars/dict support.

Simple dataclasses configuration management for Python with hocon/json/yaml/properties/env-vars/dict support, based on awesome and lightweight pyhocon parsing library.

Teo Stocco 62 Dec 23, 2022
A helper for organizing Django project settings by relying on well established programming patterns.

django-configurations django-configurations eases Django project configuration by relying on the composability of Python classes. It extends the notio

Jazzband 955 Jan 05, 2023
Dag-bakery - Dag Bakery enables the capability to define Airflow DAGs via YAML.

DAG Bakery - WIP 🔧 dag-bakery aims to simplify our DAG development by removing all the boilerplate and duplicated code when defining multiple DAG cro

Typeform 2 Jan 08, 2022
Apt2sbom python package generates SPDX or YAML files

Welcome to apt2sbom This package contains a library and a CLI tool to convert a Ubuntu software package inventory to a software bill of materials. You

Eliot Lear 15 Nov 13, 2022
ConfZ is a configuration management library for Python based on pydantic.

ConfZ – Pydantic Config Management ConfZ is a configuration management library for Python based on pydantic. It easily allows you to load your configu

Zühlke 164 Dec 27, 2022
Django-environ allows you to utilize 12factor inspired environment variables to configure your Django application.

Django-environ django-environ allows you to use Twelve-factor methodology to configure your Django application with environment variables. import envi

Daniele Faraglia 2.7k Jan 03, 2023