sqlconfig: manage your config files with sqlite

Overview

sqlconfig: manage your config files with sqlite

The problem

Your app probably has a lot of configuration in git. Storing it as files in a git repo has a lot of advantages, including:

  • Diffing
  • Rollbacks
  • Blame
  • Branching

However, flat files in a repo can get unwieldy:

  • Different files need to be kept in sync with each other
  • Bulk operations are challenging
  • It's easy to add invalid data
  • They're disorganized

sqlconfig gives you all the advantages of config files stored in version control, with the power, flexibility and safety of SQL.

The solution

Model your config as a SQLite database. sqlconfig can deterministically turn that database into diffable flat files that live in the repo and back again.

Tutorial

Install sqlconfig. You need Python 3.

$ pip install sqlconfig

Next, design your config in SQLite. For this example, we'll use Twitch's unfairly ridiculued spam system.

create table keyword_categories (id integer primary key, name text not null unique); sqlite> create table keywords (id integer primary key, category_id integer not null references keyword_categories(id), keyword text unique); sqlite> insert into keyword_categories (name) values ("spam"),("hate"),("false_positives"); sqlite> insert into keywords (category_id, keyword) values (1, "viagra"),(1, "nigerian prince"),(2, "suck"),(2, "jerk"),(3, "fanny hands lane"); ">
$ sqlconfig --shell --dir example_config --overwrite
Running shell in read-write mode.
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> create table keyword_categories (id integer primary key, name text not null unique);
sqlite> create table keywords (id integer primary key, category_id integer not null references keyword_categories(id), keyword text unique);
sqlite> insert into keyword_categories (name) values ("spam"),("hate"),("false_positives");
sqlite> insert into keywords (category_id, keyword) values (1, "viagra"),(1, "nigerian prince"),(2, "suck"),(2, "jerk"),(3, "fanny hands lane");

We can see that sqlconfig has created files on disk:

$ ls example_config/
keyword_categories.json  keywords.json  schema.sql

Note that the JSON files are printed deterministically and in a format that's easy for git diff to work with.

Note one of the advantages of using SQL is that it's harder to write a bad config. For example, if we delete a keyword category without deleting all the keywords in it, we get an error.

delete from keyword_categories where id=1; sqlite> error: 2 rows failed foreign key integrity checks. Run "pragma foreign_key_check" in the sqlite shell for more information. ">
$ sqlconfig --shell --dir example_config --overwrite
Running shell in read-write mode.
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> delete from keyword_categories where id=1;
sqlite>
error: 2 rows failed foreign key integrity checks. Run "pragma foreign_key_check" in the sqlite shell for more information.

Reading your config

It's recommended to put your schema.sql and .json files in a git repository and to review changes to them as part of your normal code review or pull request process. Through CI, you can publish this repo to your configuration management system of choice.

If you don't want to read the files directly, you can also read them with SQL. Either via the command line:

$ sqlconfig --shell --dir example_config -- -header -csv 'select keyword from keywords where category_id=1'
Running shell in read-only mode. Pass --overwrite to save your changes
keyword
viagra
"nigerian prince"

or by exporting a SQLite database to disk with --load and reading it from your application or another tool.

$ sqlconfig --load --db config.sqlite --dir example_config
$ sqlite3 config.sqlite -header -csv 'select keyword from keywords where category_id=1'
keyword
viagra
"nigerian prince"

Best practices

It's best to put the schema.sql and

.jsonsqlconfig

$ sqlconfig --check --dir example_config/
OK

API

You can use sqlconfig.lib.load(db, dir) and sqlconfig.lib.dump(db, dir) to programmatically perform these operations. Read the code if you have any questions :)

files in a git repo, and use a CI job to push the config to whatever system should serve it. Before doing so, it's a good idea to use to validate the configuration:
Owner
Pete Hunt
CEO @ Smyte, ex-React team at Facebook
Pete Hunt
Python 3+ compatible port of the configobj library

configobj Python 3+ compatible port of the configobj library. Documentation You can find a full manual on how to use ConfigObj at readthedocs. If you

Differently Sized Kittens 288 Dec 14, 2022
Event Coding for the HV Protocol MEG datasets

Scripts for QA and trigger preprocessing of NIMH HV Protocol Install pip install git+https://github.com/nih-megcore/hv_proc Usage hv_process.py will

2 Nov 14, 2022
filetailor is a peer-based configuration management utility for plain-text files such as dotfiles.

filetailor filetailor is a peer-based configuration management utility for plain-text files (and directories) such as dotfiles. Files are backed up to

5 Dec 23, 2022
A modern simfile parsing & editing library for Python 3

A modern simfile parsing & editing library for Python 3

ash garcia 38 Nov 01, 2022
A set of Python scripts and notebooks to help administer and configure Workforce projects.

Workforce Scripts A set of Python scripts and notebooks to help administer and configure Workforce projects. Notebooks Several example Jupyter noteboo

Esri 75 Sep 09, 2022
This Ivy plugin adds support for TOML file headers.

This Ivy plugin adds support for TOML file headers as an alternative to YAML.

Darren Mulholland 1 Nov 09, 2021
Sync any your configuration file to remote. Currently only support gist.

Sync your configuration to remote, such as vimrc. You can use EscSync to manage your configure of editor, shell, etc.

Me1onRind 0 Nov 21, 2022
Pydantic-ish YAML configuration management.

Pydantic-ish YAML configuration management.

Dribia Data Research 18 Oct 27, 2022
Read configuration settings from python configuration files.

Maison Read configuration settings from python configuration files. Motivation When developing a python application, e.g a command-line tool, it can b

9 Jan 04, 2023
MOHAconfig - Gerador de arquivo de configuração para Medal of Honor: Airborne

MOHAconfig Gerador de arquivo de configuração para Medal of Honor: Airborne MOHA - Gerador de arquivo de configuração. Essa aplicação foi feita em pyt

1 Dec 31, 2021
A Python library to parse PARI/GP configuration and header files

pari-utils A Python library to parse PARI/GP configuration and header files. This is mainly used in the code generation of https://github.com/sagemath

Sage Mathematical Software System 3 Sep 18, 2022
Load Django Settings from Environmental Variables with One Magical Line of Code

DjEnv: Django + Environment Load Django Settings Directly from Environmental Variables features modify django configuration without modifying source c

Daniel J. Dufour 28 Oct 01, 2022
environs is a Python library for parsing environment variables.

environs: simplified environment variable parsing environs is a Python library for parsing environment variables. It allows you to store configuration

Steven Loria 920 Jan 04, 2023
sqlconfig: manage your config files with sqlite

sqlconfig: manage your config files with sqlite The problem Your app probably has a lot of configuration in git. Storing it as files in a git repo has

Pete Hunt 4 Feb 21, 2022
Inject your config variables into methods, so they are as close to usage as possible

Inject your config variables into methods, so they are as close to usage as possible

GDWR 7 Dec 14, 2022
An application pulls configuration information from JSON files generated

AP Provisioning Automation An application pulls configuration information from JSON files generated by Ekahau and then uses Netmiko to configure the l

Cisco GVE DevNet Team 1 Dec 17, 2021
A small example project for efficiently configuring a Python application with YAMLs and the CLI

Hydra Example Project for Python A small example project for efficiently configuring a Python application with YAMLs and the CLI. Why should I care? A

Florian Wilhelm 4 Dec 31, 2022
Dag-bakery - Dag Bakery enables the capability to define Airflow DAGs via YAML.

DAG Bakery - WIP 🔧 dag-bakery aims to simplify our DAG development by removing all the boilerplate and duplicated code when defining multiple DAG cro

Typeform 2 Jan 08, 2022
Organize Django settings into multiple files and directories. Easily override and modify settings. Use wildcards and optional settings files.

Organize Django settings into multiple files and directories. Easily override and modify settings. Use wildcards in settings file paths and mark setti

Nikita Sobolev 942 Jan 05, 2023
Apt2sbom python package generates SPDX or YAML files

Welcome to apt2sbom This package contains a library and a CLI tool to convert a Ubuntu software package inventory to a software bill of materials. You

Eliot Lear 15 Nov 13, 2022