Python command line tool and python engine to label table fields and fields in data files.

Overview

Metacrafter

Python command line tool and python engine to label table fields and fields in data files. It could help to find meaningful data in your tables and data files or to find Personal identifable information (PII).

Installation

To install Python library use pip install metacrafter via pip or python setup.py install

Features

Metacrafter is a rule based tool that helps to label fields of the tables in databases. It scans table and finds person names, surnames, midnames, PII data, basic identifiers like UUID/GUID. These rules written as .yaml files and could be easily extended.

File formats supported:

  • CSV
  • JSON lines
  • JSON (array of dicts)
  • BSON
  • Parquet

Databases support:

Metacrafter key features:

  • 25 basic and PII rules.
  • all labels metadata collected into Metacrafter registry public repository -
  • 312 date detection rules/patterns, date detection using qddate, "quick and dirty" date detection library
  • extendable set of rules using PyParsing, exact text match and validation functions
  • support any database supported by SQLAlchemy
  • advanced context and language management. You could apply only rules relevant to certain data of choosen language
  • built-in API server
  • commercial support and additional rules available

Command line examples

File analysis examples

# Scan CSV file
$ metacrafter scan-file --format short somefile.csv

# Scan CSV file with delimiter ';' and windows-1251 encoding
$ metacrafter scan-file --format short --encoding windows-1251 --delimiter ';' somefile.csv

# Scan JSON lines file, output results as stats table to file file
$ metacrafter scan-file --format stats -o somefile_result.json somefile.jsonl

Result example of 'full' type of formatting

key                   ftype    tags    matches
--------------------  -------  ------  -------------------------------
name                  str      uniq
addressresidence      str      uniq    address 59.80
addressactivities     str              address 50.98
addressobjects        str              address 28.00
bin                   int              ogrn 99.02
inn                   str              inn 100.00,inn 99.02
purposeaudit          str              runpa 8.82
dateregistration      str              datetime 94.12 (dt:date:date_2)
expirydate            str              datetime 18.63 (dt:date:date_2)
startdateactivity     str              datetime 28.43 (dt:date:date_2)
othergrounds          str      dict
startdateaudit        str              datetime 65.69 (dt:date:date_2)
workdays              int      dict
workhours             str      dict
formaudit             str      dict
namestatecontrol      str
assignment decree     str      dict
effectivedate         str      dict
Inspectionenddate     str      empty
riskcategory          str      dict
expirationdate        str      empty
startupnotifications  str      empty
daylastcheck          str      empty
otherreasonsrefusal   str      empty
numbersystem          str      empty

Database analysis examples

# Scan MongoDB database 'fns', save results as result.json and format output as 'stats'
$ metacrafter scan-mongodb --dbname fns -o result.json -f full

# Scan Postgres database 'dbname', with schema 'public'.
$ metacrafter scan-db --schema public --connstr postgresql+psycopg2://username:[email protected]:15432/dbname

Rules

All rules described as YAML files and by default rules loaded from directory 'rules' or from list of directories provided in .metacrafter file with YAML format

All rules could be applied to fields or data .

Compare engines defined in match parameter in rule description:

  • text - scan text for exact match to one of text values. Text values delimited by comma (',')
  • ppr - scan text for PyParsing. PyParsing rule defined as Python code with PyParsing objects like Word(nums, exact=4)
  • func - scan text using Python function provided. Function shoud accept one string parameter and shoud return True or False

How to write rules

Function (func)

Example Russian administrative legal act/law matched by custom function

  runpabyfunc:
    key: runpa
    name: Russian legal act / law
    maxlen: 500
    minlen: 3
    priority: 1
    match: func
    type: data
    rule: metacrafter.rules.ru.gov.is_ru_law

Exact text match (text)

Example midname matching by exact field name

  midname:
    key: person_midname
    name: Person midname by known
    rule: midname,secondname,middlename,mid_name,middle_name
    type: field
    match: text

PyParsing rule (ppr)

Example Russian cadastral number

  rukadastr:
    key: rukadastr
    name: Russian land territory cadastral identifier
    rule: Word(nums, min=1, max=2) + Literal(':').suppress() + Word(nums, min=1, max=2) + Literal(':').suppress() + Word(nums, min=6, max=7) + Literal(':').suppress() + Word(nums, min=1, max=6)
    maxlen: 20
    minlen: 12
    priority: 1
    match: ppr
    type: data

Commercial support

Please write [email protected] or [email protected] to request beta access to commercial API. Commercial API support 195 fields and data rules and provided with dedicated support.

Comments
  • (sqlite3.OperationalError) no such module: VirtualSpatialIndex

    (sqlite3.OperationalError) no such module: VirtualSpatialIndex

    Error processing 008564_pal_features_v3.sqlite - (sqlite3.OperationalError) no such module: VirtualSpatialIndex [SQL: SELECT * FROM 'SpatialIndex' LIMIT 10000] (Background on this error at: http://sqlalche.me/e/13/e3q8)

    File 008564_pal_features_v3.zip

    bug 
    opened by ivbeg 2
  • Can I apply rules (eg pii) during scan-db

    Can I apply rules (eg pii) during scan-db

    I have successfully run scan-db against my database.

    I want to run scan-db with the pii rule but cannot see how this is possible from the examples. Is there an option to do this?

    Many thanks

    opened by ian-lewis-d 1
  • Consider adding boolean rules with prefixes

    Consider adding boolean rules with prefixes "is_" and "has_", 'was_', postfixes "flag" and e.t.c.

    From GitSchemas datasets analysis consider adding rules:

    • [ ] prefix based with prefixes: "is_" and "has_", "show_", 'was_'
    • [ ] name based with names: "deleted", "enabled", "approved", "active"
    • [ ] postfix based with postfixes: "_flag"

    Additional verification should include that field has no more than 2 values (yes or no) or 3 values including NULL (yes, no, None).

    enhancement 
    opened by ivbeg 0
  • Is there an integration for Datahub?

    Is there an integration for Datahub?

    Hi,

    I'm in the process of setting up Datahub (https://datahubproject.io) at our organisation and I wanted to know if there is a way to load the Metacrafter PII labels onto entities in Datahub?

    Many thanks, Ian

    enhancement 
    opened by ian-lewis-d 1
  • Add schema for report JSON and improve reporting

    Add schema for report JSON and improve reporting

    Right now JSON file of the metadata scanning report is not structured well enough. Improvements should include:

    • [ ] Add Cerberus schema (more info https://docs.python-cerberus.org)
    • [ ] Add scanning datetime
    • [ ] Add source info: source type, filename, connection string e.t.c. Make sure no secrets in connection string
    • [ ] Move 'table' to 'source' subtag
    • [ ] Add tests to validate reports with Cerberus validator
    enhancement 
    opened by ivbeg 0
  • Add support of NoSQL databases

    Add support of NoSQL databases

    Add support for the following NoSQL databases and search engines: MongoDB, ArangoDB, Milvus, ArcadeDB, ElasticSearch, OpenSearch, MeiliSearch, Apache Cassandra, StarGate (MongoDB-like API over NoSQL databases)

    The current state of database support:

    • [x] MongoDB
    • [ ] ArangoDB
    • [ ] ElasticSearch
    • [ ] Meilisearch
    • [ ] Milvus
    • [ ] OpenSearch
    • [ ] ArcadeDB

    Other tasks:

    • [ ] Write universal class for NoSQL document based databases
    • [ ] Replace command-line command 'scan-mongodb' with 'scan-nosql' or update command 'scan-db' with NoSQL databases connection strings
    • [ ] Write documentation with connection strings examples and limitations
    • [ ] Write tests for each database type
    enhancement 
    opened by ivbeg 0
  • Object of type bytes is not JSON serializable - Error processing some SQLite files

    Object of type bytes is not JSON serializable - Error processing some SQLite files

    Error Object of type bytes is not JSON serializable caused by table fields with bytes type. Better detection of types needed and serialization of bytes type in JSON report. Error caused not by processing, but by reporting function.

    Example 000001_run-546.zip

    bug 
    opened by ivbeg 1
  • sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) Could not decode to UTF-8 column

    sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) Could not decode to UTF-8 column

    Error processing SQLite database with non-unicode names for fields. Example 000012_world.zip

    `Traceback (most recent call last): File "C:\Users\ibegt\AppData\Roaming\Python\Python310\site-packages\sqlalchemy\engine\result.py", line 1284, in fetchall l = self.process_rows(self._fetchall_impl()) File "C:\Users\ibegt\AppData\Roaming\Python\Python310\site-packages\sqlalchemy\engine\result.py", line 1230, in _fetchall_impl return self.cursor.fetchall() sqlite3.OperationalError: Could not decode to UTF-8 column 'name' with text '\ufffdland Islands'

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last): File "C:\Program Files\Python310\Scripts\metacrafter-script.py", line 33, in sys.exit(load_entry_point('metacrafter==0.0.2', 'console_scripts', 'metacrafter')()) File "C:\Program Files\Python310\lib\site-packages\metacrafter-0.0.2-py3.10.egg\metacrafter_main_.py", line 12, in main exit_status = cli() File "C:\Program Files\Python310\lib\site-packages\click\core.py", line 1130, in call return self.main(*args, **kwargs) File "C:\Program Files\Python310\lib\site-packages\click\core.py", line 1055, in main rv = self.invoke(ctx) File "C:\Program Files\Python310\lib\site-packages\click\core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Program Files\Python310\lib\site-packages\click\core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "C:\Program Files\Python310\lib\site-packages\click\core.py", line 760, in invoke return __callback(*args, **kwargs) File "C:\Program Files\Python310\lib\site-packages\metacrafter-0.0.2-py3.10.egg\metacrafter\core.py", line 464, in scan_db acmd.scan_db( File "C:\Program Files\Python310\lib\site-packages\metacrafter-0.0.2-py3.10.egg\metacrafter\core.py", line 359, in scan_db items = [dict(u) for u in queryres.fetchall()] File "C:\Users\ibegt\AppData\Roaming\Python\Python310\site-packages\sqlalchemy\engine\result.py", line 1288, in fetchall self.connection.handle_dbapi_exception( File "C:\Users\ibegt\AppData\Roaming\Python\Python310\site-packages\sqlalchemy\engine\base.py", line 1510, in handle_dbapi_exception util.raise( File "C:\Users\ibegt\AppData\Roaming\Python\Python310\site-packages\sqlalchemy\util\compat.py", line 182, in raise raise exception File "C:\Users\ibegt\AppData\Roaming\Python\Python310\site-packages\sqlalchemy\engine\result.py", line 1284, in fetchall l = self.process_rows(self._fetchall_impl()) File "C:\Users\ibegt\AppData\Roaming\Python\Python310\site-packages\sqlalchemy\engine\result.py", line 1230, in _fetchall_impl return self.cursor.fetchall() sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) Could not decode to UTF-8 column 'name' with text '\ufffdland Islands' (Background on this error at: http://sqlalche.me/e/13/e3q8) `

    bug 
    opened by ivbeg 1
Releases(second)
Owner
APICrafter
APICrafter Data API project
APICrafter
A CLI application that downloads your AC submissions from OJ's like Atcoder,Codeforces,CodeChef and distil it into beautiful Submission HeatMap.

Yoda A CLI that takes away the hassle of managing your submission files on different online-judges by automating the entire process of collecting and organizing your code submissions in one single Di

Nikhar Manchanda 1 Jul 28, 2022
ctree - command line christmas tree

ctree ctree - command line christmas tree It is small python script that prints a christmas tree in terminal. It is colourful and always gives you a d

15 Aug 15, 2022
A Python3 rewrite of my original PwnedConsole project from almost a decade ago

PwnedConsoleX A CLI shell for performing queries against the HaveIBeenPwned? API to gather breach information for user-supplied email addresses. | wri

1 Jul 23, 2022
Code for "Salient Deconvolutional Networks, Aravindh Mahendran, Andrea Vedaldi, ECCV 2016"

deconvnet_analysis Code for "Salient Deconvolutional Networks, Aravindh Mahendran, Andrea Vedaldi, ECCV 2016" Parts of this code Generate figures in t

Aravindh Mahendran 12 Jan 25, 2021
As easy as /aitch-tee-tee-pie/ 🥧 Modern, user-friendly command-line HTTP client for the API era. JSON support, colors, sessions, downloads, plugins & more. https://twitter.com/httpie

HTTPie: human-friendly CLI HTTP client for the API era HTTPie (pronounced aitch-tee-tee-pie) is a command-line HTTP client. Its goal is to make CLI in

HTTPie 25.4k Dec 30, 2022
Tablicate - Python library for easy table creation and output to terminal

Tablicate Tablicate - Python library for easy table creation and output to terminal Features Column-wise justification alignment (left, right, center)

3 Dec 14, 2022
Quickly open any path on your terminal window in your $EDITOR of choice!

Tmux fpp Plugin wrapper around Facebook PathPicker. Quickly open any path on your terminal window in your $EDITOR of choice! Demo Dependencies fpp - F

257 Dec 28, 2022
Simple Tool To Grab Like-Card Coupon

Simple Tool To Grab Like-Card Coupon

Soud 10 Jan 30, 2022
AlienFX is a CLI and GUI utility to control the lighting effects of your Alienware computer.

AlienFX is a Linux utility to control the lighting effects of your Alienware computer. At present there is a CLI version (alienfx) and a gtk GUI versi

Stephen Harris 218 Dec 26, 2022
A simple web-based SSH client.

Kommander A simple web-based SSH client. It supports: entering SSH login details (including private key and custom ports) and connecting user authenti

KingWaffleIII 2 Jan 01, 2022
Wordle-cli - Command-line clone of Josh Wardle's WORDLE

Command-line clone of Josh Wardle's WORDLE, inspired by Paul Battley's Ruby vers

Klipspringer 32 Jan 03, 2023
Simple CLI interface for linear task manager

Linear CLI (Unmaintained) Simple CLI interface for linear task manager Usage Install: pip install linearcli Setup: Generate a pe

Mike Lyons 1 Jan 07, 2022
Shellcode runner to execute malicious payload and bypass AV

buffshark-shellcode-runner Python Shellcode Runner to execute malicious payload and bypass AV This script utilizes mmap(for linux) and win api wrapper

Momo Lenard 9 Dec 29, 2022
A terminal UI dashboard to monitor requests for code review across Github and Gitlab repositories.

A terminal UI dashboard to monitor requests for code review across Github and Gitlab repositories.

Kyle Harrison 150 Dec 14, 2022
Generate folder trees directly from the terminal.

Dir Tree Artist 🎨 🌲 Intro Easily view folder structure, with parameters to sieve out what you want. Choose to exclude files from being viewed (.git,

Glenda T 0 May 17, 2022
This is a command line program to play cricket made using Python.

SimpleCricketPython This is a command line program to play cricket made using Python How it works First you have the option of selecting whether you

Imira Randeniya 1 Sep 11, 2022
A next-generation CLI and TUI that aims to be your personal assistant for everything competitive programming related. 🚀

Competitive Programming Tool Kit The Competitive Programming Tool Kit (cptk for short), is a command line and terminal user interface (CLI and TUI) th

Alon 4 May 21, 2022
A powerful Minecraft command library.

Mecha A powerful Minecraft command library. from mecha import Mecha

32 Dec 10, 2022
CLI Web-CAT interface for people who use VIM.

CLI Web-CAT CLI Web-CAT interface. Installation git clone https://github.com/phuang1024/cliwebcat cd cliwebcat python setup.py bdist_wheel sdist cd di

Patrick 4 Apr 11, 2022
Python remote shell.

Python remote shell.

Steven 9 Oct 17, 2022