Automatizando a criação de DAGs usando Jinja e YAML

Overview

Automatizando a criação de DAGs no Airflow usando Jinja e YAML

Arquitetura do Repo:

  • Pastas por contexto de negócio (ex: Marketing, Analytics, HR, etc), dentro da pasta contexts/;
    • Dentro de cada pasta, temos os YAMLs com as regras da criação da DAG, sendo 1 YAML = 1 DAG;
  • Pasta de templates, uma vez que possuímos diversos operators, gerando a necessidade de diversos templates
  • Pasta com os scripts para criação de DAGs

Como o código funciona

  • A pessoa adiciona/atualiza o arquivo YAML com todas as informações necessárias para criar a DAG ( para isso veja o CONTEXTS_README presente na pasta contexts/ )
  • O script scripts/convert_yaml_in_dag.py precisa ser executado, passando qual o nome do YAML, e ele fará a conversão para uma DAG.

Sobre as pastas:

  • contexts/: Pasta onde ficam armazenados os YAMLs por contexto da empresa.
    Dentro dessa pasta há um README explicando os padrões de criação dos YAMLs

  • templates/: Pasta onde ficam os arquivos .template, que são as bases para o Jinja criar as DAGs.

    • temos o arquivo dag.template
    • temos os operators já adicionados via template
    • temos um README com os operators já adicionados
  • scripts/ : Scripts tanto para transformar YAML em DAG, quanto para validar se o YAML está correto

    • yml_converter.py: Responsável por criar DAGs a partir do YAML
  • dags/: Pasta padrão do airflow, pode ficar nesse repositório ou em outro, ou até em outra branch.

Referencias:

TBD - Próximos passos

  • Automatizar a execução do script convert_yaml_in_dag.py
  • Generalizar o script acima, para mais opções de contextos, criando um input recebendo isto
  • Implementar testes na estrutura dos YAMLs. Uma sugestão é usar a lib cerberus
  • Implementar CI para os testes. Uma sugestão é usar o proprio github_actions
Owner
Arthur Henrique Dell' Antonia
Arthur Henrique Dell' Antonia
This python package provides a simple password reset strategy for django rest framework

Django Rest Password Reset This python package provides a simple password reset strategy for django rest framework, where users can request password r

Anexia 363 Dec 24, 2022
python-social-auth and oauth2 support for django-rest-framework

Django REST Framework Social OAuth2 This module provides OAuth2 social authentication support for applications in Django REST Framework. The aim of th

1k Dec 22, 2022
Pingo provides a uniform API to program devices like the Raspberry Pi, BeagleBone Black, pcDuino etc.

Pingo provides a uniform API to program devices like the Raspberry Pi, BeagleBone Black, pcDuino etc. just like the Python DBAPI provides an uniform API for database programming in Python.

Garoa Hacker Clube 12 May 22, 2022
This program automatically logs you into a Zoom session at your alloted time

This program automatically logs you into a Zoom session at your alloted time. Optionally you can choose to have end the session at your allotted time.

9 Sep 19, 2022
A simple Boilerplate to Setup Authentication using Django-allauth 🚀

A simple Boilerplate to Setup Authentication using Django-allauth, with a custom template for login and registration using django-crispy-forms.

Yasser Tahiri 13 May 13, 2022
Simple extension that provides Basic, Digest and Token HTTP authentication for Flask routes

Flask-HTTPAuth Simple extension that provides Basic and Digest HTTP authentication for Flask routes. Installation The easiest way to install this is t

Miguel Grinberg 1.1k Jan 05, 2023
Skit-auth - Authorization for skit.ai's platform

skit-auth This is a simple authentication library for Skit's platform. Provides

Skit 3 Jan 08, 2022
Graphical Password Authentication System.

Graphical Password Authentication System. This is used to increase the protection/security of a website. Our system is divided into further 4 layers of protection. Each layer is totally different and

Hassan Shahzad 12 Dec 16, 2022
Automatizando a criação de DAGs usando Jinja e YAML

Automatizando a criação de DAGs no Airflow usando Jinja e YAML Arquitetura do Repo: Pastas por contexto de negócio (ex: Marketing, Analytics, HR, etc)

Arthur Henrique Dell' Antonia 5 Oct 19, 2021
A JSON Web Token authentication plugin for the Django REST Framework.

Simple JWT Abstract Simple JWT is a JSON Web Token authentication plugin for the Django REST Framework. For full documentation, visit django-rest-fram

Simple JWT 3.3k Jan 01, 2023
A JSON Web Token authentication plugin for the Django REST Framework.

Simple JWT Abstract Simple JWT is a JSON Web Token authentication plugin for the Django REST Framework. For full documentation, visit django-rest-fram

Jazzband 3.2k Dec 29, 2022
Django-react-firebase-auth - A web app showcasing OAuth2.0 + OpenID Connect using Firebase, Django-Rest-Framework and React

Demo app to show Django Rest Framework working with Firebase for authentication

Teshank Raut 6 Oct 13, 2022
Luca Security Concept

Luca Security Concept This is the document source of luca's security concept. Please go here for the HTML version: https://luca-app.de/securityconcept

luca 43 Oct 22, 2022
Phishing Abusing Microsoft 365 OAuth Authorization Flow

Microsoft365_devicePhish Abusing Microsoft 365 OAuth Authorization Flow for Phishing Attack This is a simple proof-of-concept script that allows an at

bigb0ss 11 Dec 11, 2022
Authentication for Django Rest Framework

Dj-Rest-Auth Drop-in API endpoints for handling authentication securely in Django Rest Framework. Works especially well with SPAs (e.g React, Vue, Ang

Michael 1.1k Jan 03, 2023
python implementation of JSON Web Signatures

python-jws 🚨 This is Unmaintained 🚨 This library is unmaintained and you should probably use For histo

Brian J Brennan 57 Apr 18, 2022
API-key based security utilities for FastAPI, focused on simplicity of use

FastAPI simple security API key based security package for FastAPI, focused on simplicity of use: Full functionality out of the box, no configuration

Tolki 154 Jan 03, 2023
Automatic login utility of free Wi-Fi captive portals

wicafe Automatic login utility of free Wi-Fi captive portals Disclaimer: read and grant the Terms of Service of Wi-Fi services before using it! This u

Takumi Sueda 8 May 31, 2022
A JSON Web Token authentication plugin for the Django REST Framework.

Simple JWT Abstract Simple JWT is a JSON Web Token authentication plugin for the Django REST Framework. For full documentation, visit django-rest-fram

Jazzband 3.2k Dec 28, 2022
Customizable User Authorization & User Management: Register, Confirm, Login, Change username/password, Forgot password and more.

Flask-User v1.0 Attention: Flask-User v1.0 is a Production/Stable version. The previous version is Flask-User v0.6. User Authentication and Management

Ling Thio 997 Jan 06, 2023