Automatizando a criação de DAGs usando Jinja e YAML

Overview

Automatizando a criação de DAGs no Airflow usando Jinja e YAML

Arquitetura do Repo:

  • Pastas por contexto de negócio (ex: Marketing, Analytics, HR, etc), dentro da pasta contexts/;
    • Dentro de cada pasta, temos os YAMLs com as regras da criação da DAG, sendo 1 YAML = 1 DAG;
  • Pasta de templates, uma vez que possuímos diversos operators, gerando a necessidade de diversos templates
  • Pasta com os scripts para criação de DAGs

Como o código funciona

  • A pessoa adiciona/atualiza o arquivo YAML com todas as informações necessárias para criar a DAG ( para isso veja o CONTEXTS_README presente na pasta contexts/ )
  • O script scripts/convert_yaml_in_dag.py precisa ser executado, passando qual o nome do YAML, e ele fará a conversão para uma DAG.

Sobre as pastas:

  • contexts/: Pasta onde ficam armazenados os YAMLs por contexto da empresa.
    Dentro dessa pasta há um README explicando os padrões de criação dos YAMLs

  • templates/: Pasta onde ficam os arquivos .template, que são as bases para o Jinja criar as DAGs.

    • temos o arquivo dag.template
    • temos os operators já adicionados via template
    • temos um README com os operators já adicionados
  • scripts/ : Scripts tanto para transformar YAML em DAG, quanto para validar se o YAML está correto

    • yml_converter.py: Responsável por criar DAGs a partir do YAML
  • dags/: Pasta padrão do airflow, pode ficar nesse repositório ou em outro, ou até em outra branch.

Referencias:

TBD - Próximos passos

  • Automatizar a execução do script convert_yaml_in_dag.py
  • Generalizar o script acima, para mais opções de contextos, criando um input recebendo isto
  • Implementar testes na estrutura dos YAMLs. Uma sugestão é usar a lib cerberus
  • Implementar CI para os testes. Uma sugestão é usar o proprio github_actions
Owner
Arthur Henrique Dell' Antonia
Arthur Henrique Dell' Antonia
Auth for use with FastAPI

FastAPI Auth Pluggable auth for use with FastAPI Supports OAuth2 Password Flow Uses JWT access and refresh tokens 100% mypy and test coverage Supports

David Montague 95 Jan 02, 2023
Ready-to-use and customizable users management for FastAPI

FastAPI Users Ready-to-use and customizable users management for FastAPI Documentation: https://frankie567.github.io/fastapi-users/ Source Code: https

François Voron 2.4k Jan 04, 2023
Login qr line & qr image

login-qr-line-qr-image login qr line & qr image python3 & linux ubuntu api source: https://github.com/hert0t/BEAPI-BETA import httpx import qrcode fro

Alif Budiman 1 Dec 27, 2021
A JSON Web Token authentication plugin for the Django REST Framework.

Simple JWT Abstract Simple JWT is a JSON Web Token authentication plugin for the Django REST Framework. For full documentation, visit django-rest-fram

Jazzband 3.2k Dec 29, 2022
AddressBookApp - Address Book App in Django

AddressBookApp Application Name Address Book App in Django, 2022 Technologies La

Joshua K 1 Aug 18, 2022
FastAPI extension that provides JWT Auth support (secure, easy to use, and lightweight)

FastAPI JWT Auth Documentation: https://indominusbyte.github.io/fastapi-jwt-auth Source Code: https://github.com/IndominusByte/fastapi-jwt-auth Featur

Nyoman Pradipta Dewantara 468 Jan 01, 2023
A host-guest based app in which host can CREATE the room. and guest can join room with room code and vote for song to skip. User is authenticated using Spotify API

A host-guest based app in which host can CREATE the room. and guest can join room with room code and vote for song to skip. User is authenticated using Spotify API

Aman Raj 5 May 10, 2022
JSON Web Token Authentication support for Django REST Framework

REST framework JWT Auth JSON Web Token Authentication support for Django REST Framework Overview This package provides JSON Web Token Authentication s

Styria Digital Development 178 Jan 02, 2023
This is a Python library for accessing resources protected by OAuth 2.0.

This is a client library for accessing resources protected by OAuth 2.0. Note: oauth2client is now deprecated. No more features will be added to the l

Google APIs 787 Dec 13, 2022
API-key based security utilities for FastAPI, focused on simplicity of use

FastAPI simple security API key based security package for FastAPI, focused on simplicity of use: Full functionality out of the box, no configuration

Tolki 154 Jan 03, 2023
Brute force a JWT token. Script uses multithreading.

JWT BF Brute force a JWT token. Script uses multithreading. Tested on Kali Linux v2021.4 (64-bit). Made for educational purposes. I hope it will help!

Ivan Šincek 5 Dec 02, 2022
Django CAS 1.0/2.0/3.0 client authentication library, support Django 2.0, 2.1, 2.2, 3.0 and Python 3.5+

django-cas-ng django-cas-ng is Django CAS (Central Authentication Service) 1.0/2.0/3.0 client library to support SSO (Single Sign On) and Single Logou

django-cas-ng 347 Dec 18, 2022
Python One-Time Password Library

PyOTP - The Python One-Time Password Library PyOTP is a Python library for generating and verifying one-time passwords. It can be used to implement tw

PyAuth 2.2k Dec 26, 2022
Alisue 299 Dec 06, 2022
Library - Recent and favorite documents

Thingy Thingy is used to quickly access recent and favorite documents. It's an XApp so it can work in any distribution and many desktop environments (

Linux Mint 23 Sep 11, 2022
Storefront - A store App developed using Django, RESTFul API, JWT

Storefront A store App developed using Django, RESTFul API, JWT. SQLite has been

Muhammad Algshy 1 Jan 07, 2022
Complete Two-Factor Authentication for Django providing the easiest integration into most Django projects.

Django Two-Factor Authentication Complete Two-Factor Authentication for Django. Built on top of the one-time password framework django-otp and Django'

Bouke Haarsma 1.3k Jan 04, 2023
Official implementation of the AAAI 2022 paper "Learning Token-based Representation for Image Retrieval"

Token: Token-based Representation for Image Retrieval PyTorch training code for Token-based Representation for Image Retrieval. We propose a joint loc

Hui Wu 42 Dec 06, 2022
Flask App With Login

Flask App With Login by FranciscoCharles Este projeto basico é o resultado do estudos de algumas funcionalidades do micro framework Flask do Python. O

Charles 3 Nov 14, 2021
REST implementation of Django authentication system.

djoser REST implementation of Django authentication system. djoser library provides a set of Django Rest Framework views to handle basic actions such

Sunscrapers 2.2k Jan 01, 2023