Yet another Airflow plugin using CLI command as RESTful api, supports Airflow v2.X.

Overview

中文版文档

Airflow Extended API Plugin

Airflow Extended API, which export airflow CLI command as REST-ful API to extend the ability of airflow official API.

This plugin is available for airflow 2.x Version and extensible, as you can easily define your own API to execute any Airflow CLI command so that it fits your demand.

Current Supported Commands

The following commands are supported now, and more is coming.

  • airflow dags backfill
  • airflow tasks run
  • airflow tasks clear

Plugin Install

  1. Install the plugin via pip
  pip install airflow-extended-api
  1. Restart the Airflow Web Server

  2. Open Airflow UI in Docs - Extended API OpenAPI or http://localhost:8080/ to view extended API details in swagger UI. img.png

Usage

Examples

curl request example:

curl -X POST --user "airflow:airflow" https://localhost:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'

Response Schema:

{
  "executed_command": "string",
  "exit_code": 0,
  "output_info": [
    "string"
  ],
  "error_info": [
    "string"
  ]
}

curl without Credentials data

Note that you will need to pass credentials' data in --user "{username}:{password}" format, or you will get an Unauthorized error.

curl -X POST http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'

response

{
  "detail": null,
  "status": 401,
  "title": "Unauthorized",
  "type": "https://airflow.apache.org/docs/apache-airflow/2.2.5/stable-rest-api-ref.html#section/Errors/Unauthenticated"
}

curl with wrong CLI Command

curl -X POST --user "airflow:airflow"  http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'

response

{
  "error_info": [
    "Traceback (most recent call last):",
    "  File \"/home/airflow/.local/bin/airflow\", line 8, in <module>",
    "    sys.exit(main())",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py\", line 48, in main",
    "    args.func(args)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py\", line 48, in command",
    "    return func(*args, **kwargs)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 92, in wrapper",
    "    return f(*args, **kwargs)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py\", line 506, in task_clear",
    "    dags = get_dags(args.subdir, args.dag_id, use_regex=args.dag_regex)",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 203, in get_dags",
    "    return [get_dag(subdir, dag_id)]",
    "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 193, in get_dag",
    "    f\"Dag {dag_id!r} could not be found; either it does not exist or it failed to parse.\"",
    "airflow.exceptions.AirflowException: Dag 'string' could not be found; either it does not exist or it failed to parse.",
    ""
  ],
  "executed_command": "airflow tasks clear string -e 2019-08-24T14:15:22+00:00 -s 2019-08-24T14:15:22+00:00 -t string -y -d",
  "exit_code": 1,
  "output_info": [
    "[\u001b[34m2022-04-22 10:05:50,538\u001b[0m] {\u001b[34mdagbag.py:\u001b[0m500} INFO\u001b[0m - Filling up the DagBag from /opt/airflow/dags\u001b[0m",
    ""
  ]
}

Project Plan

  • support custom configuration

Links and References

You might also like...
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!

Streamify A data pipeline with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more! Description Objective The project will stre

A python script to decrypt media files encrypted using the Android application 'Secret Calculator Photo Vault'. Supports brute force of PIN also.

A python script to decrypt media files encrypted using the Android application 'Secret Calculator Photo Vault'. Supports brute force of PIN also.

✨ Udemy Coupon Finder For Discord. Supports Turkish & English Language.

Udemy Course Finder Bot | Udemy Kupon Bulucu Botu This bot finds new udemy coupons and sends to the channel. Before Setup You must have python = 3.6

An esoteric programming language that supports concurrency, regex, and web requests.

The Hofstadter Esoteric Programming Language Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's La

Zeus is an open source flight intellingence tool which supports more than 13,000+ airlines and 250+ countries.

Zeus Zeus is an open source flight intellingence tool which supports more than 13,000+ airlines and 250+ countries. Any flight worldwide, at your fing

Aero is an open source airplane intelligence tool. Aero supports more than 13,000 airlines and 250 countries. Any flight worldwide at your fingertips.

Aero Aero supports more than 13,000 airlines and 250 countries. Any flight worldwide at your fingertips. Features Main : Flight lookup Aircraft lookup

A feed generator. Currently supports generating RSS feeds from Google, Bing, and Yahoo news.

A feed generator. Currently supports generating RSS feeds from Google, Bing, and Yahoo news.

Implements a polyglot REPL which supports multiple languages and shared meta-object protocol scope between REPLs.
Implements a polyglot REPL which supports multiple languages and shared meta-object protocol scope between REPLs.

MetaCall Polyglot REPL Description This repository implements a Polyglot REPL which shares the state of the meta-object protocol between the REPLs. Us

A minimalist personal blogging system that natively supports Markdown, LaTeX, and code highlighting.
A minimalist personal blogging system that natively supports Markdown, LaTeX, and code highlighting.

December Welcome to the December blogging system's code repository! Introduction December is a minimalist personal blogging system that natively suppo

Releases(v1.1.2)
  • v1.1.2(Apr 22, 2022)

    中文版文档

    Airflow Extended API Plugin

    Airflow Extended API, which export airflow CLI command as REST-ful API to extend the ability of airflow official API.

    Features

    • 👏Available: Probably the only available CLI command plugin on Github that supports the Airflow 2.x version.
    • 🎉Extensible: Easily define your own API to execute any Airflow CLI command so that it fits your demand.

    Plugin Install

    1. Install the plugin via pip
      pip install airflow-extended-api
    
    1. Restart the Airflow Web Server

    2. Open Airflow UI in Docs - Extended API OpenAPI or http://localhost:8080/ to view extended API details in swagger UI. img.png

    Usage

    Examples

    curl request example:

    curl -X POST --user "airflow:airflow" https://localhost:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'
    

    Response Schema:

    {
      "executed_command": "string",
      "exit_code": 0,
      "output_info": [
        "string"
      ],
      "error_info": [
        "string"
      ]
    }
    

    curl without Credentials data

    Note that you will need to pass credentials' data in --user "{username}:{password}" format, or you will get an Unauthorized error.

    curl -X POST http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'
    

    response

    {
      "detail": null,
      "status": 401,
      "title": "Unauthorized",
      "type": "https://airflow.apache.org/docs/apache-airflow/2.2.5/stable-rest-api-ref.html#section/Errors/Unauthenticated"
    }
    

    curl with wrong CLI Command

    curl -X POST --user "airflow:airflow"  http://127.0.0.1:8080/api/extended/clear -H "Content-Type: application/json" -d '{"dagName": "string","downstream": true,"endDate": "2019-08-24T14:15:22Z","jobName": "string","startDate": "2019-08-24T14:15:22Z","username": "Extended API"}'
    

    response

    {
      "error_info": [
        "Traceback (most recent call last):",
        "  File \"/home/airflow/.local/bin/airflow\", line 8, in <module>",
        "    sys.exit(main())",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py\", line 48, in main",
        "    args.func(args)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py\", line 48, in command",
        "    return func(*args, **kwargs)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 92, in wrapper",
        "    return f(*args, **kwargs)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py\", line 506, in task_clear",
        "    dags = get_dags(args.subdir, args.dag_id, use_regex=args.dag_regex)",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 203, in get_dags",
        "    return [get_dag(subdir, dag_id)]",
        "  File \"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py\", line 193, in get_dag",
        "    f\"Dag {dag_id!r} could not be found; either it does not exist or it failed to parse.\"",
        "airflow.exceptions.AirflowException: Dag 'string' could not be found; either it does not exist or it failed to parse.",
        ""
      ],
      "executed_command": "airflow tasks clear string -e 2019-08-24T14:15:22+00:00 -s 2019-08-24T14:15:22+00:00 -t string -y -d",
      "exit_code": 1,
      "output_info": [
        "[\u001b[34m2022-04-22 10:05:50,538\u001b[0m] {\u001b[34mdagbag.py:\u001b[0m500} INFO\u001b[0m - Filling up the DagBag from /opt/airflow/dags\u001b[0m",
        ""
      ]
    }
    

    Project Plan

    • [ ] support dags backfill cli command
    • [ ] support custom configuration

    Links and References

    Source code(tar.gz)
    Source code(zip)
Owner
Eric Cao
Stay hungry, Stay foolish。
Eric Cao
Cvdl-hw2 - Find Contour, Camera Calibration, Augmented Reality and Stereo Disparity Map

opevcvdl-hw2 This project uses openCV and Qt to achieve the requirements. Version Python 3.7 opencv-contrib-python 3.4.2.17 Matplotlib 3.1.1 pyqt5 5.1

Kenny Cheng 3 Aug 17, 2022
An After Effects render queue for ShotGrid Toolkit.

AEQueue An After Effects render queue for ShotGrid Toolkit. Features Render multiple comps to locations defined by templates in your Toolkit config. C

Brand New School 5 Nov 20, 2022
Free and open source qualitative research tool

Taguette A spin on the phrase "tag it!", Taguette is a free and open source qualitative research tool that allows users to: Import PDFs, Word Docs (.d

Remi Rampin 48 Jan 02, 2023
Small projects for python beginners.

Python Mini Projects For Beginners I recently started doing the #100DaysOfCode Challenge in Python. I've used Python before, but I had switched to JS

Sreekesh Iyer 10 Dec 12, 2022
The Python Achievements Framework!

Pychievements: The Python Achievements Framework! Pychievements is a framework for creating and tracking achievements within a Python application. It

Brian 114 Jul 21, 2022
Shows VRML team stats of all players in your pubs

VRML Team Stat Searcher Displays Team Name, Team Rank (Worldwide), and tier of all the players in your pubs. GUI WIP: Only username search works (for

Hamish Burke 2 Dec 22, 2022
BlackIP-Rep is a tool designed to gather the reputation and information of Bulk IP's.

BlackIP-Rep is a tool designed to gather the reputation and information of Bulk IP's. Focused on increasing the workflow of Security Operations(SOC) team during investigation.

0LiVEr 6 Dec 12, 2022
Q-Tracker is originally a High School Project created by Admins of Cirus Lab.

Q-Tracker is originally a High School Project created by Admins of Cirus Lab. It's completly coded in python along with mysql.(Tkinter For GUI)

Adithya Krishnan 2 Nov 14, 2022
A numbers check python package

A numbers check python package

Fayas Noushad 3 Nov 28, 2021
Imitate Moulinette written in Python

Imitate Moulinette written in Python

Pumidol Leelerdsakulvong 2 Jul 26, 2022
🤖️ Plugin for Sentry which allows sending notification via DingTalk robot.

Sentry DingTalk Sentry 集成钉钉机器人通知 Requirments sentry = 21.5.1 特性 发送异常通知到钉钉 支持钉钉机器人webhook设置关键字 配置环境变量 DINGTALK_WEBHOOK: Optional(string) DINGTALK_CUST

1 Nov 04, 2021
aaencode for python,把python代码转换为颜文字

py-aaencode aaencode for python,把python代码转换为颜文字 compile.py: 将python编译成颜文字,编译结果有随机性,可以选择BPE词表压缩代码 compile_min.py: 最小化的编译器 compiled_min.txt: 编译得到的最小的com

11 Dec 30, 2021
The ldapconsole script allows you to perform custom LDAP requests to a Windows domain

ldapconsole The ldapconsole script allows you to perform custom LDAP requests to a Windows domain. Features Authenticate with password Authenticate wi

Podalirius 38 Dec 09, 2022
Basit bir cc generator'ü.

Basit bir cc generator'ü. Setup What To Do; Python Installation We install python from CLICK Generator Board After installing the file and python, we

Lâving 7 Jan 09, 2022
Syntax highlighting for yarn.lock and bun.lockb files

Yarn.lock Syntax Highlighting Syntax highlighting for yarn.lock and bun.lockb files Installation Plugin is not publushed yet on Package Control, to in

Alexander Kuznetsov 4 Jul 06, 2022
Aggressor script that gets the latest commands from CobaltStrikes web site and creates an aggressor script based on tool options.

opsec-aggressor Aggressor script that gets the latest commands from CobaltStrikes opsec page and creates an aggressor script based on tool options. Gr

JP 10 Nov 26, 2022
A step-by-step tutorial for how to work with some of the most basic features of Nav2 using a Jupyter Notebook in a warehouse environment to create a basic application.

This project has a step-by-step tutorial for how to work with some of the most basic features of Nav2 using a Jupyter Notebook in a warehouse environment to create a basic application.

Steve Macenski 49 Dec 22, 2022
Basic Clojure REPL for Sublime Text

Basic Clojure REPL for Sublime Text Goals: Decomplected: just REPL, nothing more Zero dependencies: works directly with pREPL Compact: Display code ev

Nikita Prokopov 23 Dec 24, 2021
Pyfetch - Simple Fetch written in Python

pyfetch Simple Fetch written in Python Screenshots Install Clone this repository

2 Sep 02, 2022