An alternative to OpenFaaS nats-queue-worker for long-running functions

Overview

OpenFaas Job Worker

OpenFaas Job Worker is a fork of project : OSCAR Worker - https://github.com/grycap/oscar-worker Thanks to Sebástian Risco @srisco and Alfonso Pérez @alpegon for the initial project.

As oscar-worker is no longer supported, i decide to fork it, and add few improvments :

  • OpenFaas Asynchronous Callback is now support using http header : X-Callback-Url.
  • Labels and Annotations are injected in Jobs
  • Default-use Queue Name is now : job-faas-request, but it can be changed using env NATS_QUEUE setting.
  • openfaas-job-Worker run in openfaas namespace, jobs are create in namespace openfaas-job-fn
  • openfaas-job-Worker does not replace nats-queue-worker, openfaas default async still work.

OpenFaas Job Worker enables launching long-running functions as Kubernetes Jobs when /async-function/ path is used to make requests, and if function was deployed with annotation: com.openfaas.queue=job-faas-request

The goal is to ensure that each invocation has the specified resources and, furthermore, that functions can be executed in parallel depending on the resources available in the cluster.

Configuration

You can configure the worker through environment variables. To modify the default values you can edit the openfaas-job-worker-dep.yaml file:

...
        env:
        # Token to access the k8s API server (if not set reads the content of '/var/run/secrets/kubernetes.io/serviceaccount/token')  
        # - name: KUBE_TOKEN
        #   value: "xxx"
        - name: KUBERNETES_SERVICE_HOST
          value: "kubernetes.default"
        - name: KUBERNETES_SERVICE_PORT
          value: "443"
        - name: NATS_ADDRESS
          value: "nats.openfaas"
        - name: NATS_PORT
          value: "4222"
        - name: JOB_TTL_SECONDS_AFTER_FINISHED
          value: 60
        - name: JOB_BACKOFF_LIMIT
          value: 3
        - name: NATS_QUEUE
          value: "job-faas-request"

...

Deployment

In order to deploy the OpenFaas Job Worker you need to have already installed OpenFaaS in the Kubernetes cluster.

And create the required namespaces, RBAC and deployment:

kubectl apply -f yaml/openfaas-job-worker-namespaces.yaml
kubectl apply -f yaml/openfaas-job-worker-rbac.yaml
kubectl apply -f yaml/openfaas-job-worker-dep.yaml

Secrets

If your OpenFaaS function have secrets defined, you must duplicate them to the openfaas-job-fn namespace for granting access to jobs:

kubectl get secret <SECRET_NAME> -n openfaas-fn -o yaml \
| sed s/"namespace: openfaas-fn"/"namespace: openfaas-job-fn"/\
| kubectl apply -n openfaas-job-fn -f -

Logs

If you want to inspect worker's logs run:

kubectl logs deploy/openfaas-job-worker -n openfaas

To see specific function invocation logs, first get all pods of the openfaas-job-fn namespace and then query the one you want. You have to specify the container Name by adding "-c" and your fuction name .

kubectl get pods -n openfaas-job-fn 
kubectl logs POD_NAME -n openfaas-job-fn -c FUNCTION_NAME

Clear completed Jobs

Completed Jobs can be automatically deleted after finishing by enabling the TTLAfterFinished feature gate of Kubernetes versions >= v1.12. TTL Seconds to clean up Jobs can be configured through the JOB_TTL_SECONDS_AFTER_FINISHED environment variable of the worker.

To delete completed jobs manually, execute:

kubectl delete job $(kubectl get job -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}' -n openfaas-job-fn) -n openfaas-job-fn
Owner
Sebastien Aucouturier
Sebastien Aucouturier
A simple Telegram bot that analyses a given word, built with python-telegram-bot

Telegram Word Bot A simple Telegram bot that analyses a given word, built with python-telegram-bot. The bot is fairly useless - unless you often need

Chris N 2 Jul 14, 2022
With this program you can work English & Turkish

1 - How Can I Work This? You must have Python compilers in order to run this program. First of all, download the compiler in the link. Compiler 2 - Do

Mustafa Bahadır Doğrusöz 3 Aug 07, 2021
Reddit bot for r/khiphop

khiphop-bot Description This project is a collection of scripts that better the state of the r/khiphop subreddit, which represents Korean Hip-Hop and

1 Dec 21, 2021
Faster Twitch Alerts is a highly customizable, lightning-fast alternative to Twitch's slow mobile notification system

Faster Twitch Alerts What is "Faster Twitch Alerts"? Faster Twitch Alerts is a highly customizable, lightning-fast alternative to Twitch's slow mobile

6 Dec 22, 2022
An Advance Discord Generator Written in python Verified Email and Phone Number For Free!

Intro An Advance Discord Generator Written in python It can generate nearly fully verified tokens USAGE put server invite code inside ( invitecode = "

36 May 02, 2022
A Telegram Bot That Can Find Lyrics Of Song

Lyrics-Search-Bot A Telegram Bot That Can Find Lyrics Of Song A Simple Telegram Bot That Can Extract Lyrics Of Any Songs Deploy Commands start - To St

Muhammed Fazin 11 Oct 21, 2022
Send pm to Admin - Telegram

Send pm to Admin - Telegram

Ahoora 3 Nov 17, 2022
Discord Unverified Token Gen

Discord-Unverified-Token-Gen This is a token gen that was made in an hour and just generates unverified tokens, most will be locked. Usage: in cmd jus

Aran 2 Oct 23, 2022
Lambda-function - Python codes that allow notification of changes made to some services using the AWS Lambda Function

AWS Lambda Function This repository contains python codes that allow notificatio

Elif Apaydın 3 Feb 11, 2022
A python wrapper for interacting with the LabArchives API.

LabArchives API wrapper for Python A python wrapper for interacting with the LabArchives API. This very simple package makes it easier to make arbitra

Marek Cmero 3 Aug 01, 2022
Elkeid HUB - A rule/event processing engine maintained by the Elkeid Team that supports streaming/offline data processing

Elkeid HUB - A rule/event processing engine maintained by the Elkeid Team that supports streaming/offline data processing

Bytedance Inc. 61 Dec 29, 2022
A Python client for the Softcite software mention recognizer server

Softcite software mention recognizer client Python client for using the Softcite software mention recognition service. It can be applied to individual

4 Feb 02, 2022
Aws-cidr-finder - A Python CLI tool for finding unused CIDR blocks in AWS VPCs

aws-cidr-finder Overview An Example Installation Configuration Contributing Over

Cooper Walbrun 18 Jul 31, 2022
A simple API wrapper for the Tenor API

Gifpy A simple API wrapper for the Tenor API Installation Python 3.9 or higher is recommended python3 -m pip install gifpy Clone repository: $ git cl

Juan Ignacio Battiston 4 Dec 22, 2021
An async python wrapper to interact with the Steam API and its CMs

steam.py A modern, easy to use, and async ready package to interact with the Steam API. Heavily inspired by discord.py and borrowing functionality fro

James Hilton-Balfe 90 Dec 15, 2022
:snake: Python SDK to query Scaleway APIs.

Scaleway SDK Python SDK to query Scaleway's APIs. Stable release: Development: Installation The package is available on pip. To install it in a virtua

Scaleway 114 Dec 11, 2022
Binance Futures Client

Binance Futures Client

4 Aug 02, 2022
A Python app to serve Conveyor room requests and run approvals through Slack

✨ CONVEYOR FOR SLACK ✨ This is a friendly little Python app that will allow you to integrate your instance of Conveyor with your Slack workspace. In o

Vivienne 4 Sep 27, 2021
Super Fast Telegram UserBot Made With Python.

Description Super Fast Telegram UserBot Made With Python. LOGO Made With Support of All Userbots Dev's Dark-Venom is a Light-Weight Userbot. It's unde

2 Sep 14, 2021
ZELDA USERBOT adalah userbot Telegram modular yang berjalan di Python3 dengan database sqlalchemy.

ZELDA USERBOT TELEGRAM Userbot Yang Di Buat Karena Sering Gabut Di Telegram. ZELDA USERBOT adalah userbot Telegram modular yang berjalan di Python3 de

1 Dec 23, 2021