An alternative to OpenFaaS nats-queue-worker for long-running functions

Overview

OpenFaas Job Worker

OpenFaas Job Worker is a fork of project : OSCAR Worker - https://github.com/grycap/oscar-worker Thanks to Sebástian Risco @srisco and Alfonso Pérez @alpegon for the initial project.

As oscar-worker is no longer supported, i decide to fork it, and add few improvments :

  • OpenFaas Asynchronous Callback is now support using http header : X-Callback-Url.
  • Labels and Annotations are injected in Jobs
  • Default-use Queue Name is now : job-faas-request, but it can be changed using env NATS_QUEUE setting.
  • openfaas-job-Worker run in openfaas namespace, jobs are create in namespace openfaas-job-fn
  • openfaas-job-Worker does not replace nats-queue-worker, openfaas default async still work.

OpenFaas Job Worker enables launching long-running functions as Kubernetes Jobs when /async-function/ path is used to make requests, and if function was deployed with annotation: com.openfaas.queue=job-faas-request

The goal is to ensure that each invocation has the specified resources and, furthermore, that functions can be executed in parallel depending on the resources available in the cluster.

Configuration

You can configure the worker through environment variables. To modify the default values you can edit the openfaas-job-worker-dep.yaml file:

...
        env:
        # Token to access the k8s API server (if not set reads the content of '/var/run/secrets/kubernetes.io/serviceaccount/token')  
        # - name: KUBE_TOKEN
        #   value: "xxx"
        - name: KUBERNETES_SERVICE_HOST
          value: "kubernetes.default"
        - name: KUBERNETES_SERVICE_PORT
          value: "443"
        - name: NATS_ADDRESS
          value: "nats.openfaas"
        - name: NATS_PORT
          value: "4222"
        - name: JOB_TTL_SECONDS_AFTER_FINISHED
          value: 60
        - name: JOB_BACKOFF_LIMIT
          value: 3
        - name: NATS_QUEUE
          value: "job-faas-request"

...

Deployment

In order to deploy the OpenFaas Job Worker you need to have already installed OpenFaaS in the Kubernetes cluster.

And create the required namespaces, RBAC and deployment:

kubectl apply -f yaml/openfaas-job-worker-namespaces.yaml
kubectl apply -f yaml/openfaas-job-worker-rbac.yaml
kubectl apply -f yaml/openfaas-job-worker-dep.yaml

Secrets

If your OpenFaaS function have secrets defined, you must duplicate them to the openfaas-job-fn namespace for granting access to jobs:

kubectl get secret <SECRET_NAME> -n openfaas-fn -o yaml \
| sed s/"namespace: openfaas-fn"/"namespace: openfaas-job-fn"/\
| kubectl apply -n openfaas-job-fn -f -

Logs

If you want to inspect worker's logs run:

kubectl logs deploy/openfaas-job-worker -n openfaas

To see specific function invocation logs, first get all pods of the openfaas-job-fn namespace and then query the one you want. You have to specify the container Name by adding "-c" and your fuction name .

kubectl get pods -n openfaas-job-fn 
kubectl logs POD_NAME -n openfaas-job-fn -c FUNCTION_NAME

Clear completed Jobs

Completed Jobs can be automatically deleted after finishing by enabling the TTLAfterFinished feature gate of Kubernetes versions >= v1.12. TTL Seconds to clean up Jobs can be configured through the JOB_TTL_SECONDS_AFTER_FINISHED environment variable of the worker.

To delete completed jobs manually, execute:

kubectl delete job $(kubectl get job -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}' -n openfaas-job-fn) -n openfaas-job-fn
Owner
Sebastien Aucouturier
Sebastien Aucouturier
OpenZeppelin Contracts written in Cairo for StarkNet, a decentralized ZK Rollup

OpenZeppelin Cairo Contracts A library for secure smart contract development written in Cairo for StarkNet, a decentralized ZK Rollup. ⚠️ WARNING! ⚠️

OpenZeppelin 592 Jan 04, 2023
A library that allows you to easily mock out tests based on AWS infrastructure.

Moto - Mock AWS Services Install $ pip install moto[ec2,s3,all] In a nutshell Moto is a library that allows your tests to easily mock out AWS Services

Steve Pulec 6.5k Jan 02, 2023
Automation for grabbing keys from a Linux host. Useful during red team exercises to quickly help assess what access to a Linux host can lead to.

keygrabber Automation for grabbing keys from a Linux host. This can be helpful during red team exercises when you gain access to a Linux host and want

Cedric Owens 14 Sep 27, 2022
A Python script that exports users from one Telegram group to another using one or more concurrent user bots.

ExportTelegramUsers A Python script that exports users from one Telegram group to another using one or more concurrent user bots. Make sure to set all

Fasil Minale 17 Jun 26, 2022
Awslogs - AWS CloudWatch logs for Humans™

awslogs awslogs is a simple command line tool for querying groups, streams and events from Amazon CloudWatch logs. One of the most powerful features i

Jorge Bastida 4.5k Dec 30, 2022
Powerful and Async API for AnimeWorld.tv 🚀

Powerful and Async API for AnimeWorld.tv 🚀

1 Nov 13, 2021
Instrument asyncio Python for distributed tracing with AWS X-Ray.

xraysink (aka xray-asyncio) Extra AWS X-Ray instrumentation to use distributed tracing with asyncio Python libraries that are not (yet) supported by t

Gary Donovan 12 Nov 10, 2022
Generate Heroku-like random names to use in your python applications

HaikunatorPY Generate Heroku-like random names to use in your python applications. Installation pip install haikunator Usage Haikunator is pretty sim

Atrox 116 Nov 15, 2022
Template to create a telegram bot in python

Template for Telegram Bot Template to create a telegram bot in python. How to Run Set your telegram bot token as environment variable TELEGRAM_BOT_TOK

PyTopia 12 Aug 14, 2022
Scanner and Checker for Binance Scam Contracts

Money Printer by Warranty Voider well this isnt exactly a printer, but it helps you find and check new token startups. In the end its a nice scam cont

12 Nov 24, 2022
Oussama has taken his first dose of vaccine D days ago

Oussama has taken his first dose of vaccine D days ago. He may take the second dose no less than L days and no more than R days since his first dose. Determine if Oussama is too early, too late, or i

INDIA - ENSAM Rabat 2 Feb 01, 2022
An Advance Discord Generator Written in python Verified Email and Phone Number For Free!

Intro An Advance Discord Generator Written in python It can generate nearly fully verified tokens USAGE put server invite code inside ( invitecode = "

36 May 02, 2022
𝐀 𝐦𝐨𝐝𝐮𝐥𝐚𝐫 𝐓𝐞𝐥𝐞𝐠𝐫𝐚𝐦 𝐆𝐫𝐨𝐮𝐩 𝐦𝐚𝐧𝐚𝐠𝐞𝐦𝐞𝐧𝐭 𝐛𝐨𝐭 𝐰𝐢𝐭𝐡 𝐮𝐥𝐭𝐢𝐦𝐚𝐭𝐞 𝐟𝐞𝐚𝐭𝐮𝐫𝐞𝐬 !!

𝐇𝐨𝐰 𝐓𝐨 𝐃𝐞𝐩𝐥𝐨𝐲 For easiest way to deploy this Bot click on the below button 𝐌𝐚𝐝𝐞 𝐁𝐲 𝐒𝐮𝐩𝐩𝐨𝐫𝐭 𝐆𝐫𝐨𝐮𝐩 𝐒𝐨𝐮𝐫𝐜𝐞𝐬 𝐆𝐞𝐧𝐞?

Mukesh Solanki 1 Dec 10, 2021
Automate and Manage Telegram Channels

Channel Automation Bot @ChannelAutomateBot A star ⭐ from you means a lot to us! Telegram bot to automate and manage channels. Usage Deploy to Heroku T

Stark Bots 61 Dec 29, 2022
SkyzoMusicBot - Bot Music Telegram By Skyzo

SKYZO MUSIC BOT Telegram Music Bot And Stream Feature New Version Ready to use m

Skyzo 19 Apr 08, 2022
A telegram bot to track whales activities on multiple blockchains.

Telegram Bot : Whale Watcher A straightforward telegram bot written in python to track whales activity on multiple blockchains, using whale-alert API

Laurenz Bougan 1 Dec 10, 2021
A telegram bot help you to get stylish fonts and text

Stylish Font Bot 🐿 This is a telegram bot help you to get stylish fonts and text. Config Vars 🤖 API_HASH: Get this value from my.telegram.org. API_K

MSTL updates 1 Nov 08, 2021
Music cog for discord bots. Supports YouTube, YoutubeMusic, SoundCloud and Spotify.

dismusic Music cog for discord bots. Supports YouTube, YoutubeMusic, SoundCloud and Spotify. Installation python3 -m pip install dismusic Usage from d

Md Shahriyar Alam 59 Jan 08, 2023
ShotsGram - For sending captures from your monitor to a telegram chat (robot)

ShotsGram pt-BR Envios de capturas do seu monitor para um chat do telegram. Essa

Carlos Alberto 1 Apr 24, 2022
Terraform Cloud CLI for Managing Workspace Terraform Versions

Terraform Cloud Version Manager This tiny script makes it easy to update the Terraform Version on all of the Workspaces inside Terraform Cloud. It wil

Robert Hafner 1 Jan 07, 2022