Event-driven-model-serving - Unified API of Apache Kafka and Google PubSub

Overview

event-driven-model-serving

Unified API of Apache Kafka and Google PubSub

1. Project Structure

.event-driven-model-serving
+-- Modelling
|   +-- model_checkpoints
|       +-- mnist_cnn_model.h5
|   +-- plots
|       +-- loss_plot.png
|
|   +-- __init__.py
|   +-- data_preparation.py
|   +-- train_cnn.py
|   +-- predict_cnn.py
|
+-- MessageBroker
|   +-- __init__.py
|   +-- broker_interface.py
|   +-- google_pubsub_handler.py
|   +-- kafka_handler.py
|   +-- message_broker_factory.py
|   +-- kafka_config.ini
|   +-- pubsub_config.ini
|
+-- Queue
|   +-- log.txt
|
+-- DB
|   +-- db_for_app_2.txt
|
+-- app_1.py
+-- model_server_sub.py
+-- model_server_pub.py
+-- app_2.py

2. Cloud service configuration

2-1. Confluent Kafka

  1. In MessageBroker/kafka_config.ini, fill in bootstrap.servers, sasl.username, and sasl.password.

  2. Create two topics in your Kafka cluster. Name one as new_image_topic, another as inference_topic.

    Note:
    There is a create_topic() function in each handler, which are not working yet. I will make an update on this soon. For now, you can creat topics by manually in the confluent console.

2-2. Google Pub/Sub

  1. In MessageBroker/pubsub_config.ini, fill in your project_id.

    Note:
    If you are using gcloud CLI, after you set you project with gcloud config set project <Your project name>, project_id can be found by gcloud config get-value project.

  2. Set up Service account and Cloud IAM role.

    Go to https://console.cloud.google.com/iam-admin/ and pick "service account" tab.
    Make one and create JSON key.

    Then set a environment variable with your JSON key like below:
    export GOOGLE_APPLICATION_CREDENTIALS=~/Downloads/pubsub-trial-key.json

  3. Like you did in Kafka, please make two topics.
    Name one as new_image_topic, another as inference_topic.

    In GCP console, when you make these topic, check Add a default subscription as an option. This will create two according subscription_id named, new_image_topic-sub and inference_topic-sub

Every thing is ready for the cloud configuration!

3. About the architecture

  1. Queue directory.

    In the log.txt file, I simulate Redis Queue and FIFO strategy.
    This is to get results queued here before sending the results to application 2.

  2. DB directory.

    In the db_for_app_2.txt file, I simulate a simple database for application 2.
    This shows that application 2 successfully pull result from model server's publisher.

  3. BrokerInterface class is a abstract base class that makes interface for two handlers: KafkaHandler and PubSubHandler.

  4. MessageBrokerFactory creates instance object of either two Handler class.
    Default is set to KafkaHandler.

  5. However, no matter what Handler is created, this unified API ensures the same method names, parameters, and the behaviours.

Side note

  • Currently, the image recognition model is only trained with 28x28 pixel grey scale image. It does not support different size nor color image.
Owner
Danny Toeun Kim
MEng. Computer Science at @UCL
Danny Toeun Kim
A Pluggable And Powerful Telegram Manager Bot

FRIDAY-USERBOT 🇮🇳 A Powerful, Smart And Simple Userbot In Pyrogram. Support 🚑 Inspiration & Credits Userge-X Userge Pokurt Pyrogram Code Owners Chs

DevsExpo 371 Mar 26, 2022
Get random jokes bapack2 on telegram

Jokes Bapack2 Telegram Bot Get random jokes bapack2 from jokes-bapack2-api on telegram bot Screenshot Requirements python pip pipenv python-telegram-b

Miftah Afina 2 Nov 17, 2021
An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.

TWINT - Twitter Intelligence Tool No authentication. No API. No limits. Twint is an advanced Twitter scraping tool written in Python that allows for s

TWINT Project 14.2k Jan 03, 2023
This is a python wrapper for "the best api in the world"

This is a python wrapper for my api api_url = "https://api.dhravya.me/" This wrapper now has async support, its basically the same except it uses asyn

Dhravya Shah 3 Dec 21, 2021
Simple Telegram webscrap bot

webscrap-bot Simple Telegram webscrap bot Configs TOKEN - Get bot token from @BotFather API_ID - From my.telegram.org API_HASH - From my.telegram.org

lokaman chendekar 10 Oct 21, 2022
A project that alerts me when there's a dog outside so I can go look at it.

Dog Detector A project that alerts me when there's a dog outside so I can go look at it. Tech Specs This script uses the YOLOv3 object detection model

rydercalmdown 58 Jul 29, 2022
A Simple Telegram Bot By @AsmSafone to Download Files From Mega.nz and Upload It to Telegram

MegaDL-Bot A Simple Telegram Bot By @AsmSafone to Download Files From Mega.nz and Upload It to Telegram Features No Login Required All Mega.nz File Li

SAF ONE 92 Dec 02, 2022
A Telegram user bot to count telegram channel subscriber or group member.

Subscriber Count Userbot A Telegram user bot to count telegram channel subscriber or group member. This tool is only for educational purpose. You coul

IDNCoderX 8 Nov 30, 2022
Archive tweets and make them searchable

Tweeter Archive and search your tweets and liked tweets using AWS Lambda, DynamoDB and Elasticsearch. Note: this project is primarily being used a tes

Kamil Sindi 8 Nov 18, 2022
A Python app which retrieves the rank and players' equipped skins during a match

VALORANT rank yoinker About The Project Usage Contributing Contact Acknowledgements Disclaimer About The Project Their Queue Current Skin Current Rank

Isaac Kenyon 270 Jan 04, 2023
A python client for the Software-Challenge Germany.

sc-client-python A python client for the Software-Challenge Germany. Creating a new project (Optional) Install virtualenv virtualenv is a tool that cr

rpkak 3 Jan 22, 2022
A Discord Bot for the Pygame Community Server

PygameCommunityBot The Pygame Community Discord bot The bot is capable of doing a lot of stuff, the command prefix is pg!. For help on all the bot com

PygameCommunityDiscord 23 Nov 30, 2022
A discord.py bot template with Cogs implemented.

discord-cogs-template A discord.py bot template with Cogs implemented. Instructions Before you start âš  Basic knowledge of python is required. Steps If

censor 2 Sep 02, 2022
MCNameBot is a fast discord bot that is used to check the availability of a Minecraft name with a simple command.

MCNameBot MCNameBot is a fast discord bot that is used to check the availability of a Minecraft name with a simple command. If you would like to just

Killin 2 Oct 11, 2022
LEC_Ditto is a bot that tracks the follows and unfollows of Twitter accounts

✨ LEC_Ditto ✨ I'm Ditto, and I'm a bot 🤖 . Getting Started | Installation | Usage Getting Started LEC_Ditto is a bot that tracks the follows and unfo

2 Mar 30, 2022
Python bot for send videos of a Youtube channel to a telegram group , channel or chat

py_youtube_to_telegram Usage: If you want to install ytt and use it, run this command: sudo sh -c "$(curl -fsSL https://raw.githubusercontent.com/nima

Nima Fanniasl 8 Nov 22, 2022
Tracker to check the covid shot slot availability in India and send mobile alerts via Twilio Messaging Service.

Cowin-Slot-Tracker Tracker to check the covid vaccine slot availability in India and send mobile notifications through Twilio Messaging Service. Requi

invalid username 27 Nov 12, 2022
Simple Telegram Bot to extract various types of archives from a telegram file or a direct link

Unzipper Bot A Telegram Bot to Extract Various Types Of Archives Features Extract various types of archives like rar, zip, tar, 7z, tar.xz etc. Passwo

I'm Not A Bot #Left_TG 93 Dec 27, 2022
Use Seaborn to visualize interpret the byte layout of Solana account types

solana-account-vis Use Seaborn to visually interpret the byte layout of Solana account types Usage from account_visualization import generate_account_

Jarry Xiao 15 Aug 25, 2022
A collection of discord tools I've made.

Discord A collection of discord tools i've made. What's in here? Basically every discord related project i've worked on can be found here, i'll try an

?? ?? ?? 6 Nov 13, 2021