This is a repository for the Duke University Cloud Computing course project on Serveless Data Engineering Pipeline. For this project, I recreated the below pipeline.

Overview

AWS Data Engineering Pipeline

This is a repository for the Duke University Cloud Computing course project on Serverless Data Engineering Pipeline. For this project, I recreated the below pipeline in iCloud9 (reference: https://github.com/noahgift/awslambda):

drawing

Below are the steps of how to build this pipeline in AWS:

1️⃣ Create a new iCloud9 environment dedicated to this project.

🤔 Need a refresher? Please check this repo.

⚠️ Make sure to use name as your unique id for your items in the fang table.

2️⃣ Create a fang table in DynamoDB and SQS queue.

You can check how to do it here.

3️⃣ Build producer Lambda Function

  1. In iCloud9, initialize a serverless application with SAM template:

    sam init 

Inputs: 1, 2, 4, "producer"

  1. Set virtual environment and source it:

    # I called my virtual environment "comprehendProducer"
    python3 -m venv ~/.comprehendProducer
    source ~/.comprehendProducer/bin/activate
  2. Add the code for your application to app.py

  3. Add relevant packages used in your app to requirements.txt file

  4. Install requirements

     cd hello_world/
     pip install -r requirements.txt 
     cd .. 
  5. Create a repository (producer) in Elastic Container Registry (ECR) and copy its URI

  6. Build and deploy your serverless application:

    sam build 
    sam deploy --guided

    When prompted to input URI, paste the URI for the producer repository that you've just created.

  7. Create IAM Role granting Administrator Access to the Producer Lambda function.

    🤔 Not sure how to create IAM Role? Check out this video (17 min ).

  8. Add the execution role that you created to the Producer Lambda function.

    In case you forgot how to do it:

    In AWS console: Lambda ➡️ click on producer function ➡️ configuration ➡️ permissions ➡️ Edit ➡️ Select the role under Existing role.

  9. You are all set with the producer function! Now deactivate virtual environment:

    deactivate 
    cd .. 
    

4️⃣ Create an S3 bucket and note its name

5️⃣ Build consumer Lambda Function

Repeat steps in 3️⃣ .

⚠️ In #3 when you add the code for a consumer app to app.py, make sure to replace bucket="fangsentiment" with the name of your S3 bucket.

6️⃣ Add triggers to Lambda Functions

🤔 Not sure how to do it? Check out this video (start times are noted below):

Producer Lambda Function: CloudWatchEvent(30 min)

Consumer Lambda Function: SQS (42 min)

7️⃣ If all goes well, you will see sentiment results in your S3 bucket:

s3

💡 Tip: If you've already deployed your Lambda function but need to edit your application, you can make the necessary edits to your app and build and deploy the app again:

sam build && sam deploy 

💡 Tip: If you don't have space left on disk, you may want to remove a few docker containers that you don't use.

#list containers 
docker image ls 
# remove a container 
docker image rm <containerId>
A youtube videos or channels tag finder python module

A youtube videos or channels tag finder python module

Fayas Noushad 4 Dec 03, 2021
Python script to decode the EU Covid-19 vaccine certificate

vacdec Python script to decode the EU Covid-19 vaccine certificate This script takes an image with a QR code of a vaccine certificate as the parameter

Hanno Böck 244 Nov 30, 2022
A python Discord wrapper made in well, python.

discord.why A python Discord wrapper made in well, python. Made to be used by devs who want something a bit more, general. Basic Examples Sending a me

HellSec 6 Mar 26, 2022
Discord Selfbot, 90+ commands

Setting the bot up. STEP 1: copy the directory yook.club selfbot was downloaded and extracted into, open cmd and type "cd " then paste. STEP 2: python

yook 1 Dec 12, 2021
Group Management Bot

❤️ 𝗦𝗛𝗔𝗗𝗜𝗬𝗢 ❤️ A Powerful, Smart And Advance Group Manager ... Written with AioGram , Pyrogram and Telethon... ⭐️ Thanks to everyone who starred

Abdisamad Omar Mohamed 4 Dec 01, 2021
Slack->DynamDB->Some applications

slack-event-subscriptions About The Project Do you want to get simple attendance checks? If you are using Slack, participants can just react on a spec

UpstageAI 26 May 28, 2022
Market calendar RESTful API with holiday, late open, and early close. Over 50+ unique exchange calendars for global equity and futures markets.

Trading Calendar Market calendar RESTful API with holiday, late open, and early close. Over 50+ unique exchange calendars for global equity and future

Apptastic Software 1 Feb 03, 2022
Telegram Bot to store Posts and Documents and it can Access by Special Links.

File-sharing-Bot Telegram Bot to store Posts and Documents and it can Access by Special Links. I Guess This Will Be Usefull For Many People..... 😇 .

Code X Botz 1.2k Jan 08, 2023
Python wrapper for the Intercom API.

python-intercom Not officially supported Please note that this is NOT an official Intercom SDK. The third party that maintained it reached out to us t

Intercom 215 Dec 22, 2022
Get your Pixiv token (for running upbit/pixivpy)

gppt: get-pixivpy-token Get your Pixiv token (for running upbit/pixivpy) Refine pixiv_auth.py + its fork Install ❭ pip install gppt Run Note: In advan

haruna 58 Jan 04, 2023
This library is for simplified work with the sms-man.com API

SMSMAN Public API Python This is a lightweight library that works as a connector to Sms-Man public API Installation pip install smsman Documentation h

13 Nov 19, 2022
Instagram bot for promoting ROKA trainee soldier(just like me)'s consolation letters.

Instagram_bot (필자를 포함한) 모든 대한민국 훈련병들을 위한 인스타그램 인편지기입니다. Instagram bot for promoting ROKA trainee soldier(just like me)'s consolation letters. 들어가기 (Ge

Lee, Jongjun 2 Nov 21, 2021
Opencontactbook - Bulk-manage large numbers of vCard contacts with built-in geolocation

Open Contact Book Open Contact Book is a buiness-oriented, cross-platform, Pytho

Aurélien PIERRE 2 Aug 08, 2022
The most fresh and updateable Telegram userbot. By one of the most active contibutors to GeekTG

Installation Script installation: Simply run this command out of root: . (wget -qO- http://gg.gg/get_hikka) Manual installation: apt update && apt in

Dan Gazizullin 150 Jan 04, 2023
Telegram bot to host python bots

Host-Bot Setup the api Upload the flask api on your host #its not important to do #i used it just for simple captcha system + save ids on your host!

Plugin 15 Feb 11, 2022
SkyzoMusicBot - Bot Music Telegram By Skyzo

SKYZO MUSIC BOT Telegram Music Bot And Stream Feature New Version Ready to use m

Skyzo 19 Apr 08, 2022
doi, pubmed, arxiv.org的查询服务API接口,部署于vercel云函数

article-search-service doi, pubmed, arxiv.org的查询服务API接口,部署于vercel云函数 云函数 vercel,国内可能被qiang了。 DOI接口 POST https://article-search-service.vercel.app/api/

HyokaChen 2 Oct 10, 2021
Wats2PDF - Convert whatsapp exported chat(without media) into a readable pdf format

Wats2PDF convert whatsApp exported chat into a readable pdf format. convert with

5 Apr 26, 2022
Terraform wrapper to manage state across multiple cloud providers(AWS, GCP, and Azure)

Terraform Remote State Manager(tfremote) tf is a python package for managing terraform remote state for: Google(Gcloud), AWS, and Azure. It sets a def

tomarv2 1 Dec 08, 2021
YouTube-Discord-Bot - Discord Bot to Search YouTube

YouTube Bot Info YouTube Bot is a discord bot where you can search for anything

Riceblades11 10 Mar 05, 2022