This is a repository for the Duke University Cloud Computing course project on Serveless Data Engineering Pipeline. For this project, I recreated the below pipeline.

Overview

AWS Data Engineering Pipeline

This is a repository for the Duke University Cloud Computing course project on Serverless Data Engineering Pipeline. For this project, I recreated the below pipeline in iCloud9 (reference: https://github.com/noahgift/awslambda):

drawing

Below are the steps of how to build this pipeline in AWS:

1️⃣ Create a new iCloud9 environment dedicated to this project.

🤔 Need a refresher? Please check this repo.

⚠️ Make sure to use name as your unique id for your items in the fang table.

2️⃣ Create a fang table in DynamoDB and SQS queue.

You can check how to do it here.

3️⃣ Build producer Lambda Function

  1. In iCloud9, initialize a serverless application with SAM template:

    sam init 

Inputs: 1, 2, 4, "producer"

  1. Set virtual environment and source it:

    # I called my virtual environment "comprehendProducer"
    python3 -m venv ~/.comprehendProducer
    source ~/.comprehendProducer/bin/activate
  2. Add the code for your application to app.py

  3. Add relevant packages used in your app to requirements.txt file

  4. Install requirements

     cd hello_world/
     pip install -r requirements.txt 
     cd .. 
  5. Create a repository (producer) in Elastic Container Registry (ECR) and copy its URI

  6. Build and deploy your serverless application:

    sam build 
    sam deploy --guided

    When prompted to input URI, paste the URI for the producer repository that you've just created.

  7. Create IAM Role granting Administrator Access to the Producer Lambda function.

    🤔 Not sure how to create IAM Role? Check out this video (17 min ).

  8. Add the execution role that you created to the Producer Lambda function.

    In case you forgot how to do it:

    In AWS console: Lambda ➡️ click on producer function ➡️ configuration ➡️ permissions ➡️ Edit ➡️ Select the role under Existing role.

  9. You are all set with the producer function! Now deactivate virtual environment:

    deactivate 
    cd .. 
    

4️⃣ Create an S3 bucket and note its name

5️⃣ Build consumer Lambda Function

Repeat steps in 3️⃣ .

⚠️ In #3 when you add the code for a consumer app to app.py, make sure to replace bucket="fangsentiment" with the name of your S3 bucket.

6️⃣ Add triggers to Lambda Functions

🤔 Not sure how to do it? Check out this video (start times are noted below):

Producer Lambda Function: CloudWatchEvent(30 min)

Consumer Lambda Function: SQS (42 min)

7️⃣ If all goes well, you will see sentiment results in your S3 bucket:

s3

💡 Tip: If you've already deployed your Lambda function but need to edit your application, you can make the necessary edits to your app and build and deploy the app again:

sam build && sam deploy 

💡 Tip: If you don't have space left on disk, you may want to remove a few docker containers that you don't use.

#list containers 
docker image ls 
# remove a container 
docker image rm <containerId>
A tool for exporting Telegram group chats into static websites, preserving chat history like mailing list archives.

tg-archive is a tool for exporting Telegram group chats into static websites, preserving chat history like mailing list archives. Preview The @fossuni

Kailash Nadh 400 Dec 27, 2022
Discord bot for Shran development

shranbot A discord bot named Herbert West that will monitor the Shran development discord server. Using dotenv shranbot uses a .env file to load secre

Matt Williams 1 Jul 29, 2022
Univerity-student oriented (lithuanian) discord bot

Univerity-student oriented (lithuanian) discord bot

3 Nov 30, 2021
:lock: Python 2.7/3.X client for HashiCorp Vault

hvac HashiCorp Vault API client for Python 3.x Tested against the latest release, HEAD ref, and 3 previous minor versions (counting back from the late

hvac 1k Dec 29, 2022
Create Multiple CF entry for multiple websites

AWS-CloudFront Problem: Deploy multiple CloudFront for account with multiple domains. Functionality: Running this script in loop and deploy CloudFront

Giten Mitra 5 Nov 18, 2022
Innocent-Bot - A Discord client self-bot for destroying, nuking and causing mischief in servers

Innocent-bot A Discord client self-bot for destroying, nuking and causing mischi

†† 5 Jan 26, 2022
Huggingface inference with GPU Docker on AWS

This repository contains code to containerize and deploy a GPU docker on AWS for summarization task. Find a detailed blogpost here Youtube Video Versi

Ramsri Goutham Golla 21 Dec 30, 2022
A telegram bot to download mega.nz links. (made with pyrogram).

Mega Link Downloader Bot This is a telegram bot to download mega.nz links and return them as files/videos - Made by a 100% noob! (When I mean noob I r

171 Dec 27, 2022
A telegram bot writen in python for mirroring files on the internet to our beloved Google Drive

[] Mirror Bot This is a telegram bot writen in python for mirroring files on the internet to our beloved Google Drive. Deploying on Heroku Give Star &

43 Mar 06, 2022
IdeasBot - Funny telegram bot to generate ideas for a project

Repository of PIdeas_bot About Funny telegram bot for generating projects ideas.

Just Koala 5 Oct 16, 2022
Senexia - A powerful telegram bot to manage your groups as effectively as possible

⚡ Kenechi bot ⚡ A Powerful, Smart And Simple Group Manager ... Written with AioG

Akhi 2 Jan 11, 2022
Template to create a telegram bot in python

Template for Telegram Bot Template to create a telegram bot in python. How to Run Set your telegram bot token as environment variable TELEGRAM_BOT_TOK

PyTopia 12 Aug 14, 2022
An open source API to validate the EU Covid Certificates / Green Certificates

Open Covid Certificate Validator This an open source API to validate EU Digital COVID Certificates. It receives a COVID certificate and validates it u

Merlin Schumacher 47 May 30, 2022
Baby Villager Bot

This script scrapes and stores the availability of timeslots for Car Driving Test at all RTA Serivce NSW centres in the state. Dependencies Account wi

Milo Weinberg 3 Dec 16, 2021
A discord tool to use bugs and exploits

DiscordTool A discord tool to use bugs and exploits Features: send a buggy messa

6 Aug 19, 2022
DDoS Script (DDoS Panel) with Multiple Bypass ( Cloudflare UAM,CAPTCHA,BFM,NOSEC / DDoS Guard / Google Shield / V Shield / Amazon / etc.. )

KARMA DDoS DDoS Script (DDoS Panel) with Multiple Bypass ( Cloudflare UAM,CAPTCHA,BFM,NOSEC / DDoS Guard / Google Shield / V Shield / Amazon / etc.. )

Hyuk 256 Jan 02, 2023
Add Reactions to your Channel Posts!

• Shaaak - Post Reaction Bot Simple and Minimalistic telegram bot to add Reactions and Comments to your Channel Posts! - What's Unique About it?

Harsh Raj 4 Jan 31, 2022
TwitterDataStreaming - Twitter data streaming using APIs

Twitter_Data_Streaming Twitter data streaming using APIs Use Case 1: Streaming r

Rita Kushwaha 1 Jan 21, 2022
Join & Leave spam for aminoapps using aminoboi

JLspam.py Join & Leave spam for https://aminoapps.com using aminoboi Instalação apt-get update -y apt-get upgrade -y apt-get install git pkg install

Moleey 1 Dec 21, 2021
Weather_besac is a French twitter bot that tweet the weather of the city of Besançon in Franche-Comté in France every day at 8am and 4pm.

Weather Bot Besac Weather_besac is a French twitter bot that tweet the weather of the city of Besançon in Franche-Comté in France every day at 8am and

Rgld_ 1 Nov 15, 2021