This project aims to evaluate the feasibility of an architecture based on serverless technology for a web crawler using scrapy as a framework of choice. If you are interested in this project you can visit my blog post and read about it.
Scraping COVID-19 data from DDC Dashboard Scraping Thailand COVID-19 data from the DDC's tableau dashboard. Data is updated at 07:30 and 08:00 daily.
体温上报助手 简介 每天 10:30 GMT+8 自动完成体温上报，如想修改定时运行的时间，可修改 .github/workflows/SduHealthReport.yml 中 schedule 属性。 如果当日有异常，请手动在小程序端/PC 端填写！
hash_spider A module for CME that spiders hashes across the domain with a given hash. Installation Simply copy hash_spider.py to your CME module folde
Sportyfin Description Stream sports events straight from your Jellyfin server. Sportyfin allows users to scrape for live streamed events and watch str
PyMDL An API for MyDramaList(MDL) based on webscraping for python. Description An API for MDL to make your life easier in retriving and working on dat
KKBA Intruoduction A low-code tool that generates python crawler code based on curl or url Requirement Python = 3.6 Install pip install kkba Usage Co
Dailyiptvlist.com scraper Info Made in python Linux only script Script requires to have wget installed Running script Clone repository with: git clone
Twitter Turbo / Auto Claimer / Swapper Version: 1.0 Last Update: 01/26/2022 Use this at your own descretion. I've only used this on test accounts and
This is a quick-and-dirty tool used to scrape bitcoin/bitcoin pull request and commentary data. Each output/pr number folder contains comments.json:
TikTok Scraper An utility library to scrape data from TikTok hassle-free Go to the website » View Demo · Report Bug · Request Feature About The Projec
crawlersuseragents This Python script can be used to check if there is any differences in responses of an application when the request comes from a se
About 千葉県の地域別の詳細感染者統計(Excelファイル) をCSVに変換し、かつ地域別の日時感染者集計値を出力するスクリプトです。 Requirement POSIX互換なシェル, e.g. GNU Bash (1) curl (1) python = 3.8 pandas = 1.1.
Frontera Overview Frontera is a web crawling framework consisting of crawl frontier, and distribution/scaling primitives, allowing to build a large sc
STW-Collection Scrap The Web Collection; blog posts. This repo contains Scrapy sample code to scrap the following kind of websites: Do you want to lea
QuickInfo PYPI link : https://pypi.org/project/quickinfo/ This is the library that you've all been searching for, it's built for developers and allows
wallstreetbets-tracker Using Python and Pushshift.io to Track stocks on the WallStreetBets subreddit.
Open Crawl Vietnamese Text This repo contains crawled Vietnamese text from multiple sources. This list of a topic-centric public data sources in high
Pixiv-spider A Pixiv spider module WARNING It's an unfinished work, browsing the code carefully before using it. Features 0004 - Readme.md updated, co
Robox is a simple library with a clean interface for exploring/scraping the web or testing a website you’re developing. Robox can fetch a page, click on links and buttons, and fill out and submit for
pytrends Introduction Unofficial API for Google Trends Allows simple interface for automating downloading of reports from Google Trends. Only good unt
5 Jan 04, 2022
23 Sep 15, 2022
37 Sep 08, 2022
38 Nov 05, 2022
6 Dec 10, 2022
8 Sep 20, 2021
1 Oct 16, 2021
6 May 02, 2022
8 Oct 12, 2022
6 Jan 08, 2023
13 Dec 27, 2022
1 Nov 29, 2021
1.2k Jan 02, 2023
15 Jun 08, 2022
2 Dec 13, 2021
91 Dec 08, 2022
4 Jan 05, 2022
1 Nov 14, 2021
79 Nov 27, 2022
2.6k Dec 28, 2022