Description
Python Crawler written Python 3. (Supports major Python releases Python3.6, Python3.7 and Python 3.8)
Installation and Use
Setup VirtualEnv
which python3 this will output the path of your python3
#now setup a python3 virtualenv
mkvirtualenv crawl3 -p $(which python3)
workon crawler
python main.py -d5 http://gotchacode.com // -d5 means crawl to the depth of 5.
Results:
And the output is:
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 29200.11it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 22563.50it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 21375.28it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 22227.37it/s]
CRAWLER STARTED:
https://vinitkumar.me, will crawl upto depth 2
https://vinitkumar.me/
http://changer.nl
https://twitter.com/vinitkme
https://vinitkumar.me/about
https://vinitkumar.github.io/vinit_kumar.pdf
https://vinitkumar.me/values
https://github.com/vinitkumar
https://vinitkumar.me/2013-03-24-life-has-changed/
https://vinitkumar.me/2013-03-24-my-javascript-love/
https://vinitkumar.me/2013-03-27-twitter-like-app-in-nodejs/
http://twitter.com/vinitkme
https://vinitkumar.me/2013-04-07-first-flight-and-vacation-after-months/
====================================================================================================
Crawler Statistics
====================================================================================================
No of links Found: 12
No of followed: 3
Found all links after 0.54s
Issues
Create an issue here if you encounter a bug: create-issue
3 Oct 4, 2022
1 Nov 14, 2021
46 Dec 16, 2022
1 Dec 30, 2021
1 Jan 10, 2022
47 Nov 23, 2022
1 Dec 20, 2021
2 Feb 17, 2022
105 Dec 29, 2022
11 Jul 27, 2022
2 Nov 01, 2021
185 Jul 23, 2022
3.8k Jan 02, 2023
2 Dec 14, 2022
32 Oct 10, 2022
6 Jan 08, 2023
2 Oct 27, 2022
1 Dec 26, 2021
0 Oct 10, 2021
2 Jan 04, 2022
1 Jan 06, 2022
2.2k Jan 05, 2023
27 Dec 18, 2022
2.6k Dec 31, 2022
1 Feb 10, 2022
2.1k Jan 06, 2023
163 Dec 30, 2022
1 Jan 04, 2022