SCRAPEGOAT
Scrapegoat is a python library that can be used to scrape the websites from internet based on the relevance of the given topic irrespective of language using Natural Language Processing. It can be mainly used for non-English language to get accurate and relevant scraped text.
Concept
Initially the data is scraped from a website and processed ( to remove English words if the data required is in other language). The BERT model is feed with processed data and topic to compute the cosine similarity of the given topic with each word of the scraped data then mean of cosine similarity scores of is computed. If the mean is greater than threshold then scraped data is generated as output. Also there is a section where we are using Adaptive threshold.
BERT Model
BERT, which stands for Bidirectional Encoder Representations from Transformers, is based on Transformers, a deep learning model in which every output element is connected to every input element, and the weightings between them are dynamically calculated based upon their connection. The BERT framework was pre-trained using text from Wikipedia. The transformer is the part of the model that gives BERT its increased capacity for understanding context and ambiguity in language. The transformer does this by processing any given word in relation to all other words in a sentence, rather than processing them one at a time. By looking at all surrounding words, the Transformer allows the BERT model to understand the full context of the word, and therefore better understand searcher intent.
Cosine Similarity
Cosine similarity is one of the metrics to measure the text-similarity between two documents irrespective of their size in Natural language Processing. A word can be represented in the vector form, therefore the text documents are represented in n-dimensional vector space. If the Cosine similarity score is 1, it means two vectors have the same orientation. The value closer to 0 indicates that the two documents have less similarity. The Cosine similarity of two documents will range from 0 to 1.
Multi Processing
The multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. The basic ideology of Multi-Processing is that if you have an algorithm that can be divided into different workers (small processors/cores), then you can speed up the program. Machines nowadays come with 4,6,8 and 16 cores, therefore parts of the code can be deployed in parallel.
Using Scrapegoat
The examples/test.py file contains these
url = "https://hindi.newslaundry.com/2021/01/22/loan-developing-countries-and-epidemics#:~:text=120%20%E0%A4%A8%E0%A4%BF%E0%A4%AE%E0%A5%8D%E0%A4%A8%20%E0%A4%94%E0%A4%B0%20%E0%A4%AE%E0%A4%A7%E0%A5%8D%E0%A4%AF%E0%A4%AE%20%E0%A4%86%E0%A4%AF,8.1%20%E0%A4%85%E0%A4%B0%E0%A4%AC%20%E0%A4%A1%E0%A5%89%E0%A4%B2%E0%A4%B0%20%E0%A4%B9%E0%A5%8B%20%E0%A4%97%E0%A4%AF%E0%A4%BE."
topic = "Debt of developing countries"
language = 'hi'
if __name__=="__main__":
from scrapegoat.utils import automate
from scrapegoat.main import getLinkData
text,score = getLinkData(url, topic, language=language, tag='p')
print(text, score)
3 Feb 13, 2022
7 Dec 3, 2022
7 Sep 30, 2022
1 Jan 12, 2022
2 Jul 20, 2022
5 Jun 23, 2022
3 Feb 18, 2022
15 Jul 16, 2022
105 Dec 29, 2022
1 Nov 10, 2021
1 Dec 03, 2022
14 Aug 12, 2022
1 Dec 30, 2021
71 Oct 04, 2022
14 Dec 15, 2022
1 Dec 15, 2022
1 Jan 28, 2022
6 Apr 05, 2022
1 Dec 05, 2021
2 Jul 14, 2022
5 Nov 24, 2022
299 Dec 15, 2022
10 Aug 18, 2022
3 Feb 10, 2022
0 Nov 22, 2021
1.6k Jan 01, 2023
859 Dec 29, 2022
2 Nov 01, 2021