PyQuery-based scraping micro-framework.

Last update: Jul 20, 2022

Related tags

Web Crawling demiurge

Overview

demiurge

PyQuery-based scraping micro-framework. Supports Python 2.x and 3.x.

Documentation: http://demiurge.readthedocs.org

Installing demiurge

$ pip install demiurge

Quick start

Define items to be scraped using a declarative (Django-inspired) syntax:

import demiurge

class TorrentDetails(demiurge.Item):
    label = demiurge.TextField(selector='strong')
    value = demiurge.TextField()

    def clean_value(self, value):
        unlabel = value[value.find(':') + 1:]
        return unlabel.strip()

    class Meta:
        selector = 'div#specifications p'

class Torrent(demiurge.Item):
    url = demiurge.AttributeValueField(
        selector='td:eq(2) a:eq(1)', attr='href')
    name = demiurge.TextField(selector='td:eq(2) a:eq(2)')
    size = demiurge.TextField(selector='td:eq(3)')
    details = demiurge.RelatedItem(
        TorrentDetails, selector='td:eq(2) a:eq(2)', attr='href')

    class Meta:
        selector = 'table.maintable:gt(0) tr:gt(0)'
        base_url = 'http://www.mininova.org'


>>> t = Torrent.one('/search/ubuntu/seeds')
>>> t.name
'Ubuntu 7.10 Desktop Live CD'
>>> t.size
u'695.81\xa0MB'
>>> t.url
'/get/1053846'
>>> t.html
u'<td>19\xa0Dec\xa007</td><td><a href="/cat/7">Software</a></td><td>...'

>>> results = Torrent.all('/search/ubuntu/seeds')
>>> len(results)
116
>>> for t in results[:3]:
...     print t.name, t.size
...
Ubuntu 7.10 Desktop Live CD 695.81 MB
Super Ubuntu 2008.09 - VMware image 871.95 MB
Portable Ubuntu 9.10 for Windows 559.78 MB
...

>>> t = Torrent.one('/search/ubuntu/seeds')
>>> for detail in t.details:
...     print detail.label, detail.value
... 
Category: Software > GNU/Linux
Total size: 695.81 megabyte
Added: 2467 days ago by Distribution
Share ratio: 17 seeds, 2 leechers
Last updated: 35 minutes ago
Downloads: 29,085

See documentation for details: http://demiurge.readthedocs.org

Why demiurge?

Plato, as the speaker Timaeus, refers to the Demiurge frequently in the Socratic dialogue Timaeus, c. 360 BC. The main character refers to the Demiurge as the entity who "fashioned and shaped" the material world. Timaeus describes the Demiurge as unreservedly benevolent, and hence desirous of a world as good as possible. The world remains imperfect, however, because the Demiurge created the world out of a chaotic, indeterminate non-being.

http://en.wikipedia.org/wiki/Demiurge

Contributors

Martín Gaitán (@mgaitan)

Comments

Reausable cleaning functions
You can now add a "clean" kwarg containing a function to a field.

This makes it easy to use quick filtering (I want this data to be an int) and to re-use functions such as parsedatetime.

score = demiurge.TextField(selector=".score .upvoted", clean=int)
opened by traverseda 5
proof of concept: subitem field

short rationale: Sometimes I need to scrap a page to retrieve the actual links where the items are. I would like a way to nest Item classes, analog (in some way) to a in ForeignKey / ManyToManyField in Django.

This is a first PR as a proof of concept, to discuss the idea and its API.

opened by mgaitan 5
RelatedItems only work across urls

An obvious use of RelatedItems (or a similar construct) is recursively mapping a comment tree. Right now there's no elegant way to do that.

An example

http://pastebin.com/WDL4RjkE

Reading through the actual code, I think I might be wrong about this. I'll try and make the docs clearer.

opened by traverseda 2
Use lib "requests" for downloading

I'm right now making use of https://pypi.python.org/pypi/requests-cache which creates a cache of the downloaded stuff magically, and it's awesome. So, I would like to be able to take advantage of it using demiurge.

I don't know if just as an option or as a replacement of pyquery downloader.

What do you think?

opened by jmansilla 2
docs: fix simple typo, ocurrence -> occurrence

There is a small typo in docs/index.rst.

Should read occurrence rather than ocurrence.

Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

opened by timgates42 1
Fix when no selector defined
the default selector is the whole page ('html') but this is applied through PyQuery.find wich traverses down. example:

In [2]: PyQuery('<html>hello</html>').find('html') Out[2]: [] In [3]: PyQuery('<html>hello</html>')('html') Out[3]: [<html>]
opened by mgaitan 1
support self reference in RelatedItem

RelatedItem('self'). Also, the relateditem's item class could be given by its name (i.eRelatedItem("ItemClass")` ) A typical use case is a listing page with a "next page" link.

opened by mgaitan 0

Releases(v0.2)

v0.2(Sep 20, 2014)
Added docs.

Added RelatedItem.

Added field clean support.

Source code(tar.gz)
Source code(zip)

Owner

Matias Bordese

GitHub Repository http://demiurge.readthedocs.org

Scraping weather data using Python to receive umbrella reminders

A Python package which scrapes weather data from google and sends umbrella reminders to specified email at specified time daily.

1 Aug 23, 2022

Scraping news from Ucsal portal with Scrapy.

NewsScraping Esse é um projeto de raspagem das últimas noticias, de 2021, do portal da universidade Ucsal http://noosfero.ucsal.br/institucional Tecno

0 Sep 30, 2021

Automatically download and crop key information from the arxiv daily paper.

Arxiv daily 速览功能：按关键词筛选arxiv每日最新paper，自动获取摘要，自动截取文中表格和图片。 1 测试环境 Ubuntu 16+ Python3.7 torch 1.9 Colab GPU 2 使用演示首先下载权重baiduyun 提取码:il87，放置于code/Pars

20 Jul 30, 2022

🐞 Douban Movie / Douban Book Scarpy

Python3-based Douban Movie/Douban Book Scarpy crawler for cover downloading + data crawling + review entry.

1 Dec 03, 2022

Pythonic Crawling / Scraping Framework based on Non Blocking I/O operations.

Pythonic Crawling / Scraping Framework Built on Eventlet Features High Speed WebCrawler built on Eventlet. Supports relational databases engines like

173 Dec 05, 2022

Automated Linkedin bot that will improve your visibility and increase your network.

LinkedinSpider LinkedinSpider is a small project using browser automating to increase your visibility and network of connections on Linkedin. DISCLAIM

2 Nov 26, 2021

Script for scrape user data like "id,username,fullname,followers,tweets .. etc" by Twitter's search engine .

TwitterScraper Script for scrape user data like "id,username,fullname,followers,tweets .. etc" by Twitter's search engine . Screenshot Data Users Only

19 Nov 17, 2022

Scrapes all articles and their headlines from theonion.com

The Onion Article Scraper Scrapes all articles and their headlines from the satirical news website https://www.theonion.com Also see Clickhole Article

0 Nov 17, 2021

Python scraper to check for earlier appointments in Clalit Health Services

clalit-appt-checker Python scraper to check for earlier appointments in Clalit Health Services Some background If you ever needed to schedule a doctor

16 Sep 17, 2022

Web-Scraping using Selenium Master

Web-Scraping using Selenium What is the need of Selenium? Some websites don't like to be scrapped and in that case you need to disguise your webscrapi

1 Oct 26, 2021

This is a web scraper, using Python framework Scrapy, built to extract data from the Deals of the Day section on Mercado Livre website.

Deals of the Day This is a web scraper, using the Python framework Scrapy, built to extract data such as price and product name from the Deals of the

1 Jan 12, 2022

PyQuery-based scraping micro-framework.

Related tags

Overview

demiurge

Installing demiurge

Quick start

Why demiurge?

Contributors

Comments

Reausable cleaning functions

proof of concept: subitem field

RelatedItems only work across urls

Use lib "requests" for downloading

docs: fix simple typo, ocurrence -> occurrence

Fix when no selector defined

support self reference in RelatedItem

Releases(v0.2)

v0.2(Sep 20, 2014)

Owner

Matias Bordese

Scraping weather data using Python to receive umbrella reminders

Scraping news from Ucsal portal with Scrapy.

Automatically download and crop key information from the arxiv daily paper.

🐞 Douban Movie / Douban Book Scarpy

Pythonic Crawling / Scraping Framework based on Non Blocking I/O operations.

Automated Linkedin bot that will improve your visibility and increase your network.

Script for scrape user data like "id,username,fullname,followers,tweets .. etc" by Twitter's search engine .

Scrapes all articles and their headlines from theonion.com

Python scraper to check for earlier appointments in Clalit Health Services

Web-Scraping using Selenium Master

This is a web scraper, using Python framework Scrapy, built to extract data from the Deals of the Day section on Mercado Livre website.

This is python to scrape overview and reviews of companies from Glassdoor.

Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

Pro Football Reference Game Data Webscraper

Use Flask API to wrap Facebook data. Grab the wapper of Facebook public pages without an API key.

Examine.com supplement research scraper!

Raspi-scraper is a configurable python webscraper that checks raspberry pi stocks from verified sellers

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster

A web scraper for nomadlist.com, made to avoid website restrictions.

A simplistic scraper made to download tons of random screenshots made by people.