PathfinderMonsterDatabase - A database of all monsters in Pathfinder 1e, created by parsing aonprd.com

Overview

PathfinderMonsterDatabase

A database of all monsters in Pathfinder 1e, created by parsing aonprd.com

Setup

Run the following line to install all required libraries:

pip install -r requirements.txt

Downloading and parsing the data

  1. Run download_page_list.py in order to download the list of monster entry URLs:
python download_page_list.py

By default, this will get all monsters, NPCs, and mythic monsters. You can also specify a URL as a parameter to the script to get just the monster entries from there (e.g. https://aonprd.com/Monsters.aspx?Letter=All). Alternatively, you can just create the data/urls.txt file yourself.

  1. Run download_pages.py to download each individual monster entry page:
python download_pages.py

This will take a bit. The script limits itself to a maximum of 5 requests a second to not overload aonprd.com, but you can adjust this number if you are still having trouble. You can also give additional parameters to the script if you want to pull from a different file other than data/urls.txt, or if you want to write the results to a folder other than data.

  1. Run main.py to parse the raw HTML into a database:
python main.py

This script is where the magic happens. If you want to adjust anything from adding special cases, changing how parsing is done, or changing the output format of the database, you'll need to change it here. The script will pull from the data folder by default, but you can give it a different folder as an argument (where it will look for a urls.txt and appropriately named html files). This script will also look for the broken_urls.txt file, which contains all URLs to ignore, usually because their HTML is broken or their monster statblocks are malformed in some way. If you want to exclude 3.5e monster entries, change the include3_5 variable at the top of this script to False. When finished, this script produces a data.json containing the database.

Exploring the data

Run explore_data.py in order to look through the data:

python -i explore_data.py

This script will load up the database and create some useful dictionaries for accessing it, which you can explore using python's interactive mode. It also gives you some useful utility functions.

The dictionaries it creates are:

  • d - the database. Index it with a url, and you can see the statblock of the monster at that URL. Example usage:
>>> pprint(d["https://aonprd.com/MonsterDisplay.aspx?ItemName=Wolf"])
{'AC': {'AC': 14,
        'components': {'dex': 2, 'natural': 2},
        'flat_footed': 12,
        'touch': 12},
 'BAB': 1,
 'CMB': 2,
 'CMD': 14,
 'CMD_other': '18 vs. trip',
 'CR': 1,
 'HP': {'HP': 13, 'long': '2d8+4'},
 'XP': 400,
 'ability_scores': {'CHA': 6,
                    'CON': 15,
                    'DEX': 15,
                    'INT': 2,
                    'STR': 13,
                    'WIS': 12},
 'alignment': 'N',
 'attacks': {'melee': [[{'attack': 'bite',
                         'bonus': [2],
                         'entries': [[{'damage': '1d6+1'}, {'effect': 'trip'}]],
                         'text': 'bite +2 (1d6+1 plus trip)'}]]},
 'desc_long': 'Wandering alone or in packs, wolves sit at the top of the food '
              'chain. Ferociously territorial and exceptionally wide-ranging '
              'in their hunting, wolf packs cover broad areas. A wolf’s wide '
              'paws contain slight webbing between the toes that assists in '
              'moving over snow, and its fur is a thick, water-resistant coat '
              'ranging in color from gray to brown and even black in some '
              'species. Its paws contain scent glands that mark the ground as '
              'it travels, assisting in navigation as well as broadcasting its '
              'whereabouts to fellow pack members. Generally, a wolf stands '
              'from 2-1/2 to 3 feet tall at the shoulder and weighs between 45 '
              'and 150 pounds, with females being slightly smaller.',
 'desc_short': 'This powerful canine watches its prey with piercing yellow '
               'eyes, darting its tongue across sharp white teeth.',
 'ecology': {'environment': 'cold or temperate forests',
             'organization': 'solitary, pair, or pack (3-12)',
             'treasure_type': 'none'},
 'feats': [{'name': 'Skill Focus (Perception)'}],
 'initiative': {'bonus': 2},
 'saves': {'fort': 5, 'ref': 5, 'will': 1},
 'senses': {'low-light vision': True, 'scent': True},
 'size': 'Medium',
 'skills': {'Perception': {'_': 8},
            'Stealth': {'_': 6},
            'Survival': {'_': 1, 'scent tracking': 5},
            '_racial_mods': {'Survival': {'when tracking by scent': 4}}},
 'sources': [{'link': 'http://paizo.com/products/btpy8auu?Pathfinder-Roleplaying-Game-Bestiary',
              'name': 'Pathfinder RPG Bestiary',
              'page': 278}],
 'speeds': {'base': 50},
 'title1': 'Wolf',
 'title2': 'Wolf',
 'type': 'animal'}
  • unique_leaves. This dictionary shows the unique values of every leaf in the database. You can use it when looking for what values a certain leaf can take - e.g. see all speeds, sizes, senses, and so on. Example usage:\
>>> p(unique_leaves["speeds"]["fly"])
[
  10,
  15,
  20,
  30,
  40,
  45,
  50,
  60,
  70,
  80,
  90,
  100,
  110,
  120,
  150,
  160,
  180,
  200,
  250,
  480
]
  • unique_leaves_lookups. This dictionary is identical to unique_leaves, except that each value instead points to a list of all URLs that have that value for that property. This dictionary is useful once you've found something of interest in unique_leaves and want to see where it appears. For example, let's say after running the previous query you want to find out which entries have a 480-foot fly speed - example usage:
>>> p(unique_leaves_lookup["speeds"]["fly"][480])
[
  "https://aonprd.com/MonsterDisplay.aspx?ItemName=Anemos"
]
  • unique_leaves_counts. This dictionary is identical to `unique_leaves, except that it shows how many times each unique leaf appears. It's useful when wanting to get a sense of the relative rarity of different properties. For example, let's say you wanted to know common creatures of different sizes are in Pathfinder - example usage:
>>> p(unique_leaves_counts["size"])
{
  "Huge": 348,
  "Medium": 1471,
  "Tiny": 208,
  "Diminutive": 58,
  "Small": 364,
  "Large": 757,
  "Fine": 22,
  "Gargantuan": 154,
  "Colossal": 86
}

The utility functions this script provides are:

  • p and pprint - for pretty-printing dictionaries, use whichever you prefer. pprint produces more compact output, p does better sorting.

  • search - for searching through a dictionary or sub-dictionary. You can use this function to find something across multiple different parts of the hierarchy. For example, if you wanted to find all entries that mention the Limited Wish spell, you could run

>>> p(search(d, "limited wish", caseSensitive=False))

This will find any entry that has limited wish in it, whether it appears in the Spells or Spell-Like Abilities blocks, in the description, or wherever else. (It will also show you where in the entry the "limited wish" string appears.)

This does not search the raw entries - only the database - so if you search for stuff like "Caster Level", for example, you won't find every single monster with a Spells block, since that block is parsed into data that does not contain the string "caster level".

You can change case sensitivity with the caseSensitive parameter (default True), and/or you can pass regex=True to use a regex instead of a string to search.

Owner
Yoni Lerner
Yoni Lerner
Makes google's political ad database actually useful

Making Google's political ad transparency library suck less This is a series of scripts that takes Google's political ad transparency data and makes t

The Guardian 7 Apr 28, 2022
Лабораторные работы по Postgresql за 5 семестр

Практикум по Postgresql ERD для заданий 2.x: ERD для заданий 3.x: Их делал вот тут Ниже есть 2 инструкции — по установке postgresql на manjaro и по пе

Danila 10 Oct 31, 2022
This project is related to a No-SQL database, whose data are referred to autoctone botanic species

This project is related to a No-SQL database, whose data are referred to autoctone botanic species. The final goal is creating a function that performs the estimation of the ornamental value, given t

Amatofrancesco99 2 Mar 08, 2022
ChaozzDBPy - A python implementation based on the original ChaozzDB from Chaozznl with some new features

ChaozzDBPy About ChaozzDBPy is a python implementation based on the original Cha

Igor Iglesias 1 May 25, 2022
Simpledb-py: Simple JSON database

Simpledb-py: Simple JSON database

тейлс 2 Feb 09, 2022
Given a metadata file with relevant schema, an SQL Engine can be run for a subset of SQL queries.

Mini-SQL-Engine Given a metadata file with relevant schema, an SQL Engine can be run for a subset of SQL queries. The query engine supports Project, A

Prashant Raj 1 Dec 03, 2021
A simple GUI that interacts with a database to keep track of a collection of US coins.

CoinCollectorGUI A simple gui designed to interact with a database. The goal of the database is to make keeping track of collected coins simple. The G

Builder212 1 Nov 09, 2021
PathfinderMonsterDatabase - A database of all monsters in Pathfinder 1e, created by parsing aonprd.com

PathfinderMonsterDatabase A database of all monsters in Pathfinder 1e, created by parsing aonprd.com Setup Run the following line to install all requi

Yoni Lerner 11 Jun 12, 2022
Python object-oriented database

ZODB, a Python object-oriented database ZODB provides an object-oriented database for Python that provides a high-degree of transparency. ZODB runs on

Zope 574 Dec 31, 2022
Python function to query SQLite files stored on S3

sqlite-s3-query Python function to query a SQLite file stored on S3. It uses multiple HTTP range requests per query to avoid downloading the entire fi

Michal Charemza 189 Dec 27, 2022
This repo contains the backend of the KMK project

KMK Backend This repository contains the backend part of the KMK project Demo Watch it on Youtube Getting started Pre-commit hooks After you cloned th

21 Nov 26, 2022
Mongita is to MongoDB as SQLite is to SQL

Mongita is a lightweight embedded document database that implements a commonly-used subset of the MongoDB/PyMongo interface. Mongita differs from MongoDB in that instead of being a server, Mongita is

Scott Rogowski 809 Jan 07, 2023
TelegramDB - A library which uses your telegram account as a database for your projects

TelegramDB A library which uses your telegram account as a database for your projects. Basic Usage from pyrogram import Client from telegram import Te

Kaizoku 79 Nov 22, 2022
A Modular MWDB Utility to Collect Fresh Malware Samples

MWDB Feeds A Modular MWDB Utility to Collect Fresh Malware Samples This project is FREE as in FREE 🍺 , use it commercially, privately or however you

c3rb3ru5 32 Jul 07, 2022
Python function to extract all the rows from a SQLite database file while iterating over its bytes, such as while downloading it

Python function to extract all the rows from a SQLite database file while iterating over its bytes, such as while downloading it

Department for International Trade 16 Nov 09, 2022
Codeqlcompile - 自动反编译闭源应用,创建codeql数据库

codeql_compile 自动反编译闭源应用,创建codeql数据库 准备 首先下载ecj.jar和idea提供反编译的java-decompiler.ja

236 Jan 05, 2023
Youtube Kanalinda tanittigim ve Programladigim SQLite3 ile calisan Kütüphane Programi

SQLite3 Kütüphane Uygulamasi SQLite3 ile calisan Kütüphane Arayüzü Yükleme Yerel veritabani olusacaktir. Yaptiginiz islemler kaybolmaz! Temel Gereksin

Mikael Pikulski 6 Aug 13, 2022
pickleDB is an open source key-value store using Python's json module.

pickleDB pickleDB is lightweight, fast, and simple database based on the json module. And it's BSD licensed! pickleDB is Fun import pickledb

Harrison Erd 738 Jan 04, 2023
A very simple document database

DockieDb A simple in-memory document database. Installation Build the Wheel Fork or clone this repository and run python setup.py bdist_wheel in the r

1 Jan 16, 2022
Monty, Mongo tinified. MongoDB implemented in Python !

Monty, Mongo tinified. MongoDB implemented in Python ! Was inspired by TinyDB and it's extension TinyMongo

David Lai 523 Jan 02, 2023