Utility for downloading works from AO3 (Archive Of Our Own)

Related tags

Downloaderfroyo
Overview

froyo

demo

A small graphical application for batch downloading works from Archive Of Our Own (AO3).

Curate a fic repo of your own today :)

Features

  • Batch downloading works to supported formats (AZW3, EPUB, HTML, MOBI, PDF)
  • Download individual works by their URLs
  • Download works in a series
  • Download all bookmarks in your account (public and private)
  • Download all works and (public) bookmarks from another user
  • Download works in any arbitrary AO3 search result, collection or tag
  • Supports multi-threaded requests and retrying after being rate limited

Installation

Pre-compiled executable binaries

The latest binary releases can be found here. Please download the appropriate binary for your platform version, unzip the folder and run the executable. No installation is required.

Running from source

If you are using another platform, you can run the application from source as long as Python is supported on your platform.

Requirements: Python >= 3.8 (with the pip package manager)

  1. Clone the repository (or download the .ZIP).
$ git clone https://github.com/fIux-dev/froyo.git
  1. Go into the froyo directory and install the required packages.
$ cd froyo
$ pip install wheel
$ pip install -r requirements.txt
  1. Run.
$ python3 froyo.py

Building the binary for release

  1. Peform the same first two steps in the above Running from source section.
  2. Install PyInstaller.
$ pip install pyinstaller
  1. Build the executable with PyInstaller. After running this step, there should be a new folder generated named dist/. The binary will be inside this folder.
$ pyinstaller --noconsole --onefile froyo.py
  1. Copy static resources to the distribution folder.
$ cp -r resources dist/resources
  1. Now the dist/ folder containing the binary is ready for distribution.

Usage

Downloading works, series, bookmarks

Please see the animated previews for an example of how to interact with the application.

It is not required to log in to download works, however some functionality may be restricted. Certain works may be restricted to logged-in users and cannot be viewed in a guest session. In addition, bookmarks can only be imported when logged in.

Downloading works from any AO3 listing

generic2

To download all works in a tag, collection, or search result, use the Add generic URL button. You can simply paste the URL listing the works in the field and select a page range to download. Use the range 1 to 0 to get works in all pages of the listing.

Note: Search appears to be rate limited a lot more aggressively than just accessing works by ID. If you are attempting to download a large number of results, note that this may take a long time.

Rate limiting

retry

If you are attempting to download a large number of works, you may be rate limited by AO3. The application will attempt to retry requests that failed due to rate limiting after some period of time, with exponential waiting times. If it looks like the retry times are too long, you can also manually try to trigger downloads by clicking the Download all button again.

If you find that you are still getting errors trying to load series or users, you can try enabling the rate limit option in settings.

It is advisable to keep browsing of AO3 through the browser to a minimum while using the application. This is because rate limiting is shared across all your connections. If you are being rate limited in the application, you will also see a "Retry later" message when trying to access AO3 in the browser.

Settings

image

AO3 Login

A username and password can be entered in this section to authenticate with AO3. This will allow you to import bookmarks from your account, as well as view restricted works that are only available to logged-in users. It is not recommended to select "Remember me" on public computers.

Downloads

This section controls settings for the downloads. The directory the downloaded files will be saved to can be specified here. By default, it should be the Downloads folder in your home directory. If you are logged in, the downloads will appear in a subfolder with the same name as your AO3 username. A filetype for the downloads can also be selected from the supported formats.

Engine

This section controls behaviors of the application itself. Allowing multithreading will give slightly more responsive performance. The concurency limit sets the maximum number of parallel worker threads.

Rate limiting is not enabled by default, instead the application attempts to retry with a timeout in between requests if a rate limit error occurs. If you are attempting to download all works or bookmarks from a user or series and the number of works is very large, it is recommended to turn on rate limiting. This will limit the number of requests to 12 per minute, and will slow down the speed of the application significantly, but should prevent any issues with being able to load users or series.

Troubleshooting

A log.txt file is generated in the same directory as the application when it is run. If you encounter any crashes or errors, please create an issue and attach this log file.

Known issues

  • A black screen is shown for a bit when the application first starts. This is due to font loading taking a while.
  • Bookmarked series cannot be downloaded yet. A workaround is to bookmark each work in the series individually.
  • Currently it is not possible to cancel loading bookmarks, series, user works, user bookmarks, generic URL pages until the individual works are loaded. A workaround is to restart the application.
  • If you have saved a login in your settings file, and you are currently being rate limited by AO3, you will not be able to start up the application and will see a message about rate limiting. A workaround is to clear your login information in settings.ini.
  • Clicking Download all while bookmarks, series, collections, etc. are loading will not download the individual works in them. At least the individual work IDs must be loaded for the download to be queued.
  • Closing the application while requests are ongoing can take a while to respond, in order to terminate all background threads cleanly. If rate limiting is enabled, closing the application will take even longer and may appear to freeze as it is shutting down.
  • Ctrl + C through the command-line will not terminate the application cleanly. This may cause issues with the log file not being written completely, etc.
  • If you have a lot of bookmarks and/or works, the downloader will likely be rate limited by AO3. Right now, attempting to add works/bookmarks from a user or works from a series can potentially fail if the list is very large. This is because even if the request is retried, we may be rate limited trying to fetch the entire list again on retry. The workaround is to enable the rate limiting flag in the settings. Alternatively, you can attempt to submit the page as a generic URL instead to have individual retry attempts per page.
  • Enabling rate limiting is pretty buggy, you may see some unexpected errors like pop from empty list. These originate from the AO3 API's requester module and show up when rate limiting occurs. Please attempt to retry the download when the work appears to be loaded.

License

MIT

You might also like...
Userscript qutebrowser for downloading audio / video from youtube using aria2

Yt-Downloader Userscript qutebrowser for downloading video / audio from youtube using aria2 by hint links. Requirements Rofi youtube-dl aria2 dunst In

A very fast file streaming bot used for streaming and downloading movies
A very fast file streaming bot used for streaming and downloading movies

FileStreamBot GIVE A STAR AND FORK ELSE NO MORE OPENSOURCE A Telegram bot to turn all media and documents files to web link . Report a Bug | Request F

Python script for downloading audio from YouTube songs/videos.

Python script for downloading audio from YouTube songs/videos. All you have to do is specify the path to your folder and then type song's/video's name and the sound will be downloaded into your folder.

A web app for downloading Facebook comments as a csv file
A web app for downloading Facebook comments as a csv file

Facebook Comment Downloader A small web app for downloading comments from a public facebook page post. Comment downloading from https://github.com/min

Youtube_dl_helper - A hacky python script meant to automate the process of downloading mp3 files from YouTube using youtube-dl library

youtube_dl_helper A helper program meant to automate the process of downloading mp3 files from YouTube using youtube-dl library Dependencies In order

Gogoanime-dl - Gogoanime downloader for downloading anime.

gogoanime-dl With this script, you can download episodes of your favorite anime from Gogoanime. The current site that's developed against is https://w

A Python package for downloading / archiving all available episodes from a podcast RSS feed.
A Python package for downloading / archiving all available episodes from a podcast RSS feed.

allcasts 📻 🗃 A Python package for downloading all available episodes from a podcast RSS feed. Useful for making private archives of your favourite p

A simple python script and it's used for mp4 type video downloading from youtube.
A simple python script and it's used for mp4 type video downloading from youtube.

This is a simple python script and it's used for mp4 type video downloading from youtube. also, it's used inbuilt python module pytube. Furthermore, I know we have so many apps and online websites to do the same thing so it's just an experiment to study how to do those things in python.

Heroic-gogdl - GOG Downloading module for Heroic Games Launcher

heroic-gogdl GOG download module for Heroic Games Launcher Purpose This will tak

Comments
  • Froyo as example app on Dear PyGui showcase

    Froyo as example app on Dear PyGui showcase

    Hi,

    Froyo looks like a good example app to add to the Dear PyGui Showcase (https://github.com/hoffstadt/DearPyGui/wiki/Dear-PyGui-Showcase).

    Is that OK by you?

    Thank you!

    opened by bandit-masked 3
  • Download button not working

    Download button not working

    I am trying to download my bookmarks (which, yeah, I can admit is a lot) but I can't to download the ones that has loaded, the button seems like it isn't doing anything.

    opened by helpmeimdum 0
Releases(v0.2.0-beta)
Owner
flux
side account
flux
1Fichier Download Manager.

1fichier-dl 1Fichier Download Manager. Features ⭐ Manage your downloads ⭐ Bypass time limits Credits All icons, including the app icon, were provided

manuGMG 470 Oct 08, 2022
Web Downloader With Python

Web Downloader Introduction This module will provide API to download the webpage components : html file, image file, css fil, javascript file, href li

3 Dec 28, 2022
Download YOUR files, documents from vk.

vk-documents-downloader Кароч эта симпл херня качает все ВАШИ документы с вк. Или я еблан, но в гх и тмб гугле я подобного не нашел. py main.py Login:

4 Jun 10, 2022
Python script to download (TCR) genes from IMGT/GENE-DB

IMGTgeneDL 0.1.0 Jamie Heather | CCR @ MGH | 2021 This script provides an alternative way to access TCR and IG genes stored in IMGT/GENE-DB. It's prim

Jamie Heather 1 Mar 30, 2022
👻🟡 Download all Snapchat video & photo memories from a data export.

Snapchat "Memories" Fetcher In compliance with the California Consumer Privacy Act of 2018 (“CCPA”), businesses which collect and store user data must

Todd Birchard 18 Dec 26, 2022
Download India Stocks Historical Data

Kite Helper - Download Stock Market Data 🌎 Website Simple Application to Download any stock market data in .csv format using Kite 🏃‍♂️ Running Serve

Pishang Ujeniya 12 Dec 06, 2022
Search & download music from a certain streaming service

Search & download music from a certain streaming service

mat 2 Mar 11, 2022
Discord Nitro Generator + Checker

Discord Nitro Generator + Checker Usage Download the project files and run main.py You will be prompted with 2 questions the first one being the amoun

509 Jan 02, 2023
YoutubeDownloader - Download any public Playlist from Youtube

YoutubeDownloader Download any public Youtube Channel / Playlist Features Bulk d

17 Nov 12, 2022
A python module to download ISO Standards

ISO Standards Downloader A python module to download ISO Standards from https://standards.iso.org/iso-iec/ Report Bug · Request Feature Table of conte

Daniel 1 Dec 29, 2021
A web app for downloading Facebook comments as a csv file

Facebook Comment Downloader A small web app for downloading comments from a public facebook page post. Comment downloading from https://github.com/min

WSDOT 23 Jan 04, 2023
An automatic beatmapset downloader via txt file, suitable for tourney mappools.

Pooler Pooler is a bulk osu! mapset downloader, perfect for use with osu! Tournament Mappools. Prerequisites Python 3.10 Requests (pip install request

Thomas 0 Feb 11, 2022
Organize your downloads easily with DownloadOrganizer

DownloadOrganizer Organize your downloads organize your downloads easily with DownloadOrganizer Instilation how to install DownloadOrganizer Method 1:

1 Dec 02, 2021
Userscript qutebrowser for downloading audio / video from youtube using aria2

Yt-Downloader Userscript qutebrowser for downloading video / audio from youtube using aria2 by hint links. Requirements Rofi youtube-dl aria2 dunst In

Ara 0 Dec 11, 2021
TikTok - TikTok Bot to download video or audio from TikTok

TikTok - TikTok Bot to download video or audio from TikTok

JMTHON 51 Mar 04, 2022
ASF Sentinel-1 Metadata Download tool

ASF Sentinel-1 Metadata Download tool Copyright: 2021-2022 Antonio Valentino Small Python tool (asfsmd) that allows to download XML files containing S

Antonio Valentino 9 Dec 04, 2022
Download courses from khanacademy.org

khan-dl A python script to download courses from Khan Academy using youtube-dl and beautifulsoup4.

rand-net 806 Jan 03, 2023
Download your bandcamp collection using this python script.

bandcamp-downloader Download your Bandcamp collection using this python script. It requires you to have a browser with a logged in session of bandcamp

72 Dec 20, 2022
FireDM is a python open source (Internet Download Manager) with multi-connections, high speed engine, it downloads general files and videos from youtube and tons of other streaming websites .

python open source (Internet Download Manager) with multi-connections, high speed engine, based on python, LibCurl, and youtube_dl https://github.com/firedm/FireDM

1.6k Apr 12, 2022