A downloader for the ISIS service of TU Berlin

Related tags

Downloaderisis_dl
Overview

isis_dl

Tests

A downloading utility for the ISIS tool of TU-Berlin.

Version 0.4

Features

  • Downloads all Material from all courses of your ISIS page.
  • Efficient and dynamic checksum computing for a very good file recognition.
  • You can whitelist / blacklist courses with a given course ID.
  • Multithreaded: A fixed number of threads can be selected at start time.
  • Compatibility: This library will run with any python interpreter that is >= 3.8.

Binary functionality:

  • Building of checksums from existing files.
  • Automatic unpacking of archives.

TL;DR

  1. Use this library instead of isia-tub. It provides a superset of features while having improved performance.
  2. Install using pip install isis_dl. For a manual installation clone this repository and pip install .
  3. For a detailed explanation of Command-Line flags please run isisdl -h.
  4. The first time you run the program you will be prompted if you want to save your password. Look at section encryption for more details.

Installation

You will need a working python3.8 interpreter or above. The script will fail for python3.7 as some new python3.8 features are used.

The recommended installation is via pip - a package manager for python. If pip is not yet installed with the python interpreter run

python3 -m ensurepip

to bootstrap pip.

pip (PyPi)

I have uploaded the repository to the Python Package index (PyPi) where one can download it with the command

pip install isis_dl

Note that you are executing arbitrary code as user. Do this at your own risk!

To run the downloader simply type

isisdl

into your favorite shell.

Please note that, if the virtual environment feature is not used, the ~/.local/bin directory must be in the PATH, otherwise the executable isisdl will not be found.

Manual

Please note that you have to be in a virtual environment in order for this to work as the installation fails otherwise.

Steps:

  • Clone this repository
  • cd isis_dl
  • pip install -e .

Developing

If you want to actively contribute to this repository you will want to install the package in editable mode along with the development requirements:

pip install -r requirements_dev.txt

This creates a symlink to the source code in the pip package location. It will be treated as if it was installed there directly.

There is no method of installation without pip - as the source code expects the module isis_dl to be installed as a package.

Benchmarks

For a comparison between isia-tub and isis_dl please see the Benchmark.md file.

Documentation on features

I am planning on moving this part of the documentation to a dedicated doc site.

File recognition

The file recognition is handled in src/isis_dl/backend/checksums.py.

The main idea is to download a small portion of the file and calculate a hash based on that.

As the requests library provides a file stream, one can only download the first n Bytes and calculate the hash. The problem with this idea is that some files have a header, which is permanently changing.

Unfortunately I don't have an idea why this is the case. In order to circumvent this problem the first portion of the file is skipped based on the file type. The lookup table is located in src/isis_dl/share/settings.py - with the variable being checksum_num_bytes.

The format is <extension>: (<#bytes to ignore>, <#bytes to read>).

This means that one can also set the number of bytes to be read for each file type. For files which store a big header ( I'm looking at you .pdf) the number of bytes to be read is quite high. For others e.g. .mp4 it is not.

Note: If the file extension is not found the default entry None is consulted.

Advantages

  • Only download 512 Bytes of every file.
  • Can verify independently of directory structure / filenames.
  • Lookup is O(1) as a HashSet is used as a datastructure.
  • Up to 255 ** 512 unique files can be saved per course using this method.

Disadvantages

  • For every file in every course x Bytes have to be downloaded.
  • Files are bound to a course.

Note that a default value of 64 suffices to

Can store your password securely

The entire encryption is handled by the src/isis_dl/backend/crypt.py.

The encryption is handled via Fernet

Fernet guarantees that a message encrypted using it cannot be manipulated or read without the key. Fernet is an implementation of symmetric (also known as “secret key”) authenticated cryptography.

The key is generated based on a password you enter and then stored securely.

TODO: This is currently untested. Please enter your password manually for the moment.

Hash Settings

Beware: If you change these settings you will not be able to recover an encrypted file without restoring the settings. I would not recommend changing them.

You may select any hashing algorithm which is supported. This is any hashes.HashAlgorithm. You may also change the number of iterations, which will increase / decrease the time it takes to encrypt / decrypt respectively.

A customizable settings file

The file is located at src/isis_dl/share/settings.py. For the most part you will want to keep the default settings, but if they don't fit your needs, you may easily change them.

Download Directory

The default download directory is ~/isis_dl_downloads. As the intended installation is via pip, there is no good "current working directory", so one cannot use that.

What can be done, however, is migrating this directory to e.g. the Desktop/ or Documents/.

Acknowledgements

isia-tub

Consider checking out the gitlab

This was the original inspiration for this library. At the time isia did not offer the functionality of uri-encoding the password which lead me to create this library. I have recently implemented this functionality into isia in order to benchmark and test both solutions.

Comparison

Downloading my entire isis directory took 22m8s with isia. This is in contrast to the 11m16s it took with isis_dl

mCoding

The structure of this project is heavily inspired by the GitHub of mCoding. Consider giving their video about automated testing a shot.

You might also like...
MMDL (Mega Music Downloader) - A tool to easily download music.
MMDL (Mega Music Downloader) - A tool to easily download music.

mmdl - Mega Music Downloader What is mmdl ❓ TLDR: MMDL is a cli app which allows you to quickly and efficiently download one or multiple songs from Yo

apkizer is a mass downloader for android applications for all available versions.

apkizer apkizer collects all available versions of an Android application from apkpure.com Purpose Sometimes mobile applications can be useful to dig

Pantheon - The fastest YouTube downloader.
Pantheon - The fastest YouTube downloader.

A Youtube downloader written in Python3, using HTTP requests and an API.

Terminal based YouTube player and downloader
Terminal based YouTube player and downloader

termitube NOTE: THIS REPOSITORY IS A FORK OF mps-youtube as mps-youtube has been unmaintained for almost a year now. Features Search and play audio/vi

Youtube playlist downloader with full metadata support
Youtube playlist downloader with full metadata support

ytrake GUI tool to embed metadata for albums on Youtube with youtube-dl. Requires youtube-dl v2021.06.06. Post-processing Album metadata: Usage ytrake

Using Youtube downloader is the fast and easy way to download and save any YouTube video.
Using Youtube downloader is the fast and easy way to download and save any YouTube video.

Youtube video downloader using Django Using Django as a backend along with pytube module to create Youtbue Video Downloader. https://yt-videos-downloa

Advance Image Downloader/Extractor (Job) is a Python-Flask web-based app, which will help the user download the any kind of Images at any date and time over the internet. These images will get downloaded as a job and then let user know that the images have been downloaded by sending them a link over an email. A prometheus exporter for torrent downloader like qbittorrent/transmission/deluge
A prometheus exporter for torrent downloader like qbittorrent/transmission/deluge

downloader-exporter A prometheus exporter for qBitorrent/Transmission/Deluge. Get metrics from multiple servers and offers them in a prometheus format

bing image downloader app used to download bulk images for a specific search term created using streamlit and bing_image_downloader python packages
bing image downloader app used to download bulk images for a specific search term created using streamlit and bing_image_downloader python packages

bing image downloader app bing image downloader app is used to download bulk images for a specific search term. bing image downloader app gets the sea

Releases(untagged-44c24ae2f0ed9de798c2)
  • untagged-44c24ae2f0ed9de798c2(Oct 23, 2021)

    Version0.2

    Version 0.2 is out! yay

    Changelog:

    • Changed download mechanism from 1 Executor per course which downloads with args.num_threads to 1 Executor which downloads everything.
    • Faster instantiation of Files
    • All files get randomly shuffled for a better utilization
    • When interrupted, will finish all remaining downloads and then exit. If prompted again will instantly exit.
    • A better status indicator
    • Moved unzipping from auto → command line argument
    • More tests!
    Source code(tar.gz)
    Source code(zip)
A python module to download ISO Standards

ISO Standards Downloader A python module to download ISO Standards from https://standards.iso.org/iso-iec/ Report Bug · Request Feature Table of conte

Daniel 1 Dec 29, 2021
Download from HBO-MAX-BLIM-TV-Paramount

#HBO MAX- BlimTV -Paramount plus 4K Downloader Tool To download 4K HDR DV SDR from HBO MAX- BlimTV -Paramount plus Hello Fellow Developers/ ! Hi! M

4 Dec 25, 2021
YTPY Youtube Downloader Made by: Ferreira, Amarau and Rodric

YTPY Youtube Downloader Made by: Ferreira, Amarau and Rodric How to Install on Linux: sudo apt install python3 python3-pip git pip install pytube git

7 Nov 24, 2022
Archivist - Easily archive 📦 Download folder to Google Drive ☁️

Archivist Script for archiving Download folder by uploading unmodified files to a Google Drive folder. Modified files will remain in the Download fold

Timing Liu 3 Sep 30, 2022
Script for YouTube creators to share dislike count with their viewers.

Stahování disliků z YouTube - milafon Tento skript slouží jako možnost zobrazit divákům počet disliků u YouTube videí. Vyžaduje implementaci ze strany

4 Sep 28, 2022
The free and open-source Download Manager written in pure Python

The free and open-source Download Manager written in pure Python

pyLoad 2.7k Dec 31, 2022
Arxiv2Kindle is a simple script written in python that converts LaTeX source downloaded from Arxiv and recompiles it to better fit a Kindle or other similar reading devices.

Arxiv2Kindle is a simple script written in python that converts LaTeX source downloaded from Arxiv and recompiles it to better fit a read

Soumik Rakshit 8 Jul 09, 2022
Ebook downloader built using python

ebook-downloader Getting Started Open a terminal and run the following commands. git clone github.com/georgemunyoro/ebook-downloader cd ./ebook-downlo

George Munyoro 1 Oct 19, 2021
Simple package for Sublime Text 4; download URL's for local viewing and editing

URLDownloader This is a simple example package that allows you to easily download the contents of any web URL to edit locally. Given a URL, the packag

Terence Martin 3 Mar 05, 2022
Fast TikTok NO Watermark Video Downloader (username or url)

💎 TD [ TikDown v4 ] Star ⭐ if you want more Discord Server * discord.gg/onlp | Waxor#9999 Why not open source anymore ? * BECAUSE PEOPLE SKID, STEA

Tekky 26 Dec 01, 2022
Python-Youtube-Downloader - An Open Source Python Youtube Downloader

Python-Youtube-Downloader Hello There This Is An Open Source Python Youtube Down

Flex Tools 3 Jun 14, 2022
Downloads yiffer.xyz comics as images

yiffer-dl Downloads comics as images from yiffer.xyz.

Maxim 2 Mar 20, 2022
Desktop utility to download images/videos/music/text from various websites, and more

Desktop utility to download images/videos/music/text from various websites, and more

Kurt Bestor 11.2k Jan 08, 2023
Download any video from YouTube playlists

youtube-dl Download any videos from YouTube playlists. Requirements Python 3 BeautifulSoup4 PyQt PyQtWebEngine pytube pyyoutube python-decouple Usage

Antonio Fortin 1 Oct 26, 2021
Source code of paper: "HRegNet: A Hierarchical Network for Efficient and Accurate Outdoor LiDAR Point Cloud Registration".

HRegNet: A Hierarchical Network for Efficient and Accurate Outdoor LiDAR Point Cloud Registration Environments The code mainly requires the following

Intelligent Sensing, Perception and Computing Group 3 Oct 06, 2022
Script that allows to download portable installers of different versions Adobe software for macOS

What is this and for what This is a script that allows you to download portable installers of programs from Adobe for macOS with different versions. T

715 Jan 06, 2023
Libretrofuzz - Fuzzy Retroarch thumbnail downloader

Fuzzy Retroarch thumbnail downloader In Retroarch, when you use the manual scann

8 Nov 26, 2022
A Python package for downloading / archiving all available episodes from a podcast RSS feed.

allcasts 📻 🗃 A Python package for downloading all available episodes from a podcast RSS feed. Useful for making private archives of your favourite p

Lewis Gentle 5 Nov 20, 2022
A Telegram bot to download TikTok videos without any watermark.

TikTok Downloader Bot A Telegram bot to download TikTok videos without any watermark. Host on Heroku Youtube: Deployment Tutorial Demo: JayBee TikTok

Joy Biswas 184 Jan 04, 2023
A collection of modules I have created to programmatically search for/download imagery from live cam feeds across the state of California.

A collection of modules that I have created to programmatically search for/download imagery from all publicly available live cam feeds across the state of California. In no way am I affiliated with a

Chad Groom 5 Nov 21, 2022