Tool for installing and updating MiSTer cores and other files

Overview

MiSTer Downloader

This tool installs and updates all the cores and other extra files for your MiSTer. It also updates the menu core, the MiSTer firmware and the Linux system. The source for all downloads is the MiSTer Distribution repository.

The MiSTer Downloader is a substitute for the MiSTer Updater, and is meant to offer a more safe and robust experience, while being much faster.

As a drawback, the Downloader is not backwards compatible with the old INI files that were configured for the MiSTer Updater. In fact, as of today, this tool doesn't implement many fine-grained features that allow you to customize the updating process in depth. In case you value these features, consider to keep using the MiSTer Updater as usual. Both tools will coexist in the near future.

Setup and Usage

Download this ZIP file and extract downloader.sh to your /Scripts folder on your primary SD card (create that folder if it doesn't exist). You only need to perform this operation once, since this tool self-updates itself.

To use it, on your MiSTer main menu, go to the Scripts screen, and select downloader.

Options

You may create a /media/fat/downloader.ini file to tweak some parameters.

Here you can see the default parameters and the options that you may change:

[MiSTer]
; base_path is where most files will be installed
;   Useful for setups with USB storage, for example: '/media/usb0/'
base_path = '/media/fat/'

; base_system_path is where system files such as 'MiSTer' and 'menu.rbf' will be installed.
;   Warning: It is recommended to NOT change this setting regardless of your setup.
base_system_path = '/media/fat/'

; allow_delete options:
;   0 -> Don't allow this tool to delete anything at all.
;   1 -> Allow this tool to delete any old file from previous updates.
;   2 -> Allow this tool to delete only old cores that receive a new version.
allow_delete = 1

; allow_reboot options:
;   0 -> Don't allow this tool to ever reboot automatically.
;   1 -> Allow this tool to reboot the system after any system file has been updated.
;   2 -> Allow this tool to reboot the system only after Linux has been updated.
allow_reboot = 1

; update_linux options:
;   true -> Updates Linux when there is a new update (very recommended).
;   false -> Doesn't update Linux.
update_linux = true

; parallel_update options:
;   true -> Tries to more than one file simultaneously.
;   false -> Will only download one file at a time.
parallel_update = true

; downloader_timeout: Can be tweaked to increase the timeout time in seconds
;   It is useful to increase this value for users with slow connections.
downloader_timeout = 300

; downloader_retries: Can be tweaked to increase the retries per failed download
;   It is useful to increase this value for users with very unstable connections.
downloader_retries = 3

Roadmap

  • Initial Release
  • Cheats fetching
  • First-run optimisations
  • Configurable custom download filters
  • Handle duplicated games folders through symlinks (GBA <-> GBA2P, and GAMEBOY <-> GAMEBOY2P)
  • Integration with MiSTer binary
Comments
  • mra_alternatives don't seem to get updated

    mra_alternatives don't seem to get updated

    If the hash for the mra_alternatives summary file doesn't match the newest, mra_alternatives_summary.json.zip gets downloaded and the hashes are updated in the store (summary and files) but the files themselves aren't updated.

    opened by lagomorph 12
  • Unable to update

    Unable to update

    When I try to run aupdate from the MiSTer menu the part where it updates the CA-certs work. But after that it fails.

    The script will run normally now:
    Running MiSTer Downloader
    
    bash: line 122: BASH_SOURCE[0]: unbound variable
    Press any key to continue
    
    opened by Dankoman 6
  • error: invalid zip file with overlapped components (possible zip bomb)

    error: invalid zip file with overlapped components (possible zip bomb)

    I started receiving errors when trying to update today. Unzip is throwing an exception because it is recognizing the downloaded file as malicious. I am running the latest version of download.sh. The same issue occurs with any update script which downloads zipped files. Previously had no issues updating.

    /root# cat /media/fat/Scripts/.config/downloader/downloader.log START!

    Reading file: /media/fat/downloader.ini Reading 'distribution_mister' db section Reading 'jtcores' db section env: { "DOWNLOADER_LAUNCHER_PATH": "/media/fat/Scripts/downloader.sh", "DOWNLOADER_INI_PATH": null, "CURL_SSL": "--cacert /etc/ssl/certs/cacert.pem", "COMMIT": "856b6d5", "ALLOW_REBOOT": null, "UPDATE_LINUX": "true", "DEFAULT_DB_URL": "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip", "DEFAULT_DB_ID": "distribution_mister", "DEFAULT_BASE_PATH": null, "DEBUG": "false", "FAIL_ON_FILE_ERROR": "false" } config: { "databases": { "distribution_mister": { "db_url": "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip", "section": "distribution_mister" }, "jtcores": { "db_url": "https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip", "section": "jtcores" } }, "config_path": "/media/fat/downloader.ini", "base_path": "/media/fat", "base_system_path": "/media/fat", "storage_priority": "prefer_sd", "allow_delete": 1, "allow_reboot": 1, "update_linux": true, "parallel_update": true, "downloader_size_mb_limit": 100, "downloader_process_limit": 300, "downloader_timeout": 300, "downloader_retries": 3, "zip_file_count_threshold": 60, "zip_accumulated_mb_threshold": 100, "filter": null, "verbose": false, "debug": false, "default_db_id": "distribution_mister", "start_time": 1654702379.9457989, "user_defined_options": [], "curl_ssl": "--cacert /etc/ssl/certs/cacert.pem", "update_linux_environment": 0, "fail_on_file_error": false, "commit": "856b6d5" } cacert file at "/etc/ssl/certs/cacert.pem" seems to be fine. Loading db from url: https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip Loading db from url: https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip Downloading 2 files: /tmp/temp_filet7etzsji /tmp/temp_filex45tz6id *..

    Checking hashes... ++ subprocess.run unzip -p /tmp/temp_filet7etzsji Return Code was '12'

    error: invalid zip file with overlapped components (possible zip bomb)

    Could not load json from "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip" subprocess.run unzip -p /tmp/temp_filex45tz6id Return Code was '12'

    error: invalid zip file with overlapped components (possible zip bomb)

    Could not load json from "https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip" Skipping local_store saving...

    =========================== Downloader 1.5 (856b6d5) by theypsilon. Run time: 0:00:02.29s Log: /media/fat/Scripts/.config/downloader/downloader.log

    Installed: none.

    Errors: https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip, https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip

    linux_descriptions_count: 0 Length of failed_dbs: 2

    opened by LuoJohnT 5
  • Question: Save updates to /media folder without appending _Consoles/_Arcade/etc folders

    Question: Save updates to /media folder without appending _Consoles/_Arcade/etc folders

    Hi there,

    Is there any way to set the downloader to update and install the files I want straight into the /media folder instead of adding the associated _Console folder or any other similar folder? I don't use a ton of cores so I'd love to just run the script and keep everything clean. This way, I can just boot the MiSTer, and immediately see the cores in the list without entering additional directories.

    (Unless this is already possible)

    opened by Chromagram 3
  • downloader failed:

    downloader failed: "Could not load storage"

    Hello,

    I am attempting to run update_all.sh script with the downloader enabled. The latest jtbeta.zip file is in my 'mame' folder and all of the necessary options are enabled, as far as I can tell, in the update_all options menu (i.e. Install Premium Cores, Install Unofficial Cores etc.) I am recieving the following error:


    Reading file: /media/fat/download.ini Reading 'distribution_mister' db section Reading 'jtcores' db section Reading 'theypsilon_unofficial_distribution" db section Could not load storage Could not load json from "https://raw.githubusercontent.com/MiSTer--devel/Distribution_MiSTer/main/db.json.zip" Could not load json from "https://raw.githubusercontent.com/jotego/jtpremium/main/jtbindb.json.zip" Could not load json from "https://raw.githubusercontent.com/theypsilon/Distribution_Unofficial_MiSTer/unofficialdb.json.zip"

    There were some errors in the Updaters. Therefore, MiSTer hasn't been fully updated.

    Check these logs from the Updaters that failed:

    • /media/fat/Scripts/ .config/downloader/downloader1.log
    • BIOS-GETTER
    • /media/fat/Scripts/ .config/downloader/downloader2.log

    I receive the same error when running just the downloader.sh script. All official cores were downloaded and installed correctly using update_all.

    Thank you!

    opened by eebaser 3
  • Issue wit pc downloader

    Issue wit pc downloader

    Hi,

    I get the following error from https://github.com/MiSTer-devel/Downloader_MiSTer/releases/download/latest/downloader.zip

    {code} Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in run_code File "c:\TEMP\roka\2\tmp2bdrzqig.zip_main.py", line 49, in NameError: name 'exit' is not defined {code}

    sys.exit instead of exit is needed

    opened by RolandKaechele 2
  • Refactor downloader scripts

    Refactor downloader scripts

    While I was reading the scripts (to understand how they work) I've started putting some comments at some places. Later I've started introducing some "echo" statement as well. Also, I've noticed /etc/ssl/certs/cacert.pem is used in many places as a string literal, so I've created a constant to pass this around. In the end, I've tried to run this outside of MiSTer and I got hit by some missing binaries, so I added a function to check for those.

    Nothing significant here, mostly cosmetic changes I would say. Feel free to approve or reject this.

    opened by drpaneas 2
  • Error unpacking BIOS zips

    Error unpacking BIOS zips

    I'm on a fresh install and I am getting this error during the BiosDB step (in update_all):

    Extracting NeoGeo UniBios from http://unibios.free.fr
    checkdir:  cannot create extraction directory: /tmp/q48as6j6/neogeo_unibios
               No such file or directory
    Traceback (most recent call last):
      File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
        exit_code = execute_full_run(
      File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
        exit_code = runner.full_run()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
        result = self._full_run_impl()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
        self._online_importer.download_dbs_contents(importer_command, full_resync)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 95, in download_dbs_contents
        db_importer.process_changed_files(changed_files, needed_zips, filtered_zip_data)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 586, in process_changed_files
        self._import_zip_contents(needed_zips, filtered_zip_data, file_downloader)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 672, in _import_zip_contents
        self._file_system.unzip_contents(temp_zip, tmp_path, list(zipped_files['files']))
      File "/tmp/dont_download.zip/downloader/file_system.py", line 335, in unzip_contents
        raise Exception("Could not unzip %s: %s" % (file, result.returncode))
    Exception: Could not unzip /tmp/hir0qr7h_neogeo_unibios_contents.zip: 2
    

    I think the parent tmp directory isn't being created after the name is generated?

    https://github.com/MiSTer-devel/Downloader_MiSTer/blob/0060984366a335546a7152ff83065c729d962d83/src/downloader/online_importer.py#L671

    opened by wizzomafizzo 2
  • Failure to remove directory when using retronas and MiSTer

    Failure to remove directory when using retronas and MiSTer

    I opened an issue for this over in the retronas repo, but it might be better suited here.

    Copy/paste from the original issue:


    Hi!

    Just getting started with retronas, and connected my MiSTer setup to it over CIFS. When I run downloader.sh, it tries to remove the empty folder for ATARI2600, but ends up failing:

    Deleting empty folder /media/fat/games/ATARI2600
    Traceback (most recent call last):
      File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
        exit_code = execute_full_run(
      File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
        exit_code = runner.full_run()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
        result = self._full_run_impl()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
        self._online_importer.download_dbs_contents(importer_command, full_resync)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 108, in download_dbs_contents
        self._remove_folders(importer_command)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 210, in _remove_folders
        system_file_system.remove_folder(full_folder_path)
      File "/tmp/dont_download.zip/downloader/file_system.py", line 239, in remove_folder
        os.rmdir(self._path(path))
    NotADirectoryError: [Errno 20] Not a directory: '/media/fat/games/ATARI2600'
    

    This makes the rest of the update/download fail as well. Creating an empty file in the ATARI2600 folder removes this failure point.

    opened by jonasrosland 2
  • Won't read from /media/fat/downloader.ini

    Won't read from /media/fat/downloader.ini

    Readme says to use this ini to make changes, but it never reads from it. It's only looking at this one according to my log: Reading file: /media/fat/Update/MiSTer_Downloader/downloader.ini

    My ini: https://pastebin.com/DNPrxXEJ

    Log: https://pastebin.com/1HwS6gh8

    I set verbose to true and log says it's false Also tried moving the ini to "/media/fat/Update/MiSTer_Downloader/downloader.ini" and it still doesn't read it.

    opened by djvj1 2
  • Updates Not Downloading After Adding downloader.ini

    Updates Not Downloading After Adding downloader.ini

    So I created downloader.ini, copy/pasted the defaults and saved it into my root directory (where MiSTer.ini) lives. I noticed that when I run the downloader script now, nothing updates. After doing some research, I found that the downloader.ini needs to point to the default MiSTer [distribution_mister] database.

    If this is the case, then shouldn't this be documented and added to the list of default ini settings?

    opened by mjj03301977 2
  •  Invalid cross-device link on update

    Invalid cross-device link on update

    Copying new MiSTer binary:
    Traceback (most recent call last):
      File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
        exit_code = execute_full_run(
      File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
        exit_code = runner.full_run()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
        result = self._full_run_impl()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
        self._online_importer.download_dbs_contents(importer_command, full_resync)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 95, in download_dbs_contents
        db_importer.process_changed_files(changed_files, needed_zips, filtered_zip_data)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 588, in process_changed_files
        file_downloader.download_files(self._is_first_run())
      File "/tmp/dont_download.zip/downloader/file_downloader.py", line 121, in download_files
        self._file_system.move(FILE_MiSTer_new, FILE_MiSTer)
      File "/tmp/dont_download.zip/downloader/file_system.py", line 189, in move
        os.replace(self._path(source), self._path(target))
    OSError: [Errno 18] Invalid cross-device link: '/media/usb0/MiSTer.new' -> '/media/fat/MiSTer'
    
    Downloader failed!
    

    Here is my download.ini

    [MiSTer]
    base_path = '/media/usb0'
    
    [distribution_mister]
    db_url = https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip
    
    [jtcores]
    db_url = https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip
    
    [bios_db]
    db_url = https://raw.githubusercontent.com/theypsilon/BiosDB_MiSTer/db/bios_db.json
    
    opened by KClough 2
  • Delete old config/DIP switch files when needed

    Delete old config/DIP switch files when needed

    A common source of problems is when the core logic changes leaving the saved config/DIP files unusable and possibly breaking some functionality if used with old files.

    For DIP switches, it is easy to detect a change by comparing the dip switch section inside the MRA with the previous MRA file. If a change in the DIP sw. section is detected, the downloader should delete the old file in the config/dips folder. This would apply to any MRA of any arcade core.

    For config file changes, I don't think it's easy to detect automatically because the configuration information shown in the OSD is embedded in the RBF file and has gone through some transformations. So it isn't plain ASCII. Maybe the downloader could read some metadata in the repositories. I could, for instance, make the config string available as plain text in the version.log file of each core. Currently, this file points to the source code git commit used to produce a specific RBF in JTBIN.

    Please consider implementing these improvements.

    opened by jotego 5
Introducing neural networks to predict stock prices

IntroNeuralNetworks in Python: A Template Project IntroNeuralNetworks is a project that introduces neural networks and illustrates an example of how o

Vivek Palaniappan 637 Jan 04, 2023
Benchmarking the robustness of Spatial-Temporal Models

Benchmarking the robustness of Spatial-Temporal Models This repositery contains the code for the paper Benchmarking the Robustness of Spatial-Temporal

Yi Chenyu Ian 15 Dec 16, 2022
Replication Package for AequeVox:Automated Fariness Testing for Speech Recognition Systems

AequeVox Replication Package for AequeVox:Automated Fariness Testing for Speech Recognition Systems README under development. Python Packages Required

Sai Sathiesh 2 Aug 28, 2022
moving object detection for satellite videos.

DSFNet: Dynamic and Static Fusion Network for Moving Object Detection in Satellite Videos Algorithm Introduction DSFNet: Dynamic and Static Fusion Net

xiaochao 39 Dec 16, 2022
Code for the paper: Fighting Fake News: Image Splice Detection via Learned Self-Consistency

Fighting Fake News: Image Splice Detection via Learned Self-Consistency [paper] [website] Minyoung Huh *12, Andrew Liu *1, Andrew Owens1, Alexei A. Ef

minyoung huh (jacob) 174 Dec 09, 2022
The easiest way to use deep metric learning in your application. Modular, flexible, and extensible. Written in PyTorch.

News December 27: v1.1.0 New loss functions: CentroidTripletLoss and VICRegLoss Mean reciprocal rank + per-class accuracies See the release notes Than

Kevin Musgrave 5k Jan 05, 2023
This repository contains the PyTorch implementation of the paper STaCK: Sentence Ordering with Temporal Commonsense Knowledge appearing at EMNLP 2021.

STaCK: Sentence Ordering with Temporal Commonsense Knowledge This repository contains the pytorch implementation of the paper STaCK: Sentence Ordering

Deep Cognition and Language Research (DeCLaRe) Lab 23 Dec 16, 2022
Official Implementation for Fast Training of Neural Lumigraph Representations using Meta Learning.

Fast Training of Neural Lumigraph Representations using Meta Learning Project Page | Paper | Data Alexander W. Bergman, Petr Kellnhofer, Gordon Wetzst

Alex 39 Oct 08, 2022
Reproduce partial features of DeePMD-kit using PyTorch.

DeePMD-kit on PyTorch For better understand DeePMD-kit, we implement its partial features using PyTorch and expose interface consuing descriptors. Tec

Shaochen Shi 8 Dec 17, 2022
Implementation of trRosetta and trDesign for Pytorch, made into a convenient package

trRosetta - Pytorch (wip) Implementation of trRosetta and trDesign for Pytorch, made into a convenient package

Phil Wang 67 Dec 17, 2022
A data annotation pipeline to generate high-quality, large-scale speech datasets with machine pre-labeling and fully manual auditing.

About This repository provides data and code for the paper: Scalable Data Annotation Pipeline for High-Quality Large Speech Datasets Development (subm

Appen Repos 86 Dec 07, 2022
A Kernel fuzzer focusing on race bugs

Razzer: Finding kernel race bugs through fuzzing Environment setup $ source scripts/envsetup.sh scripts/envsetup.sh sets up necessary environment var

Systems and Software Security Lab at Seoul National University (SNU) 328 Dec 26, 2022
[SIGMETRICS 2022] One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search

One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search paper | website One Proxy Device Is Enough for Hardware-Aware Neural Architec

10 Dec 16, 2022
Pytorch implementation for A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose Paper | Website | Data A-NeRF: Articulated Neural Radiance F

Shih-Yang Su 172 Dec 22, 2022
level1-image-classification-level1-recsys-09 created by GitHub Classroom

level1-image-classification-level1-recsys-09 ❗ 주제 설명 COVID-19 Pandemic 상황 속 마스크 착용 유무 판단 시스템 구축 마스크 착용 여부, 성별, 나이 총 세가지 기준에 따라 총 18개의 class로 구분하는 모델 ?

6 Mar 17, 2022
Code for "Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations"

Infinitely Deep Bayesian Neural Networks with SDEs This library contains JAX and Pytorch implementations of neural ODEs and Bayesian layers for stocha

Winnie Xu 95 Nov 26, 2021
ComputerVision - This repository aims at realized easy network architecture

ComputerVision This repository aims at realized easy network architecture Colori

DongDong 4 Dec 14, 2022
Camera Distortion-aware 3D Human Pose Estimation in Video with Optimization-based Meta-Learning

Camera Distortion-aware 3D Human Pose Estimation in Video with Optimization-based Meta-Learning This is the official repository of "Camera Distortion-

Hanbyel Cho 12 Oct 06, 2022
Solution to the Weather4cast 2021 challenge

This code was used for the entry by the team "antfugue" for the Weather4cast 2021 Challenge. Below, you can find the instructions for generating predi

Jussi Leinonen 13 Jan 03, 2023
A module for solving and visualizing Schrödinger equation.

qmsolve This is an attempt at making a solid, easy to use solver, capable of solving and visualize the Schrödinger equation for multiple particles, an

506 Dec 28, 2022