Tool for installing and updating MiSTer cores and other files

Overview

MiSTer Downloader

This tool installs and updates all the cores and other extra files for your MiSTer. It also updates the menu core, the MiSTer firmware and the Linux system. The source for all downloads is the MiSTer Distribution repository.

The MiSTer Downloader is a substitute for the MiSTer Updater, and is meant to offer a more safe and robust experience, while being much faster.

As a drawback, the Downloader is not backwards compatible with the old INI files that were configured for the MiSTer Updater. In fact, as of today, this tool doesn't implement many fine-grained features that allow you to customize the updating process in depth. In case you value these features, consider to keep using the MiSTer Updater as usual. Both tools will coexist in the near future.

Setup and Usage

Download this ZIP file and extract downloader.sh to your /Scripts folder on your primary SD card (create that folder if it doesn't exist). You only need to perform this operation once, since this tool self-updates itself.

To use it, on your MiSTer main menu, go to the Scripts screen, and select downloader.

Options

You may create a /media/fat/downloader.ini file to tweak some parameters.

Here you can see the default parameters and the options that you may change:

[MiSTer]
; base_path is where most files will be installed
;   Useful for setups with USB storage, for example: '/media/usb0/'
base_path = '/media/fat/'

; base_system_path is where system files such as 'MiSTer' and 'menu.rbf' will be installed.
;   Warning: It is recommended to NOT change this setting regardless of your setup.
base_system_path = '/media/fat/'

; allow_delete options:
;   0 -> Don't allow this tool to delete anything at all.
;   1 -> Allow this tool to delete any old file from previous updates.
;   2 -> Allow this tool to delete only old cores that receive a new version.
allow_delete = 1

; allow_reboot options:
;   0 -> Don't allow this tool to ever reboot automatically.
;   1 -> Allow this tool to reboot the system after any system file has been updated.
;   2 -> Allow this tool to reboot the system only after Linux has been updated.
allow_reboot = 1

; update_linux options:
;   true -> Updates Linux when there is a new update (very recommended).
;   false -> Doesn't update Linux.
update_linux = true

; parallel_update options:
;   true -> Tries to more than one file simultaneously.
;   false -> Will only download one file at a time.
parallel_update = true

; downloader_timeout: Can be tweaked to increase the timeout time in seconds
;   It is useful to increase this value for users with slow connections.
downloader_timeout = 300

; downloader_retries: Can be tweaked to increase the retries per failed download
;   It is useful to increase this value for users with very unstable connections.
downloader_retries = 3

Roadmap

  • Initial Release
  • Cheats fetching
  • First-run optimisations
  • Configurable custom download filters
  • Handle duplicated games folders through symlinks (GBA <-> GBA2P, and GAMEBOY <-> GAMEBOY2P)
  • Integration with MiSTer binary
Comments
  • mra_alternatives don't seem to get updated

    mra_alternatives don't seem to get updated

    If the hash for the mra_alternatives summary file doesn't match the newest, mra_alternatives_summary.json.zip gets downloaded and the hashes are updated in the store (summary and files) but the files themselves aren't updated.

    opened by lagomorph 12
  • Unable to update

    Unable to update

    When I try to run aupdate from the MiSTer menu the part where it updates the CA-certs work. But after that it fails.

    The script will run normally now:
    Running MiSTer Downloader
    
    bash: line 122: BASH_SOURCE[0]: unbound variable
    Press any key to continue
    
    opened by Dankoman 6
  • error: invalid zip file with overlapped components (possible zip bomb)

    error: invalid zip file with overlapped components (possible zip bomb)

    I started receiving errors when trying to update today. Unzip is throwing an exception because it is recognizing the downloaded file as malicious. I am running the latest version of download.sh. The same issue occurs with any update script which downloads zipped files. Previously had no issues updating.

    /root# cat /media/fat/Scripts/.config/downloader/downloader.log START!

    Reading file: /media/fat/downloader.ini Reading 'distribution_mister' db section Reading 'jtcores' db section env: { "DOWNLOADER_LAUNCHER_PATH": "/media/fat/Scripts/downloader.sh", "DOWNLOADER_INI_PATH": null, "CURL_SSL": "--cacert /etc/ssl/certs/cacert.pem", "COMMIT": "856b6d5", "ALLOW_REBOOT": null, "UPDATE_LINUX": "true", "DEFAULT_DB_URL": "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip", "DEFAULT_DB_ID": "distribution_mister", "DEFAULT_BASE_PATH": null, "DEBUG": "false", "FAIL_ON_FILE_ERROR": "false" } config: { "databases": { "distribution_mister": { "db_url": "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip", "section": "distribution_mister" }, "jtcores": { "db_url": "https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip", "section": "jtcores" } }, "config_path": "/media/fat/downloader.ini", "base_path": "/media/fat", "base_system_path": "/media/fat", "storage_priority": "prefer_sd", "allow_delete": 1, "allow_reboot": 1, "update_linux": true, "parallel_update": true, "downloader_size_mb_limit": 100, "downloader_process_limit": 300, "downloader_timeout": 300, "downloader_retries": 3, "zip_file_count_threshold": 60, "zip_accumulated_mb_threshold": 100, "filter": null, "verbose": false, "debug": false, "default_db_id": "distribution_mister", "start_time": 1654702379.9457989, "user_defined_options": [], "curl_ssl": "--cacert /etc/ssl/certs/cacert.pem", "update_linux_environment": 0, "fail_on_file_error": false, "commit": "856b6d5" } cacert file at "/etc/ssl/certs/cacert.pem" seems to be fine. Loading db from url: https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip Loading db from url: https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip Downloading 2 files: /tmp/temp_filet7etzsji /tmp/temp_filex45tz6id *..

    Checking hashes... ++ subprocess.run unzip -p /tmp/temp_filet7etzsji Return Code was '12'

    error: invalid zip file with overlapped components (possible zip bomb)

    Could not load json from "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip" subprocess.run unzip -p /tmp/temp_filex45tz6id Return Code was '12'

    error: invalid zip file with overlapped components (possible zip bomb)

    Could not load json from "https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip" Skipping local_store saving...

    =========================== Downloader 1.5 (856b6d5) by theypsilon. Run time: 0:00:02.29s Log: /media/fat/Scripts/.config/downloader/downloader.log

    Installed: none.

    Errors: https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip, https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip

    linux_descriptions_count: 0 Length of failed_dbs: 2

    opened by LuoJohnT 5
  • Question: Save updates to /media folder without appending _Consoles/_Arcade/etc folders

    Question: Save updates to /media folder without appending _Consoles/_Arcade/etc folders

    Hi there,

    Is there any way to set the downloader to update and install the files I want straight into the /media folder instead of adding the associated _Console folder or any other similar folder? I don't use a ton of cores so I'd love to just run the script and keep everything clean. This way, I can just boot the MiSTer, and immediately see the cores in the list without entering additional directories.

    (Unless this is already possible)

    opened by Chromagram 3
  • downloader failed:

    downloader failed: "Could not load storage"

    Hello,

    I am attempting to run update_all.sh script with the downloader enabled. The latest jtbeta.zip file is in my 'mame' folder and all of the necessary options are enabled, as far as I can tell, in the update_all options menu (i.e. Install Premium Cores, Install Unofficial Cores etc.) I am recieving the following error:


    Reading file: /media/fat/download.ini Reading 'distribution_mister' db section Reading 'jtcores' db section Reading 'theypsilon_unofficial_distribution" db section Could not load storage Could not load json from "https://raw.githubusercontent.com/MiSTer--devel/Distribution_MiSTer/main/db.json.zip" Could not load json from "https://raw.githubusercontent.com/jotego/jtpremium/main/jtbindb.json.zip" Could not load json from "https://raw.githubusercontent.com/theypsilon/Distribution_Unofficial_MiSTer/unofficialdb.json.zip"

    There were some errors in the Updaters. Therefore, MiSTer hasn't been fully updated.

    Check these logs from the Updaters that failed:

    • /media/fat/Scripts/ .config/downloader/downloader1.log
    • BIOS-GETTER
    • /media/fat/Scripts/ .config/downloader/downloader2.log

    I receive the same error when running just the downloader.sh script. All official cores were downloaded and installed correctly using update_all.

    Thank you!

    opened by eebaser 3
  • Issue wit pc downloader

    Issue wit pc downloader

    Hi,

    I get the following error from https://github.com/MiSTer-devel/Downloader_MiSTer/releases/download/latest/downloader.zip

    {code} Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in run_code File "c:\TEMP\roka\2\tmp2bdrzqig.zip_main.py", line 49, in NameError: name 'exit' is not defined {code}

    sys.exit instead of exit is needed

    opened by RolandKaechele 2
  • Refactor downloader scripts

    Refactor downloader scripts

    While I was reading the scripts (to understand how they work) I've started putting some comments at some places. Later I've started introducing some "echo" statement as well. Also, I've noticed /etc/ssl/certs/cacert.pem is used in many places as a string literal, so I've created a constant to pass this around. In the end, I've tried to run this outside of MiSTer and I got hit by some missing binaries, so I added a function to check for those.

    Nothing significant here, mostly cosmetic changes I would say. Feel free to approve or reject this.

    opened by drpaneas 2
  • Error unpacking BIOS zips

    Error unpacking BIOS zips

    I'm on a fresh install and I am getting this error during the BiosDB step (in update_all):

    Extracting NeoGeo UniBios from http://unibios.free.fr
    checkdir:  cannot create extraction directory: /tmp/q48as6j6/neogeo_unibios
               No such file or directory
    Traceback (most recent call last):
      File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
        exit_code = execute_full_run(
      File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
        exit_code = runner.full_run()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
        result = self._full_run_impl()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
        self._online_importer.download_dbs_contents(importer_command, full_resync)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 95, in download_dbs_contents
        db_importer.process_changed_files(changed_files, needed_zips, filtered_zip_data)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 586, in process_changed_files
        self._import_zip_contents(needed_zips, filtered_zip_data, file_downloader)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 672, in _import_zip_contents
        self._file_system.unzip_contents(temp_zip, tmp_path, list(zipped_files['files']))
      File "/tmp/dont_download.zip/downloader/file_system.py", line 335, in unzip_contents
        raise Exception("Could not unzip %s: %s" % (file, result.returncode))
    Exception: Could not unzip /tmp/hir0qr7h_neogeo_unibios_contents.zip: 2
    

    I think the parent tmp directory isn't being created after the name is generated?

    https://github.com/MiSTer-devel/Downloader_MiSTer/blob/0060984366a335546a7152ff83065c729d962d83/src/downloader/online_importer.py#L671

    opened by wizzomafizzo 2
  • Failure to remove directory when using retronas and MiSTer

    Failure to remove directory when using retronas and MiSTer

    I opened an issue for this over in the retronas repo, but it might be better suited here.

    Copy/paste from the original issue:


    Hi!

    Just getting started with retronas, and connected my MiSTer setup to it over CIFS. When I run downloader.sh, it tries to remove the empty folder for ATARI2600, but ends up failing:

    Deleting empty folder /media/fat/games/ATARI2600
    Traceback (most recent call last):
      File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
        exit_code = execute_full_run(
      File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
        exit_code = runner.full_run()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
        result = self._full_run_impl()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
        self._online_importer.download_dbs_contents(importer_command, full_resync)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 108, in download_dbs_contents
        self._remove_folders(importer_command)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 210, in _remove_folders
        system_file_system.remove_folder(full_folder_path)
      File "/tmp/dont_download.zip/downloader/file_system.py", line 239, in remove_folder
        os.rmdir(self._path(path))
    NotADirectoryError: [Errno 20] Not a directory: '/media/fat/games/ATARI2600'
    

    This makes the rest of the update/download fail as well. Creating an empty file in the ATARI2600 folder removes this failure point.

    opened by jonasrosland 2
  • Won't read from /media/fat/downloader.ini

    Won't read from /media/fat/downloader.ini

    Readme says to use this ini to make changes, but it never reads from it. It's only looking at this one according to my log: Reading file: /media/fat/Update/MiSTer_Downloader/downloader.ini

    My ini: https://pastebin.com/DNPrxXEJ

    Log: https://pastebin.com/1HwS6gh8

    I set verbose to true and log says it's false Also tried moving the ini to "/media/fat/Update/MiSTer_Downloader/downloader.ini" and it still doesn't read it.

    opened by djvj1 2
  • Updates Not Downloading After Adding downloader.ini

    Updates Not Downloading After Adding downloader.ini

    So I created downloader.ini, copy/pasted the defaults and saved it into my root directory (where MiSTer.ini) lives. I noticed that when I run the downloader script now, nothing updates. After doing some research, I found that the downloader.ini needs to point to the default MiSTer [distribution_mister] database.

    If this is the case, then shouldn't this be documented and added to the list of default ini settings?

    opened by mjj03301977 2
  •  Invalid cross-device link on update

    Invalid cross-device link on update

    Copying new MiSTer binary:
    Traceback (most recent call last):
      File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
        exit_code = execute_full_run(
      File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
        exit_code = runner.full_run()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
        result = self._full_run_impl()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
        self._online_importer.download_dbs_contents(importer_command, full_resync)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 95, in download_dbs_contents
        db_importer.process_changed_files(changed_files, needed_zips, filtered_zip_data)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 588, in process_changed_files
        file_downloader.download_files(self._is_first_run())
      File "/tmp/dont_download.zip/downloader/file_downloader.py", line 121, in download_files
        self._file_system.move(FILE_MiSTer_new, FILE_MiSTer)
      File "/tmp/dont_download.zip/downloader/file_system.py", line 189, in move
        os.replace(self._path(source), self._path(target))
    OSError: [Errno 18] Invalid cross-device link: '/media/usb0/MiSTer.new' -> '/media/fat/MiSTer'
    
    Downloader failed!
    

    Here is my download.ini

    [MiSTer]
    base_path = '/media/usb0'
    
    [distribution_mister]
    db_url = https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip
    
    [jtcores]
    db_url = https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip
    
    [bios_db]
    db_url = https://raw.githubusercontent.com/theypsilon/BiosDB_MiSTer/db/bios_db.json
    
    opened by KClough 2
  • Delete old config/DIP switch files when needed

    Delete old config/DIP switch files when needed

    A common source of problems is when the core logic changes leaving the saved config/DIP files unusable and possibly breaking some functionality if used with old files.

    For DIP switches, it is easy to detect a change by comparing the dip switch section inside the MRA with the previous MRA file. If a change in the DIP sw. section is detected, the downloader should delete the old file in the config/dips folder. This would apply to any MRA of any arcade core.

    For config file changes, I don't think it's easy to detect automatically because the configuration information shown in the OSD is embedded in the RBF file and has gone through some transformations. So it isn't plain ASCII. Maybe the downloader could read some metadata in the repositories. I could, for instance, make the config string available as plain text in the version.log file of each core. Currently, this file points to the source code git commit used to produce a specific RBF in JTBIN.

    Please consider implementing these improvements.

    opened by jotego 5
Peek-a-Boo: What (More) is Disguised in a Randomly Weighted Neural Network, and How to Find It Efficiently

Peek-a-Boo: What (More) is Disguised in a Randomly Weighted Neural Network, and How to Find It Efficiently This repository is the official implementat

VITA 4 Dec 20, 2022
Official implementation of ACTION-Net: Multipath Excitation for Action Recognition (CVPR'21).

ACTION-Net Official implementation of ACTION-Net: Multipath Excitation for Action Recognition (CVPR'21). Getting Started EgoGesture data folder struct

V-Sense 171 Dec 26, 2022
Planning from Pixels in Environments with Combinatorially Hard Search Spaces -- NeurIPS 2021

PPGS: Planning from Pixels in Environments with Combinatorially Hard Search Spaces Environment Setup We recommend pipenv for creating and managing vir

Autonomous Learning Group 11 Jun 26, 2022
This repository contains a PyTorch implementation of the paper Learning to Assimilate in Chaotic Dynamical Systems.

Amortized Assimilation This repository contains a PyTorch implementation of the paper Learning to Assimilate in Chaotic Dynamical Systems. Abstract: T

4 Aug 16, 2022
An official implementation of the Anchor DETR.

Anchor DETR: Query Design for Transformer-Based Detector Introduction This repository is an official implementation of the Anchor DETR. We encode the

MEGVII Research 276 Dec 28, 2022
Offical implementation of Shunted Self-Attention via Multi-Scale Token Aggregation

Shunted Transformer This is the offical implementation of Shunted Self-Attention via Multi-Scale Token Aggregation by Sucheng Ren, Daquan Zhou, Shengf

156 Dec 27, 2022
TensorFlow implementation for Bayesian Modeling and Uncertainty Quantification for Learning to Optimize: What, Why, and How

Bayesian Modeling and Uncertainty Quantification for Learning to Optimize: What, Why, and How TensorFlow implementation for Bayesian Modeling and Unce

Shen Lab at Texas A&M University 8 Sep 02, 2022
A privacy-focused, intelligent security camera system.

Self-Hosted Home Security Camera System A privacy-focused, intelligent security camera system. Features: Multi-camera support w/ minimal configuration

Scott Barnes 175 Jan 01, 2023
scAR (single-cell Ambient Remover) is a package for data denoising in single-cell omics.

scAR scAR (single cell Ambient Remover) is a package for denoising multiple single cell omics data. It can be used for multiple tasks, such as, sgRNA

19 Nov 28, 2022
[3DV 2020] PeeledHuman: Robust Shape Representation for Textured 3D Human Body Reconstruction

PeeledHuman: Robust Shape Representation for Textured 3D Human Body Reconstruction International Conference on 3D Vision, 2020 Sai Sagar Jinka1, Rohan

Rohan Chacko 39 Oct 12, 2022
This is the official implementation of the paper "Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation".

[CVPRW 2021] - Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation

Anirudh S Chakravarthy 6 May 03, 2022
Pytorch implementation of "Geometrically Adaptive Dictionary Attack on Face Recognition" (WACV 2022)

Geometrically Adaptive Dictionary Attack on Face Recognition This is the Pytorch code of our paper "Geometrically Adaptive Dictionary Attack on Face R

6 Nov 21, 2022
ANEA: Distant Supervision for Low-Resource Named Entity Recognition

ANEA: Distant Supervision for Low-Resource Named Entity Recognition ANEA is a tool to automatically annotate named entities in unlabeled text based on

Saarland University Spoken Language Systems Group 15 Mar 30, 2022
The source code of CVPR17 'Generative Face Completion'.

GenerativeFaceCompletion Matcaffe implementation of our CVPR17 paper on face completion. In each panel from left to right: original face, masked input

Yijun Li 313 Oct 18, 2022
Decentralized Reinforcment Learning: Global Decision-Making via Local Economic Transactions (ICML 2020)

Decentralized Reinforcement Learning This is the code complementing the paper Decentralized Reinforcment Learning: Global Decision-Making via Local Ec

40 Oct 30, 2022
CoRe: Contrastive Recurrent State-Space Models

CoRe: Contrastive Recurrent State-Space Models This code implements the CoRe model and reproduces experimental results found in Robust Robotic Control

Apple 21 Aug 11, 2022
MiniSom is a minimalistic implementation of the Self Organizing Maps

MiniSom Self Organizing Maps MiniSom is a minimalistic and Numpy based implementation of the Self Organizing Maps (SOM). SOM is a type of Artificial N

Giuseppe Vettigli 1.2k Jan 03, 2023
PyTorch Implementation of ECCV 2020 Spotlight TuiGAN: Learning Versatile Image-to-Image Translation with Two Unpaired Images

TuiGAN-PyTorch Official PyTorch Implementation of "TuiGAN: Learning Versatile Image-to-Image Translation with Two Unpaired Images" (ECCV 2020 Spotligh

181 Dec 09, 2022
Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search

Breaking the Curse of Space Explosion: Towards Effcient NAS with Curriculum Search Pytorch implementation for "Breaking the Curse of Space Explosion:

guoyong 17 Jan 03, 2023
Implementation of Segformer, Attention + MLP neural network for segmentation, in Pytorch

Segformer - Pytorch Implementation of Segformer, Attention + MLP neural network for segmentation, in Pytorch. Install $ pip install segformer-pytorch

Phil Wang 208 Dec 25, 2022