Script for resizing MTD partitions on a QNAP device in order to be available to upgrade from buster to bullseye

Overview

QNAP partitions resize for kirkwood devices.

As explained by Marin Michlmayr, Debian bullseye support on kirkwood QNAP devices was dropped due to [mainly] the limited size of the Kernel partition (2MB).

Indeed, Bullseye current kernel image (vmlinuz-5.10.0-8-marvell) is 2445216 bytes long (2.3MB)

In addition, partition for initrd is also limited (9MB) which may lead to space issues.

Hopefully, some space is still unused for Debian in QNAP 16MB NOR flash.

  • An additional 3MB Rootfs2 partition is used by original QNAP firmware for its own purpose (install on empty HDD ?)
  • A "NAS config" partition is 1.2MB large despite containing few configuration files (<128KB). This partition can be resized to 256KB (Flash block size) without losing the information.

New Layout

With this script, we propose to use a new flash layout

Layout

  • We keep /dev/mtdX numbers mapping, in case some other users are using a fix numbering.
  • we keep a window on legacy kernel mapping to help if we want to restore the original QNAP firmware or to install the Buster installer
  • Rootfs1 is larger but use the same start offset (simplify the transition since no write in flash is required)
  • Kernel is larger. We must be careful during the transition since offsets are different.

With this new layout, we can transparently upgrade to Bullseye

  • More room for kernel and initrd
  • Future kernel updates performed during apt upgradewill use the new layout without any further change or manual operations.

Kernel and MTD partitions

Linux has 2 methods for configuring the partitions.

  1. Device Tree

    This is the standard way to describe the device.

    see: https://github.com/torvalds/linux/blob/master/arch/arm/boot/dts/kirkwood-ts219.dtsi

    ; label = "Kernel"; }; [email protected] { reg = <0x00400000 0x00900000>; label = "RootFS1"; }; [email protected] { reg = <0x00d00000 0x00300000>; label = "RootFS2"; }; [email protected] { reg = <0x00080000 0x00040000>; label = "U-Boot Config"; }; [email protected] { reg = <0x000c0000 0x00140000>; label = "NAS Config"; }; }; ">
    			[email protected] {
    				[...]
    				[email protected] {
    					reg = <0x00000000 0x00080000>;
    					label = "U-Boot";
    				};
    				[email protected] {
    					reg = <0x00200000 0x00200000>;
    					label = "Kernel";
    				};
    				[email protected] {
    					reg = <0x00400000 0x00900000>;
    					label = "RootFS1";
    				};
    				[email protected] {
    					reg = <0x00d00000 0x00300000>;
    					label = "RootFS2";
    				};
    				[email protected] {
    					reg = <0x00080000 0x00040000>;
    					label = "U-Boot Config";
    				};
    				[email protected] {
    					reg = <0x000c0000 0x00140000>;
    					label = "NAS Config";
    				};
    			};
    

    On Debian, binary versions (dtb) are provided by the linux-image-4.19.0-16-marvell package.

    Every time a new kernel version is flashed by flash-kernel, the proper dtb blob is concatenated to the kernel image so the kernel knows the details of the machine and is able to configure all the drivers properly (including the MTD partitions).

    It is not difficult to build a DTB and configure Debian to use this alternate DTB file. It simply need to be present in /etc/flash-kernel/dtbs/directory and flash-kernel will correctly switch to our own specific file.

    Example:

    cd [clone of linux sources]/linux
    # modify arch/arm/boot/dts/kirkwood-ts219.dtsi
    
    for D in kirkwood-ts219-6281 kirkwood-ts219-6282; do
        cpp -nostdinc -I include -I arch  -undef -x assembler-with-cpp \
           ./arch/arm/boot/dts/$D.dts \
           /tmp/preprocess.dts
        dtc -O dtb -o /tmp/$D.dtb /tmp/preprocess.dts
        cp /tmp/$D.dtb /etc/flash-kernel/dtbs/
        done
    
    
  2. kernel boot cmdline

    U-boot can also override the DTB information by using mtdparts=....options as parsed by https://github.com/torvalds/linux/blob/master/drivers/mtd/parsers/cmdlinepart.c

    The kernel try use use cmdline parameters before DTB information.

    Original QNAP U-boot configuration doesn't use this method.

I select the "kernel boot cmdline" solution to configure the new layout:

  • if /etc/flash-kernel/dtbs/content is modified or erased for some reasons (new install ?) the u-boot setup and the kernel MTD usage will not be synchronized and the device will fail to boot.
  • cmdline only "patch/override" the MTD partitions information. If the original DTB file provided by Debian is updated for some reasons (driver fixes), the kernel will still continue to use those fixes.

The qnap_mtd_resize.py script will:

  • Resize the "NAS config" filesystem
  • Prepare the content of current "NAS config" partition (offset 0xc0000 to 0x200000) with the shrink FS + first 1MB of the current kernel
  • prepare the image of current "Kernel" partition (starting at 0x200000) with tail of the current kernel
  • Patch U-boot env/config for the new bootargsand bootcmdvariables.
  • Write flash from 0xc0000 to 0x200000 ("NAS config" + 1MB head of kernel)
  • write flash from 0x200000 with 1MB tail of kernel.

Resize process

First, Do a backup of your MTD

cat /dev/mtd0 > /tmp/mtd0.uboot.backup
cat /dev/mtd1 > /tmp/mtd1.kernel.backup
cat /dev/mtd2 > /tmp/mtd2.rootfs1.backup
cat /dev/mtd3 > /tmp/mtd3.rootfs2.backup
cat /dev/mtd4 > /tmp/mtd4.uboot-config.backup
cat /dev/mtd5 > /tmp/mtd5.nas-config.backup
cd /tmp
tar cvzf mtd_backup.tgz mtd?.*.backup

And save this mtd_backup.tgzon your PC, transfering the file with scp / sftp or a USB drive....

Then, run qnap_mtd_resize.py

A first run with --dry-runoption to check that everything will be fine (except flashing)

sudo ./qnap_mtd_resize.py --dry-run

Example of dry-run log here.

If everything is fine run again without --dry-run

sudo ./qnap_mtd_resize.py

And reboot...

You are now running the same system, but with more room:

$ cat /proc/mtd 
dev:    size   erasesize  name
mtd0: 00080000 00040000 "uboot"
mtd1: 00300000 00040000 "Kernel"
mtd2: 00c00000 00040000 "RootFS1"
mtd3: 00200000 00040000 "Kernel_legacy"
mtd4: 00040000 00040000 "U-Boot Config"
mtd5: 00040000 00040000 "NAS Config"

Which makes possible to install Bullseye's kernel:

$ flash-kernel 
kirkwood-qnap: machine: QNAP TS219 family
Using DTB: kirkwood-ts219-6281.dtb
Installing /usr/lib/linux-image-5.10.0-8-marvell/kirkwood-ts219-6281.dtb into /boot/dtbs/5.10.0-8-marvell/./kirkwood-ts219-6281.dtb
Taking backup of kirkwood-ts219-6281.dtb.
Installing new kirkwood-ts219-6281.dtb.
flash-kernel: installing version 5.10.0-8-marvell
flash-kernel: appending /usr/lib/linux-image-5.10.0-8-marvell/kirkwood-ts219-6281.dtb to kernel
Generating kernel u-boot image... done.
Flashing kernel (using 2455558/3145728 bytes)... done.
Flashing initramfs (using 3992060/12582912 bytes)... done.

Additional configuration to improveinitrd size

Even if we increase Rootfs1 from 9 to 12 MB, you can still decrease the initrd size by compressing with xz

/etc/initramfs-tools/conf.d/compress ">
echo "COMPRESS=xz" > /etc/initramfs-tools/conf.d/compress

List of tested devices:

Model cat /sys/firmware/devicetree/base/model DTB file uboot env
(legacy)
uboot_env
(new)
Resize log
TS-219P QNAP TS219 family kirkwood-ts219-6281.dtb
TS-419PII QNAP TS419 family kirkwood-ts419-6282.dtb QNAP_TS419_family,uboot-env.legacy QNAP_TS419_family,uboot-env.new log
Owner
Arnaud Mouiche
Arnaud Mouiche
FantasyBballHelper - Espn Fantasy Basketball Helper

ESPN FANTASY BASKETBALL HELPER The simple goal of this project is to allow fanta

1 Jan 18, 2022
Find virtual hosts (vhosts) from IP addresses and hostnames

Features Enumerate vhosts from a list of IP addresses and domain names. Virtual Hosts are enumerated using the following process: Supplied domains are

3 Jul 09, 2022
Providing a working, flexible, easier and faster installer than the one officially provided by Arch Linux

Purpose The purpose is to bring more people to Arch Linux by providing a working, flexible, easier and faster installer than the one officially provid

André Luís 0 Nov 09, 2022
Find out where all films you want to watch are streaming

Just Watch Letterboxd Find out where all films you want to watch are streaming Ever wonder what films you want to watch are already on the streaming p

Jordan Oslislo 2 Feb 04, 2022
[arXiv 2020] Video Representation Learning with Visual Tempo Consistency

Video Representation Learning with Visual Tempo Consistency [Paper] [Project Page] News Full codebae is coming soon Pretained Models For now, we provi

DeciForce: Crossroads of Machine Perception and Autonomy 24 Nov 23, 2022
Backup dc registry - A simple POC that abuses Backup Operator privileges to remote dump SAM, SYSTEM, and SECURITY

Backup Operator Registry Backup to Domain Compromise A simple POC that abuses Ba

Horizon 3 AI Inc 57 Dec 18, 2022
Python template for Advent of Code event

Advent of Code Python Starter A tamplate for Advent of Code write in Python. Usage The project use poetry for project manager. Clone this repository a

Leonardo Gago 6 Dec 31, 2022
Python client library for the Databento API

Databento Python Library The Databento Python client library provides access to the Databento API for both live and historical data, from applications

Databento, Inc. 35 Dec 24, 2022
A simple python script that print the Mandelbrot set for every power of the formal formula.

Python Mandelbrot A simple python script that print the Mandelbrot set for every power of the formal formula.

Paride Giunta 2 Apr 15, 2022
Make your functions return something meaningful, typed, and safe!

Make your functions return something meaningful, typed, and safe! Features Brings functional programming to Python land Provides a bunch of primitives

dry-python 2.5k Jan 03, 2023
Datasets with Softcatalà website content

softcatala-web-dataset This repository contains Sofcatalà web site content (articles and programs descriptions). Dataset are available in the dataset

Softcatalà 2 Dec 26, 2021
Archive, organize, and watch for changes to publicly available information.

0. Overview The Trapper Keeper is a collection of scripts that support archiving information from around the web to make it easier to study and use. I

Bill Fitzgerald 9 Oct 26, 2022
Advanced IPv4 Subnet Calculator in Python3

Advanced IPv4 Subnet Calculator in Python3 Table of Contents Getting Started Installation How it works? SVI Configuration Template Previews Getting St

Osama Abbas 1 May 10, 2022
Small C-like language compiler for the Uxn assembly language

Pyuxncle is a single-pass compiler for a small subset of C (albeit without the std library). This compiler targets Uxntal, the assembly language of the Uxn virtual computer. The output Uxntal is not

CPunch 13 Jun 28, 2022
Blender addon to import images as meshes

ImagesAsMesh Blender addon to import images as meshes. Inspired by: ImagesAsPlanes Installation It's like just about every other Blender addon. Downlo

Niccolo Zuppichini 4 Jan 04, 2022
Snack Rice - A Rice University servery finder, customized for your needs!

Snack Rice - A Rice University servery finder, customized for your needs!

Aidan Gerber 3 Sep 25, 2022
A parser of Windows Defender's DetectionHistory forensic artifact, containing substantial info about quarantined files and executables.

A parser of Windows Defender's DetectionHistory forensic artifact, containing substantial info about quarantined files and executables.

Jordan Klepser 101 Oct 30, 2022
Python script to autodetect a base set of swiftlint rules.

swiftlint-autodetect Python script to autodetect a base set of swiftlint rules. Installation brew install pipx

Jonathan Wight 24 Sep 20, 2022
An easy python calculator for those who want's to know how if statements, loops, and imports works give it a try!

A usefull calculator for any student or anyone who want's to know how to build a simple 2 mode python based calculator.

Antonio Sánchez 1 Jan 06, 2022
Binjago - Set of tools aiding in analysis of stripped Golang binaries with Binary Ninja

Binjago 🥷 Set of tools aiding in analysis of stripped Golang binaries with Bina

W3ndige 2 Jul 23, 2022