Script for resizing MTD partitions on a QNAP device in order to be available to upgrade from buster to bullseye

Overview

QNAP partitions resize for kirkwood devices.

As explained by Marin Michlmayr, Debian bullseye support on kirkwood QNAP devices was dropped due to [mainly] the limited size of the Kernel partition (2MB).

Indeed, Bullseye current kernel image (vmlinuz-5.10.0-8-marvell) is 2445216 bytes long (2.3MB)

In addition, partition for initrd is also limited (9MB) which may lead to space issues.

Hopefully, some space is still unused for Debian in QNAP 16MB NOR flash.

  • An additional 3MB Rootfs2 partition is used by original QNAP firmware for its own purpose (install on empty HDD ?)
  • A "NAS config" partition is 1.2MB large despite containing few configuration files (<128KB). This partition can be resized to 256KB (Flash block size) without losing the information.

New Layout

With this script, we propose to use a new flash layout

Layout

  • We keep /dev/mtdX numbers mapping, in case some other users are using a fix numbering.
  • we keep a window on legacy kernel mapping to help if we want to restore the original QNAP firmware or to install the Buster installer
  • Rootfs1 is larger but use the same start offset (simplify the transition since no write in flash is required)
  • Kernel is larger. We must be careful during the transition since offsets are different.

With this new layout, we can transparently upgrade to Bullseye

  • More room for kernel and initrd
  • Future kernel updates performed during apt upgradewill use the new layout without any further change or manual operations.

Kernel and MTD partitions

Linux has 2 methods for configuring the partitions.

  1. Device Tree

    This is the standard way to describe the device.

    see: https://github.com/torvalds/linux/blob/master/arch/arm/boot/dts/kirkwood-ts219.dtsi

    ; label = "Kernel"; }; [email protected] { reg = <0x00400000 0x00900000>; label = "RootFS1"; }; [email protected] { reg = <0x00d00000 0x00300000>; label = "RootFS2"; }; [email protected] { reg = <0x00080000 0x00040000>; label = "U-Boot Config"; }; [email protected] { reg = <0x000c0000 0x00140000>; label = "NAS Config"; }; }; ">
    			[email protected] {
    				[...]
    				[email protected] {
    					reg = <0x00000000 0x00080000>;
    					label = "U-Boot";
    				};
    				[email protected] {
    					reg = <0x00200000 0x00200000>;
    					label = "Kernel";
    				};
    				[email protected] {
    					reg = <0x00400000 0x00900000>;
    					label = "RootFS1";
    				};
    				[email protected] {
    					reg = <0x00d00000 0x00300000>;
    					label = "RootFS2";
    				};
    				[email protected] {
    					reg = <0x00080000 0x00040000>;
    					label = "U-Boot Config";
    				};
    				[email protected] {
    					reg = <0x000c0000 0x00140000>;
    					label = "NAS Config";
    				};
    			};
    

    On Debian, binary versions (dtb) are provided by the linux-image-4.19.0-16-marvell package.

    Every time a new kernel version is flashed by flash-kernel, the proper dtb blob is concatenated to the kernel image so the kernel knows the details of the machine and is able to configure all the drivers properly (including the MTD partitions).

    It is not difficult to build a DTB and configure Debian to use this alternate DTB file. It simply need to be present in /etc/flash-kernel/dtbs/directory and flash-kernel will correctly switch to our own specific file.

    Example:

    cd [clone of linux sources]/linux
    # modify arch/arm/boot/dts/kirkwood-ts219.dtsi
    
    for D in kirkwood-ts219-6281 kirkwood-ts219-6282; do
        cpp -nostdinc -I include -I arch  -undef -x assembler-with-cpp \
           ./arch/arm/boot/dts/$D.dts \
           /tmp/preprocess.dts
        dtc -O dtb -o /tmp/$D.dtb /tmp/preprocess.dts
        cp /tmp/$D.dtb /etc/flash-kernel/dtbs/
        done
    
    
  2. kernel boot cmdline

    U-boot can also override the DTB information by using mtdparts=....options as parsed by https://github.com/torvalds/linux/blob/master/drivers/mtd/parsers/cmdlinepart.c

    The kernel try use use cmdline parameters before DTB information.

    Original QNAP U-boot configuration doesn't use this method.

I select the "kernel boot cmdline" solution to configure the new layout:

  • if /etc/flash-kernel/dtbs/content is modified or erased for some reasons (new install ?) the u-boot setup and the kernel MTD usage will not be synchronized and the device will fail to boot.
  • cmdline only "patch/override" the MTD partitions information. If the original DTB file provided by Debian is updated for some reasons (driver fixes), the kernel will still continue to use those fixes.

The qnap_mtd_resize.py script will:

  • Resize the "NAS config" filesystem
  • Prepare the content of current "NAS config" partition (offset 0xc0000 to 0x200000) with the shrink FS + first 1MB of the current kernel
  • prepare the image of current "Kernel" partition (starting at 0x200000) with tail of the current kernel
  • Patch U-boot env/config for the new bootargsand bootcmdvariables.
  • Write flash from 0xc0000 to 0x200000 ("NAS config" + 1MB head of kernel)
  • write flash from 0x200000 with 1MB tail of kernel.

Resize process

First, Do a backup of your MTD

cat /dev/mtd0 > /tmp/mtd0.uboot.backup
cat /dev/mtd1 > /tmp/mtd1.kernel.backup
cat /dev/mtd2 > /tmp/mtd2.rootfs1.backup
cat /dev/mtd3 > /tmp/mtd3.rootfs2.backup
cat /dev/mtd4 > /tmp/mtd4.uboot-config.backup
cat /dev/mtd5 > /tmp/mtd5.nas-config.backup
cd /tmp
tar cvzf mtd_backup.tgz mtd?.*.backup

And save this mtd_backup.tgzon your PC, transfering the file with scp / sftp or a USB drive....

Then, run qnap_mtd_resize.py

A first run with --dry-runoption to check that everything will be fine (except flashing)

sudo ./qnap_mtd_resize.py --dry-run

Example of dry-run log here.

If everything is fine run again without --dry-run

sudo ./qnap_mtd_resize.py

And reboot...

You are now running the same system, but with more room:

$ cat /proc/mtd 
dev:    size   erasesize  name
mtd0: 00080000 00040000 "uboot"
mtd1: 00300000 00040000 "Kernel"
mtd2: 00c00000 00040000 "RootFS1"
mtd3: 00200000 00040000 "Kernel_legacy"
mtd4: 00040000 00040000 "U-Boot Config"
mtd5: 00040000 00040000 "NAS Config"

Which makes possible to install Bullseye's kernel:

$ flash-kernel 
kirkwood-qnap: machine: QNAP TS219 family
Using DTB: kirkwood-ts219-6281.dtb
Installing /usr/lib/linux-image-5.10.0-8-marvell/kirkwood-ts219-6281.dtb into /boot/dtbs/5.10.0-8-marvell/./kirkwood-ts219-6281.dtb
Taking backup of kirkwood-ts219-6281.dtb.
Installing new kirkwood-ts219-6281.dtb.
flash-kernel: installing version 5.10.0-8-marvell
flash-kernel: appending /usr/lib/linux-image-5.10.0-8-marvell/kirkwood-ts219-6281.dtb to kernel
Generating kernel u-boot image... done.
Flashing kernel (using 2455558/3145728 bytes)... done.
Flashing initramfs (using 3992060/12582912 bytes)... done.

Additional configuration to improveinitrd size

Even if we increase Rootfs1 from 9 to 12 MB, you can still decrease the initrd size by compressing with xz

/etc/initramfs-tools/conf.d/compress ">
echo "COMPRESS=xz" > /etc/initramfs-tools/conf.d/compress

List of tested devices:

Model cat /sys/firmware/devicetree/base/model DTB file uboot env
(legacy)
uboot_env
(new)
Resize log
TS-219P QNAP TS219 family kirkwood-ts219-6281.dtb
TS-419PII QNAP TS419 family kirkwood-ts419-6282.dtb QNAP_TS419_family,uboot-env.legacy QNAP_TS419_family,uboot-env.new log
Owner
Arnaud Mouiche
Arnaud Mouiche
Advanced Keylogger in Python

Advanced Keylogger in Python Important Disclaimer: The author will not be held r

Suvanth Erranki 1 Feb 07, 2022
A python mathematics module

A python mathematics module

Fayas Noushad 4 Nov 28, 2021
An account generator for guilded.gg that I made a while back and decided to bring back up

An account generator for guilded.gg that I made a while back and decided to bring back up

8 Nov 17, 2022
Batch Python Program Verify

Batch Python Program Verify About As a TA(teaching assistant) of Programming Class, it is very annoying to test students' homework assignments one by

Han-Wei Li 7 Dec 20, 2022
A one place destination to check whatever is trending on the top social and news websites at present.

UpTrend A one place destination to check whatever is trending on the top social and news websites at present. Explore the docs » View Demo · Report Bu

Google Developer Student Clubs - JGEC 10 Oct 03, 2021
A simple panel with IP, CNPJ, CEP and PLACA queries

Painel mpm Um painel simples com consultas de IP, CNPJ, CEP e PLACA Início 🌐 apt update && apt upgrade -y pkg i python git pip install requests Insta

MrDiniz 4 Nov 04, 2022
Dapp / Forge traces enhancer

traces-explorer Dapp / Forge traces enhancer Usage traces.py and pattern_* files should be in the same directory make test traces.txt py traces.

1 Feb 02, 2022
Simple Crud Python vs MySQL

Simple Crud Python vs MySQL The idea came when I was studying MySQ... A desire to create a python program that can give access to a "localhost" databa

Lucas 1 Jan 21, 2022
This Python library searches through a static directory and appends artist, title, track number, album title, duration, and genre to a .json object

This Python library searches through a static directory (needs to match your environment) and appends artist, title, track number, album title, duration, and genre to a .json object. This .json objec

Edan Ybarra 1 Jun 20, 2022
A Python simple Dice Simulator just for fun

Dice Simulator 🎲 A Simple Python Dice Simulator 🧩 🎮 💭 Description: That program make your RPG session more easy and simple. Roll the dice never be

Lauro Brant 17 May 14, 2022
This python code will get requests from SET (The Stock Exchange of Thailand) a previously-close stock price and return it in Thai Baht currency using beautiful soup 4 HTML scrapper.

This python code will get requests from SET (The Stock Exchange of Thailand) a previously-close stock price and return it in Thai Baht currency using beautiful soup 4 HTML scrapper.

Andre 1 Oct 24, 2022
Airbrake Python

airbrake-python Note. Python 3.4+ are advised to use new Airbrake Python notifier which supports async API and code hunks. Python 2.7 users should con

Airbrake 51 Dec 22, 2022
Team Curie is a group of people working together to achieve a common aim

Team Curie is a group of people working together to achieve a common aim. We are enthusiasts!.... We are setting the pace!.... We offer encouragement and motivation....And we believe TeamWork makes t

4 Aug 07, 2021
Cairo hooks for pre-commit

pre-commit-cairo Cairo hooks for pre-commit. See pre-commit for more details Using pre-commit-cairo with pre-commit Add this to your .pre-commit-confi

Fran Algaba 16 Sep 21, 2022
python package to showcase, test and build your own version of Pickhardt Payments

Pickhardt Payments Package The pickhardtpayments package is a collection of classes and interfaces that help you to test and implement your dialect of

Rene Pickhardt 37 Dec 18, 2022
FileTransfer - to exchange files from phone to laptop

A small website I locally host on my network to exchange files from my phone and other devices to my laptop.

Ronak Badhe 4 Feb 15, 2022
Python Script to add OpenGapps, Magisk, libhoudini translation library and libndk translation library to waydroid !

Waydroid Extras Script Script to add gapps and other stuff to waydroid ! Installation/Usage "lzip" is required for this script to work, install it usi

Casu Al Snek 331 Jan 02, 2023
A full-featured, hackable tiling window manager written and configured in Python

A full-featured, hackable tiling window manager written and configured in Python Features Simple, small and extensible. It's easy to write your own la

Qtile 3.8k Dec 31, 2022
Unfinished Python library based on ndspy, for Zelda: Phantom Hourglass and Spirit Tracks.

zed An unfinished library and toolset by me, for viewing and editing files from The Legend of Zelda: Phantom Hourglass and The Legend of Zelda: Spirit

4 Oct 13, 2022
Tools for collecting social media data around focal events

Social Media Focal Events The focalevents codebase provides tools for organizing data collected around focal events on social media. It is often diffi

Ryan Gallagher 80 Nov 28, 2022