Recovery system and personalized live distribution
Last month, I was updating an old laptop. To avoid unnecessary risks, I did a backup with timeshift. It is not a full system backup, but unless something goes terribly wrong, it is good enough.
Something did go wrong. During the update process, the wrong cable was pulled from the wall. Since I’ve removed the battery from the old laptop (it died years ago), it went off immediately.
I plugged the cable in and restarted the machine; it automatically performed a fsck, did some changes, and then hanged. I tried a couple of reboots, even using an older kernel, but there was nothing to do.
It seems that the update process did not finish, and now the system is in an inconsistent state. Thank god I made a backup, too bad I needed to boot the system to restore it.
So I searched for a thumb drive, copied over a Linux distribution, and rebooted the machine from the USB stick. I configured the network, installed Timeshift, restored the backup, and finally, I could boot the machine and update it again.
And this experience made me think: what if there was a second, most read-only partition, with a fallback system to boot in case of emergencies?
This is nothing new, but I’ve not seen any Linux distribution doing it by default. Sure, in 99% of use cases, it’s just wasted space, as you normally never need it. But for a better user experience, it is something worth considering.
Such a system does not need most applications; just a file manager, a browser, in my case Timeshift, maybe some other utilities for data recovery.
For this computer, as there are no free partitions, it is not worth adding such a system. I wanted to avoid repeating my last experience with resizing drive partitions.
But I could not let the idea go. What if instead of a partition with a minimal OS, it is simply a live iso? It is, after all, what I’ve used for recovering the system. I could just copy the image I used on the main partition, add a corresponding entry in the bootloader, and it would cover similar experiences in the future.
Having to install the required packages is not that user-friendly, so I dismissed that idea.
Until I found the official Debian tool for creating a customized live distribution: live-build.
Create a minimal LXQT distribution
I wanted a minimal system with a graphical environment.
I wanted something that I could use by myself, and something that I could even give to someone else.
In my case, this is what I did to create the "base system":
sudo apt install live-build
mkdir -p my-live-build/auto && cd my-live-build
mkdir -p auto config/package-lists
echo ' #!/bin/sh
set -e
set -u
lb config noauto \
--distribution testing \
--apt-indices false \
--debootstrap-options "--variant=minbase" \
"${@}"
' > auto/config
chmod +x auto/config
echo '#!/bin/sh
set -e
set -u
lb clean noauto \
"${@}"
rm -f config/binary config/bootstrap config/chroot config/common config/source
' > auto/clean
chmod +x auto/clean
echo '#!/bin/sh
set -e
set -u
lb build noauto \
"${@}"
' > auto/build
chmod +x auto/build
lb config --distribution testing --apt-indices false --debootstrap-options "--variant=minbase" --archive-areas "main contrib"
echo "nm-tray firefox-esr lightdm lxqt-core" >> config/package-lists/my.list.chroot
sudo lb build Once lb build finished, I’ve tested the generated .iso file in VirtualBox.
The system does not do much yet.
Enable auto-boot
I would prefer that the system does not wait for the user to press ↵ Enter to continue the boot process.
Fortunately, the documentation 🗄️ provides an example on how to add a timeout of 5 seconds:
include menu.cfg
default vesamenu.c32
prompt 0
timeout 50 The minimal distribution is big
The .iso file is already over 850MB.
The package nm-tray is necessary for configuring the network, lightdm is the login manager, and firefox-esr is the graphical browser I’ve chosen.
I did not install lxqt directly as it brings a lot of other transitive dependencies I do not want: libreoffice, meteo-qt, yt-dlp, and others.
Considering that once upon a time we had CDs (thus more or less 700 MiB of space) with graphical distros containing whole office suites, graphical editors, and many more tools, 850MB seems like a lot.
I found out a couple of things I could remove without affecting the functionality.
With lb it is possible to configure the package manager not to install the recommended dependencies; just use the flag --apt-recommends false with lb configure.
Unfortunately it cripples the lxqt environment too much, one needs to find out packages that are not installed anymore, but that I still want to install.
The easiest "workaround" is to remove some packages afterwards.
This can be done by creating a hook script config/hooks/normal/9999-remove-packages.chroot, and then creating a new iso with sudo lb clean && lb config && sudo lb build.
#!/bin/sh
set -e
set -u
apt-get purge -y oxygen-icon-theme
apt-get purge -y zutty
apt-get purge -y xscreensaver
apt-get purge -y lxqt-powermanagement
apt-get purge -y '*-l10n'
apt-get autoremove --purge -y Most of the space is actually taken from oxygen-icon-theme and it’s a transitive dependency.
I’ve noted that in the final iso there are still some *-l10n packages that can be removed, and that the saved space is more or less 5MB. Since it is not a lot of space, I did not investigate it further.
With the proposed hook, the disk image went from 878MB to 814MB, which still feels like a lot, considering that I’ve not added yet the programs I’m interested in.
But wouldn’t it be better not to install those dependencies at all?
Turns out that with apt/apt-get/aptitude it is possible to use regular expressions; in particular, it is possible to append a - to a package name to have it removed, and in this case, to avoid installing it:
nm-tray firefox-esr lightdm lxqt-core
# exclude dependencies from lxqt-core
zutty-
xscreensaver-
oxygen-icon-theme-
lxqt-powermanagement-
*-l10n- A live system to restore my backups
Just add the missing program, and create a new image:
echo "timeshift" >> config/package-lists/my.list.chroot A more generic system
Instead of having multiple disk images on my USB stick, why not create one with all the tools I want?
Fortunately, most tools can be installed as normal packages:
nm-tray firefox-esr lightdm lxqt-core
# backup
clonezilla timeshift
# file managers
mc ncdu
# resource managers
htop nmon
# disk/partition managers
gparted fdisk gpart
# recovery
testdisk magicrescue foremost extundelete recoverdm recoverjpeg gddrescue
# exclude from lxqt
zutty-
xscreensaver-
oxygen-icon-theme-
lxqt-powermanagement-
*-l10n- One can obviously also add things outside the official repository; for example the Rescuezilla UI for Clonezilla!
To my surprise, the final disk image size is 877MB; I feared it would have been much bigger, while in fact still slightly smaller than my first attempt without avoiding some dependencies.
Change default setting
Since it is my live system, why not bundle some of my settings too?
For example, settings for mc or vim.
The recommended approach is to use /etc/skel, as the user is created at boot time, and thus the content of /etc/skel will be copied into the user directory.
One important consideration is that the live user is created by live-boot at boot time, not by live-build at build time.
Thus, I’ve partially copied my ~/.config to config/includes.chroot/etc/skel/.config/.
For changing the keyboard layout (the default is us but I do not have a keyboard with that layout), just change the system-wide default, which is stored in /etc/default/keyboard.
A German keyboard layout might look like the following:
XKBMODEL="pc105"
XKBLAYOUT="de"
XKBVARIANT=""
XKBOPTIONS=""
BACKSPACE="guess" An Italian keyboard layout might look like the following:
XKBMODEL="pc105"
XKBLAYOUT="it"
XKBVARIANT=""
XKBOPTIONS=""
BACKSPACE="guess" While the system is running, it is possible to change to another layout, but it makes sense to set as default the one that I will encounter more frequently.
You can even combine multiple default layouts, at least lxqt does not have an issue with it, and permits switching between them without changing any configuration:
XKBMODEL="pc105"
XKBLAYOUT="de,it,us"
XKBVARIANT=""
XKBOPTIONS=""
BACKSPACE="guess" Reduce system size further
The easiest way to find where the space ends is to use ncdu on the chroot folder, the currently biggest offender is /usr/lib/x86_64-linux-gnu/libLLVM.so.19.1, which takes up more than 120MB.
Otherwise, /usr/share/locale is nearly 190MB big.
After adding packages, especially graphical ones, the system image will grow bigger and bigger.
Since such "fallback system" will not be used that often, it might make sense to apply some of my notes for minimizing a Debian instance.
Avoiding recommended packages with the option --apt-recommends false in lb config should be the way to go if one wants to spend some time on it.
A good starting point for a graphical environment with lxqt is given by the following packages:
# ui
lightdm
xserver-xorg xserver-xorg-core xserver-xorg-input-all xserver-xorg-video-all
x11-xkb-utils x11-xserver-utils
xfwm4 lxqt-core papirus-icon-theme lxqt-theme-debian lxqt-branding-debian
qterminal featherpad pcmanfm-qt lxqt-archiver lximage-qt
# web
nm-tray wpasupplicant
firefox-esr It should give the look and feel of a default Debian install (with some minor glitches) and taskbar settings for the network, configured keyboard layout, and some keyboard indicators.
For comparison, sorted by size
-
Clonezilla iso: 463MB
-
GParted Live iso: 562MB
-
my personal iso with
--apt-recommends false: 759MB -
system-rescue iso: 944MB
-
Rescuezilla iso: 1.3GB
Considering that my image contains Rescuezilla, Clonezilla, and GParted as programs (granted, also the Rescuezilla iso contains those programs) and many programs that are part of system-rescue, and is as vanilla-debian as I could get it: not bad.
My iso is also missing some features, and it is not as battle-tested.
Add GRUB entry
The whole reason why I made my own live system was to be able to use it as "recovery partition", without having to install the programs I needed after starting the iso.
I thought that adding a GRUB entry would have been trivial; just install grub-imageboot and copy the iso in the /boot/images/ folder.
It did not work.
grub-imageboot created the following entry:
menuentry "Bootable ISO Image: live-image-amd64.hybrid" {
insmod part_msdos
insmod ext2
set root='hd0,msdos1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 2562e084-bf0b-4319-b6ad-19b06f39da70
else
search --no-floppy --fs-uuid --set=root 2562e084-bf0b-4319-b6ad-19b06f39da70
fi
linux16 /boot/memdisk iso
initrd16 "/boot/images/live-image-amd64.hybrid.iso"
} The disk image fails to boot with an error similar to "no ramdisk image specified".
After some searching online, I ended up with adding the following menu entry in /etc/grub.d/40_custom, and it works as intended after executing update-grub:
menuentry "Recovery system" {
set isofile="/boot/images/live-image-amd64.hybrid.iso"
loopback loop (hd0,1)$isofile
linux (loop)/live/vmlinuz boot=live findiso=$isofile toram
initrd (loop)/live/initrd.img
} I’m not sure why grub-imageboot does not do something similar.
The toram parameter copies everything to RAM, which is useful if you want, for example, to use clonezilla on the same drive where the .iso file is located..
The iso with minimal packages uses 1.1GB of RAM, and without toram it used more or less 450MB. I would say that on systems with less than 2GB of RAM, you should not use the toram flag, and eventually boot from an external usb drive.
Security risk
Having the possibility to boot anytime from this fallback is a security risk.
It acts like a backdoor; it bypasses the permission of the installed system, and it permits reading and writing anything.
This is just like booting from a USB stick, but without even the need to insert a USB stick.
The solution is to prohibit booting something different than the main system. A possible workaround would be to add a password to the bootloader.
Linux image for Windows
I think I had a long time ago, a Linux-based CD for recovering Windows installations, or at least some tools for reading and (trying to) repair the Windows registry, extracting license keys, and so on.
I am currently not able to find anything like that, and it seems that the consensus is that for managing Windows installations, you are best served with a Windows-based live image.
But if you want to copy files from a non-bootable system or do a disk image, then a Linux live system is good enough.
Improvements
Creating a working disk was extremely easy; what’s a little bit more difficult is personalizing it.
Some possible improvements for the boot screen (note that this is bypassed by the handwritten grub2 entry):
-
add an option for booting without starting the graphical interface
-
remove build date from boot wallpaper, and replace it with a version number
-
add boot entry with the
toramoption
Some improvements for the login process:
-
welcome message
-
ask if the keyboard layout is correct; if not, change it interactively to the correct one
The welcome message can be implemented by opening a textual file at login:
[Desktop Entry]
Categories=
Comment=
Exec=featherpad ~/welcome.adoc
Terminal=false
Type=Application Other possible improvements:
-
reduce the disk image even further
-
hardcode a public
sshkey for remote access so that I can connect immediately overssh, or a setup for connecting remotely to the iso image (like creating ssh keys and show a qr code with all the data for connecting) -
use
$SOURCE_DATE_EPOCHinauto/configandauto/buildto make a reproducible build
Conclusion
I wondered about how to create a recovery partition for my system, and ended up creating a custom live distribution.
It is more flexible than a recovery partition, as it can be used from USB sticks too, is a single file, and thus also easy to update, or have multiple "recovery systems" at once. The main drawback is that it might be easier to corrupt the file compared to using a separate partition.
Nevertheless, the fact that most distributions do not provide a recovery system that can be activated (maybe installed through a package) is suboptimal, as most people will not copy a live iso in their /boot folder, even if it takes less than one gigabyte of space.
To my surprise, creating a functional live is extremely easy, especially if everything you want is already packaged. The biggest downside is that you need administrator rights.
Customizing the user environment is not that different from customizing a normal desktop environment from the command line.
If you want to customize the boot process, things get a little bit more difficult.
I’m actually surprised there aren’t many more "hand-made" custom live CD.
You might be interested in downloading the iso file; but you should not trust me, who knows whats contained.
Better give a look at the project, and create your own iso. ven if you do not change anything, you’ll have more up-to-date packages.
If you have questions, comments, or found typos, the notes are not clear, or there are some errors; then just contact me.