Today, I would like to show something different, than usual reverse-engineering, that appears on my blog usually. I needed to prepare a Linux distro for myself to be able to run it on my PC. But not the ordinary operating system that we download from webpage, then use fancy graphical installer to select, what we want and where. My goals were very specific. First was to have it custom-compiled. With that in mind there aren’t many choices left (maybe Gentoo?). Second was to not cross 16 MiB boundary. Why exactly that? That’s simple. I have old (15 years old to be precise) SD/MMC card made for Canon of exactly that size. Quick check showed me that this is possible. I tried buildroot and it failed to fulfill second requirement and I decided not to continue, despite the obvious optimizations on kernel modules, I could do. It’s simply too complex for such a simple task. If not buildroot, then let’s go and see how to do such thing from scratch!
The plan
Basically the plan is to have custom Linux distro compiled from scratch. It may sound like something incredibly complex and hard to do. But it’s not. There are just few problems one must learn on how to overcome. The most problematic constraint in my case is, obviously, 16 MiB limit. To not exceed it, I have to use busybox as my userspace. This by the way simplifies distro development significantly. Busybox works the way, that, if linked statically, requires only one, single binary to be able to work correctly. So, to sum up, on software side, we need Linux and busybox. You may wonder, how do I want to boot that system, then? Well. I said I need Linux 🙂 Maybe some people know, some does not, that Linux is itself a boot loader of some kind. At least, when using UEFI and this is what I want to use, it can be loaded directly by UEFI firmware. But that’s another thing to note – I will describe a way to prepare a distro for UEFI – it won’t be as simple as that, for legacy BIOS.
The whole plan will look as follows:
- Get compiler
- Compile Linux kernel
- Compile busybox (statically and stripped!)
- Prepare initramfs with whole userspace
- Format drive as EFI System Partition
- Combine kernel and initramfs into single binary
- Optionally sign the binary, in case we want Secure Boot to be enabled
- Add entry to embedded UEFI boot manager
In the meantime, I am going to show few ways to debug the system, in case of any problems.
Preparing compiler
This is maybe not so obvious, but you need new compiler. Most likely, you could use the one that your distro provides, aliased as simply gcc. But this way, you will by the way use glibc as your standard library. For a lightweight system, glibc does not fit well, as this is most heavyweight libc, we have available. Therefore, I will use uClibc. And for that, we need a compiler. Or, to be precise, a toolchain. A toolchain, consisting of kernel headers, uClibc and gcc. Here, I could show, how to build such thing from scratch. But it’s not a tutorial about building a toolchain. This requires knowledge that can fill another tutorial. Instead, I will use the toolchain prepared by my latest project – cc-factory. For those, who did not read my previous post, it is toolchain factory running in Docker container, that provides a recipe for a toolchain that just works. If you tried compiling a toolchain from scratch in the past, then you know that this can fail, even if you use crosstool-ng. With cc-factory, it works as long as Docker can start a service and has Docker Hub working. But enough of that. In cc-factory, I am experimenting with releasing binary distributions of my toolchain. And newest one is special-made for this job. You can simply download it to you disk with:
wget 'https://github.com/v3l0c1r4pt0r/cc-factory/releases/download/x86_64-gcc10.2.0-linux5.9.13-uclibc1.0.36-1/x86_64-gcc10.2.0-linux5.9.13-uclibc1.0.36-1.tar.gz'
After that, you have to install it to your /opt directory with:
tar -xvf x86_64-gcc10.2.0-linux5.9.13-uclibc1.0.36-1.tar.gz -C /
If you prefer another installation path, then head onto cc-factory project to see, how to build one from source. Then, you can choose whatever path, you like for this SDK.
Next step is to export SDK’s bin directory to you PATH for convenience:
export PATH="$PATH:/opt/x86_64-linux-uclibc/bin"
Now, it’s nice to check, if toolchain works. Let’s write one-liner C hello world program to your temp directory, compile and run:
echo -e '#include <stdio.h>\nint main() {printf("Hello, World!\\n");}' >/tmp/main.c x86_64-linux-uclibc-gcc -static -o /tmp/main /tmp/main.c /tmp/main
And you should see the familiar text on your screen. You are ready to go!
Getting kernel
That is, in my opinion, the easiest part. First, we have to download kernel image, that we want to use. For that purpose, we need to go to Kernel Archives and download latest stable tarball. Optionally, we can download PGP signature and verify its correctness. But this is outside the scope of this tutorial. Another option to get kernel is to clone its stable repo with:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
This is the path, I chose, because I don’t like leaving multiple copies of kernel sources on my hard drive, so I prefer to have one, central clone of repo and use for any project, I need it to.
In case of tarball, it now has to be unpacked with usual:
tar -xvf linux-*.tar.gz
Then, in case of tarball, you got directory with name dependent on you kernel version, in case of git repo, it is simply linux. So we can cd into it, in any of the above cases:
cd linux*
In theory at this point, we could set the compiler, we prepared to be run by kernel buildsystem. But this is optional, as we target x86_64 platform, that we run our host on, so default compiler should be as good as uClibc variant. I case, you want to use custom one, you have to export the following:
export CROSS_COMPILE=x86_64-linux-uclibc-
Now, we have to configure our kernel.
First step is to choose defconfig. Possible values can be listed in case of x86 with:
ls arch/x86/configs/
At the time of writing this returned the following options:
i386_defconfig tiny.config x86_64_defconfig xen.config
As you can see, there are only a few. So, we do:
make x86_64_defconfig
At this point, we are ready to compile. In case, you want to do some modifications, you could do:
make menuconfig
And select (or deselect) some options there. If you think, you are ready, you can type:
make -j
And wait couple of minutes (up to half an hour), depending on speed of your PC. Afterwards, you should see that bzImage has been made and is available at arch/x86/boot/bzImage. We can copy it somewhere in order to not accidentally start its recompilation:
cp arch/x86/boot/bzImage ../
Testing on hardware
At this point, it is possible to run the kernel on our system. If you don’t have Secure Boot enabled, then you can try to copy the kernel to you EFI System Partition (wherever it is, I will call this path ESP from now on). But first create new directory for your distro:
mkdir -p ESP/EFI/linux cp arch/x86/boot/bzImage ESP/EFI/linux/linuxx64.efi
I am changing the name in the meantime, as I heard that some systems does not like binaries without .efi extension. Now, we can try to boot.There is more than one way to do it. The one that works always is to utilize EFI firmware directly. But this one differs significantly between manufacturers, so if you prefer that one, please refer to his support pages. The other is to use one of existing tools. For sure KeyTool, that manages Secure Boot keys, is able to start any executable from ESP by browsing the filesystem. But the way I would like to show is, by using so called EFI shell. Why? Because EFI shell allows us to experiment with kernel parameters, by simply typing them, as we would start new user program. And, in fact, from EFI perspective, we simply start a new program.
There are many shells in the wild. I tried EDK2, that is available on Arch as edk2-shell package. It is then available at /usr/share/edk2-shell/x64/Shell.efi You can simply copy it to ESP, just like with kernel, then you have to add it to your EFI boot manager, or use boot manager that is able to autodetect it. You can find a bunch of resources on how to do it, even from inside Linux. But, please do not add the kernel, you copied in such way, to boot manager, as we will play with it a bit, later. Now, in EFI shell, you will be presented with a list of partitions, that the shell has detected. This might be not so obvious at first, but, somehow, you have to identify the one, that is you EFI partition. You should see a similar numbering to the one you see in Linux console, when you list with e.g. lsblk. You can try guessing it by typing its name, in similar way as in Windows, e.g.:
FS0:
to switch to FS0 partition. Then you can simply type ls to list its content. Once, you found the right one, it’s worth to remember it, as you might need it couple of times. How many, depends on how many problems, you would have.
Then go to the directory, where you copied your kernel:
cd EFI\linux
And ls to make sure, it is there. Now, we are ready to call it:
linuxx64.efi
Yes. It’s as simple as that. But this will fail. We did not provide any root filesystem for our kernel, so it will gonna panic. But don’t worry, as long as reason of the panic is like that:
not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
then it was expected. In case of any other error, act accordingly, as this may be hardware dependent. So, probably, something has to be reconfigured in menuconfig.
Testing in qemu
It’s quite a nice option to be able to test your work on real hardware. However, in my opinion, it is much easier to do it in virtualized environment, that you can quickly reset at any time and where you would have latest binaries for testing, all the time, without wasting time for flashing them somewhere. Therefore I recommend to prepare qemu as such environment.
For that, you will need two programs – qemu-system-x86_64, that on x86 systems should be provided with qemu package (at least this is the case on Arch) and OVMF, which stands for Open Virtual Machine Firmware and is provided as edk2-ovmf on Arch. It is good idea to create new directory for qemu, as we will copy OVMF there for convenience:
mkdir qemu && cp /usr/share/edk2-ovmf/x64/OVMF.fd qemu/bios.bin
This is how to do it on Arch Linux. Two things are important, first that we need OVMF.fd, wherever it is and, second, it has to be named bios.bin, as this is where qemu will look for it.
At this point, we can run qemu with:
qemu-system-x86_64 -L . -kernel ../bzImage
And the result should be same as on real hardware, so:
not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
One trick that is good to know here, is changing console, so output will go to serial, instead of monitor, from which we can’t copy. To do this, we can run qemu like that:
qemu-system-x86_64 -L . -kernel ../bzImage -append 'console=ttyS0'
Now, last message should be about booting the kernel. To see the rest, select View->serial0. Now, you can copy your error! And that’s another reason to use qemu, until everything works.
Getting busybox
To be able to start the kernel that will not panic, we have to provide some userspace program or programs. Kernel needs at least one program, called init that will run until the system powers off. In theory, we can simply provide a shell here, but accidental Ctrl-D will panic it anyway. Fortunately, busybox can be every program we need, including init. All we need is proper symlink. But, let’s start from the beginning.
At first, we have to go to https://www.busybox.net/downloads/ and download the latest bz2 package. Optionally, we can also download its digest or signature. But as with kernel, I will skip that part. Next, we can unpack it with:
tar -xvf busybox-*.tar.bz2 cd busybox-*
As with kernel, we should have prefix of our compiler exported:
export CROSS_COMPILE=x86_64-linux-uclibc-
First thing that is worth to do is to start menuconfig to tweak some options:
make menuconfig
As I am targeting the configuration with limited resources, I deselected CONFIG_DESKTOP. This should make busybox a little smaller at a cost of decreased compatibility. This is fine for me, as I am not going to use any sophisticated scripting.
Important change for our use case is to select CONFIG_STATIC. This will make any shared libraries unnecessary. But this makes sense only if we are not planning any further tools to be included in our distribution. Otherwise all programs will have to be linked statically. This will increase image size dramatically. On the other hand, dynamic linking makes initramfs image slightly more complicated, so I don’t want to go that way for clarity.
I left the rest unchanged. Now, we can start compiling:
make -j
And after few minutes, it should be ready. Binaries busybox and busybox_unstripped should appear in root directory of busybox sources. Now we are ready for the next step.
Create initial ramdisk (initramfs)
In the past, this was partition image. Nowadays, it is simpler. Kernel can be provided with simple archive, storing whole filesystem. It is called initial ramdisk (or in short initramfs). It can be even optionally compressed, which is the option, I will use, due to the space constraints, I have. This can be prepared in ordinary filesystem, then packed and compressed. So, we can start by creating initramfs directory and cd-ing into it:
mkdir -p initramfs && cd initramfs
Now, we can create some basic root tree:
mkdir bin dev etc proc
We could do more, but this should be sufficient to get stable system. So, let’s stop in here.
Now, the most important file – busybox. We copy it to bin:
cp ../busybox/busybox bin/
Now, we can create a symlink from every tool available in busybox to bin/ directory:
cd bin && for t in $(./busybox --list); do ln -s busybox $t; done && cd -
We need just a one, little thing in order to be able to run it:
ln -s bin/busybox init
That is, where kernel looks for its first process by default.
At this point, we have a functional operating system root. Then, it is a good moment to prepare our first initramfs. It is actually quite simple. It is just important to be in initramfs root. Then we simply do:
find . | cpio -o -H newc > ../initramfs.cpio gzip < initramfs.cpio > initramfs.cpio.gz
In theory, few other compression methods beside gzip could be used. Full list of what your kernel is capable of decompressing can be found in menuconfig.
We can try running it in qemu, but it is not fully ready yet. In case you want to try, all that has to be appended to previous method is specifying initrd parameter:
linuxx64.efi initrd=/EFI/linux/initramfs.cpio.gz
Obviously, initramfs has to be copied to ESP first. If you tried, then you see an error appearing on console every few seconds. But, if you press ENTER, then you see shell prompt. And after a second, an error again 🙁
To get rid of this, we have to provide at least some basic /dev tree. Looking at the errors, we see that it needs at least 4 first ttys. Then, let’s create them:
cd dev mknod tty1 c 4 1 mknod tty2 c 4 2 mknod tty3 c 4 3 mknod tty4 c 4 4
Let’s try again. Now it’s way better! No repeating error messages.
But we see one more error still appearing on boot. This comes from busybox, trying to read SysV init RC files, that are definitely not there.
mkdir etc/init.d echo '#!/bin/sh' > etc/init.d/rcS chmod +x etc/init.d/rcS
Now, it’s better. But, still there is something missing. Let’s try to shut the system down. It works. Let’s try on real hardware. Or maybe believe me 🙂 It’s not worth to waste time. You won’t be able to do so, until you do:
mount -t proc proc /proc
It’s better idea to have this in fstab and mount automatically, to not have to remember about it all the time:
echo -e 'proc\t /proc\t proc\t defaults\t 0 1' >etc/fstab echo 'mount -a' >> etc/init.d/rcS
Let’s try it. You should see usual process info in /proc.
Preparing new EFI system partition
As my goal was to put my new Linux on SD/MMC memory card, ESP has to be created first. I was lucky to have my card already formatted in supported format – FAT12 (sic!). In case, yours is not, all you have to do is (please do not and do the rest first):
mkfs.vfat -F12 /dev/sdz1
Obviously, sdz9 has to be substituted for the partition, you want to use. Most likely, you would also not like FAT12, as much as I do 🙂 Then -F16 or -F32 sounds like a better option. But be careful, even FAT32 supports files of size up to 4 GiB. But this should not be a problem for ESP, as its size usually does not exceed 100 MiB. For reference, here is the table of max sizes (approximate, as manipulating fs parameters could increase these values) for each FAT variant:
| Variant | Max Size |
|---|---|
| FAT12 | 16 MiB |
| FAT16 | 2 GiB |
| FAT32 | 2 TiB |
| exFAT | 128 PiB |
But, I started from the end, actually, as most important here is to have proper MBR partition table. So, we have to start fdisk as root:
sudo fdisk /dev/sdz
Where, obiously, sdz is your storage device, whatever it is. Then, it is simplest to create new table. Full log below:
Welcome to fdisk (util-linux 2.36).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xac30a01f.
Command (m for help): o
Created a new DOS disklabel with disk identifier 0xdef65f39.
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-32767, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-32767, default 32767):
Created a new partition 1 of type ‘Linux’ and of size 15 MiB.
Command (m for help): t
Selected partition 1
Hex code or alias (type L to list all): ef
Changed type of partition ‘Linux’ to ‘EFI (FAT-12/16/32)’.
Command (m for help): a
Selected partition 1
The bootable flag on partition 1 is enabled now.
Command (m for help): p
Disk /dev/sdz: 16 MiB, 16777216 bytes, 32768 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xdef65f39
Device Boot Start End Sectors Size Id Type
/dev/sdz1 * 2048 32767 30720 15M ef EFI (FAT-12/16/32)
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Now, we can create filesystem as described above:
mkfs.vfat -F12 /dev/sdz1
At this point, we can prepare working EFI partition. Just mount the partition and create the same directory structure, as we did above on our main ESP:
mount /dev/sdz1 /mnt mkdir -p /mnt/EFI/linux
And copy our kernel and initramfs there. As we have initramfs in separate file, we still have to use EFI shell to start it manually.
Single kernel-initramfs binary
As I showed above, it is enough to have kernel and initramfs in separate files. But this is not a must. And there are situations, where it is worth to have them combined in single file – Secure Boot. But this also makes life easier, as a lot of tools in EFI, including vendor firmware allows to select file from any EFI partition it could find. In case of two files, we can load it, but will end up wtih a panic due to lack of root filesystem. With combined binary, it will just work. Let’s see how to create it:
objcopy \ --add-section .osrel="os-release" --change-section-vma .osrel=0x20000 \ --add-section .cmdline="kernel-command-line.txt" --change-section-vma .cmdline=0x30000 \ --add-section .linux="bzImage" --change-section-vma .linux=0x2000000 \ --add-section .initrd="initramfs.cpio.gz" --change-section-vma .initrd=0x3000000 \ "efi.stub" "linux.efi"
As you can see, a lot is happening here. So I’ll start describing important parts from the top. First file, that we include into our new image is os-release. Its format is more or less like below:
NAME="Linux" PRETTY_NAME="Linux" ID=linux HOME_URL="https://example.com/"
Full documentation can be found in os-release(5) manual. This one is most likely not usable by our operating system as this is systemd-related. But just in case, it is good to have it.
Next one is kernel-command-line.txt. This contains kernel parameters and unlike os-release, is used by kernel to append it to your command line. So, if you need some custom parameters, then store it there. Otherwise, leave the file empty.
Next one is quite obvious. The kernel image. And the last section, we create, contains initramfs.
Last, but not least, we need EFI stub, to which all these things will be appended. On Arch Linux, you should already have it provided by systemd package. Copy it to the directory, where rest of binaries sits:
cp /usr/lib/systemd/boot/efi/linuxx64.efi.stub ./efi.stub
After execution of objcopy, output should appear in linux.efi.
Signing Linux EFI binary
If you have Secure Boot mechanism active, then you have to sign the binary, you just created. Otherwise, you can simply copy it to your new ESP, into /EFI/linux/ directory and skip this chapter. If you have Secure Boot active, chances are that you use KeyTool or shim and sign the binaries the first time, you attempt to run them. In this case, you can also skip this section. But, if you do not use KeyTool for some reason, then you want to use sbsign tool at this point:
sbsign --key DB.key --cert DB.crt --output ESP/EFI/linux/linux.efi linux.efi
Provided that you have all your keys and Linux image in the same directory. That’s it. You are ready to start the kernel.
Adding entry to EFI boot manager
To make things easier, it is worth to add new entry to your built-in EFI boot manager. Then, you would be able to select your new Linux, whenever you plug in your storage device:
efibootmgr --create --disk /dev/sdz --part 1 --loader /EFI/linux/linux.efi --label "My Own Linux Distro" --verbose
Of course, modify /dev/sdz accordingly.
Final test
Now, you can restart your PC, press button for selecting other boot device and select ‘My Own Linux Distro’. After a second or two, you should see a working Linux system. It still has few things to improve, like logging of kernel messages, but generally it is a usable Linux system. It can become a base for further improvements and enhancements.
As I showed, the process of starting development of your own Linux distro is not that hard. But on the other hand it requires some amount of work. If you would like to develop your system further, or update kernel or busybox from time to time, it generates another bunch of challenges related to automating the process and making it more repeatable. But this is a topic for another story…
How much did the final build weigh?
Around 12 megabytes.
Gentoo is a fallen project, you pretty much have to rely on their stage3 blobs. Gales Linux is the solution.
No, Gales Linux is YOUR solution. If it was a viable thing then now 2 years later everyone would be running it now wouldn’t they?
Great work! It reminds me extra small linux live-distro named slitaz: it was something about 30mb but there was preinstalled lightweith office, image viewer, player, archiver an so on.
Why did you make so small live-image? There was some specific task or just for fun to useful utilize old SD-card? Does your PC support booting from SD-card or you have used usb-sd adapter that recognized as usual usb-flash by bios?
I ask because some time ago I needed to boot linux from a sd card on a small motherboard of netbook, but it only supported booting from HDD, USB and PXE(LAN).
Well, I would say that having old SD was good excuse to learn something new 🙂
Even today PCs does not support booting from SD for reason unknown to me, as this is technically possible to do with right firmware. But, yes, I have USB adapter that allows to boot from card and I guess this is rather common feature.