showpiece/showpiece-pi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 5 Wiki Activity

Merge branch 'master' into arm64

Browse Source
This commit is contained in:
Serge Schneider 2024-02-26 14:15:02 +00:00
commit b4f2aeabde
15 changed files with 119 additions and 716 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.gitlab-ci.yml Normal file
View File

@ -0,0 +1,4 @@
include:
- project: serge/pi-gen
ref: ci
file: 'pi-gen.yml'

2
Dockerfile
View File

@ -8,7 +8,7 @@ RUN apt-get -y update && \
git vim parted \ git vim parted \
quilt coreutils qemu-user-static debootstrap zerofree zip dosfstools \ quilt coreutils qemu-user-static debootstrap zerofree zip dosfstools \
libarchive-tools libcap2-bin rsync grep udev xz-utils curl xxd file kmod bc\ libarchive-tools libcap2-bin rsync grep udev xz-utils curl xxd file kmod bc\
binfmt-support ca-certificates qemu-utils kpartx fdisk gpg pigz\ binfmt-support ca-certificates fdisk gpg pigz\
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
COPY . /pi-gen/ COPY . /pi-gen/

164
README.md
View File

@ -1,21 +1,25 @@
# pi-gen # pi-gen
Tool used to create Raspberry Pi OS images. (Previously known as Raspbian). Tool used to create Raspberry Pi OS images, and custom images based on Raspberry Pi OS,
which was in turn derived from the Raspbian project.
**Note**: Raspberry Pi OS 32 bit images are based primarily on Raspbian, while
Raspberry Pi OS 64 bit images are based primarily on Debian.
## Dependencies ## Dependencies
pi-gen runs on Debian-based operating systems. Currently it is only supported on pi-gen runs on Debian-based operating systems released after 2017, and we
either Debian Buster or Ubuntu Xenial and is known to have issues building on always advise you use the latest OS for security reasons.
earlier releases of these systems. On other Linux distributions it may be possible
to use the Docker build described below. On other Linux distributions it may be possible to use the Docker build described
below.
To install the required dependencies for `pi-gen` you should run: To install the required dependencies for `pi-gen` you should run:
```bash ```bash
apt-get install coreutils quilt parted qemu-user-static debootstrap zerofree zip \ apt-get install coreutils quilt parted qemu-user-static debootstrap zerofree zip \
dosfstools libarchive-tools libcap2-bin grep rsync xz-utils file git curl bc \ dosfstools libarchive-tools libcap2-bin grep rsync xz-utils file git curl bc \
qemu-utils kpartx gpg pigz gpg pigz xxd
``` ```
The file `depends` contains a list of tools needed. The format of this The file `depends` contains a list of tools needed. The format of this
@ -50,46 +54,22 @@ The following environment variables are supported:
* `IMG_NAME` **required** (Default: unset) * `IMG_NAME` **required** (Default: unset)
The name of the image to build with the current stage directories. Setting The name of the image to build with the current stage directories. Use this
`IMG_NAME=Raspbian` is logical for an unmodified RPi-Distro/pi-gen build, variable to set the root name of your OS, eg `IMG_NAME=Frobulator`.
but you should use something else for a customized version. Export files Export files in stages may add suffixes to `IMG_NAME`.
in stages may add suffixes to `IMG_NAME`.
* `PI_GEN_RELEASE` (Default: `Raspberry Pi reference`) * `PI_GEN_RELEASE` (Default: `Raspberry Pi reference`)
The release name to use in `/etc/issue.txt`. The default should only be used The release name to use in `/etc/issue.txt`. The default should only be used
for official Raspberry Pi builds. for official Raspberry Pi builds.
* `USE_QCOW2` **EXPERIMENTAL** (Default: `0` )
Instead of using traditional way of building the rootfs of every stage in
single subdirectories and copying over the previous one to the next one,
qcow2 based virtual disks with backing images are used in every stage.
This speeds up the build process and reduces overall space consumption
significantly.
<u>Additional optional parameters regarding qcow2 build:</u>
* `BASE_QCOW2_SIZE` (Default: 12G)
Size of the virtual qcow2 disk.
Note: it will not actually use that much of space at once but defines the
maximum size of the virtual disk. If you change the build process by adding
a lot of bigger packages or additional build stages, it can be necessary to
increase the value because the virtual disk can run out of space like a normal
hard drive would.
**CAUTION:** Although the qcow2 build mechanism will run fine inside Docker, it can happen
that the network block device is not disconnected correctly after the Docker process has
ended abnormally. In that case see [Disconnect an image if something went wrong](#Disconnect-an-image-if-something-went-wrong)
* `RELEASE` (Default: bookworm) * `RELEASE` (Default: bookworm)
The release version to build images against. Valid values are any supported The release version to build images against. Valid values are any supported
Debian release. However, since different releases will have different sets of Debian release. However, since different releases will have different sets of
packages available, you'll need to either modify your stages accordingly, or packages available, you'll need to either modify your stages accordingly, or
checkout the appropriate branch. For example, if you'd like to build a checkout the appropriate branch. For example, if you'd like to build a
`buster` image, you should do so from the `buster` branch. `bullseye` image, you should do so from the `bullseye` branch.
* `APT_PROXY` (Default: unset) * `APT_PROXY` (Default: unset)
@ -231,10 +211,10 @@ The following environment variables are supported:
If set, then instead of working through the numeric stages in order, this list will be followed. For example setting to `"stage0 stage1 mystage stage2"` will run the contents of `mystage` before stage2. Note that quotes are needed around the list. An absolute or relative path can be given for stages outside the pi-gen directory. If set, then instead of working through the numeric stages in order, this list will be followed. For example setting to `"stage0 stage1 mystage stage2"` will run the contents of `mystage` before stage2. Note that quotes are needed around the list. An absolute or relative path can be given for stages outside the pi-gen directory.
A simple example for building Raspbian: A simple example for building Raspberry Pi OS:
```bash ```bash
IMG_NAME='Raspbian' IMG_NAME='raspios'
``` ```
The config file can also be specified on the command line as an argument the `build.sh` or `build-docker.sh` scripts. The config file can also be specified on the command line as an argument the `build.sh` or `build-docker.sh` scripts.
@ -249,17 +229,17 @@ This is parsed after `config` so can be used to override values set there.
The following process is followed to build images: The following process is followed to build images:
* Loop through all of the stage directories in alphanumeric order * Interate through all of the stage directories in alphanumeric order
* Move on to the next directory if this stage directory contains a file called * Bypass a stage directory if it contains a file called
"SKIP" "SKIP"
* Run the script ```prerun.sh``` which is generally just used to copy the build * Run the script ```prerun.sh``` which is generally just used to copy the build
directory between stages. directory between stages.
* In each stage directory loop through each subdirectory and then run each of the * In each stage directory iterate through each subdirectory and then run each of the
install scripts it contains, again in alphanumeric order. These need to be named install scripts it contains, again in alphanumeric order. **These need to be named
with a two digit padded number at the beginning. with a two digit padded number at the beginning.**
There are a number of different files and directories which can be used to There are a number of different files and directories which can be used to
control different parts of the build process: control different parts of the build process:
@ -294,7 +274,7 @@ It is recommended to examine build.sh for finer details.
Docker can be used to perform the build inside a container. This partially isolates Docker can be used to perform the build inside a container. This partially isolates
the build from the host system, and allows using the script on non-debian based the build from the host system, and allows using the script on non-debian based
systems (e.g. Fedora Linux). The isolate is not complete due to the need to use systems (e.g. Fedora Linux). The isolation is not complete due to the need to use
some kernel level services for arm emulation (binfmt) and loop devices (losetup). some kernel level services for arm emulation (binfmt) and loop devices (losetup).
To build: To build:
@ -307,7 +287,7 @@ vi config # Edit your config file. See above.
If everything goes well, your finished image will be in the `deploy/` folder. If everything goes well, your finished image will be in the `deploy/` folder.
You can then remove the build container with `docker rm -v pigen_work` You can then remove the build container with `docker rm -v pigen_work`
If something breaks along the line, you can edit the corresponding scripts, and If you encounter errors during the build, you can edit the corresponding scripts, and
continue: continue:
```bash ```bash
@ -351,43 +331,36 @@ maintenance and allows for more easy customization.
`debootstrap`, which creates a minimal filesystem suitable for use as a `debootstrap`, which creates a minimal filesystem suitable for use as a
base.tgz on Debian systems. This stage also configures apt settings and base.tgz on Debian systems. This stage also configures apt settings and
installs `raspberrypi-bootloader` which is missed by debootstrap. The installs `raspberrypi-bootloader` which is missed by debootstrap. The
minimal core is installed but not configured, and the system will not quite minimal core is installed but not configured. As a result, this stage will not boot.
boot yet.
- **Stage 1** - truly minimal system. This stage makes the system bootable by - **Stage 1** - truly minimal system. This stage makes the system bootable by
installing system files like `/etc/fstab`, configures the bootloader, makes installing system files like `/etc/fstab`, configures the bootloader, makes
the network operable, and installs packages like raspi-config. At this the network operable, and installs packages like raspi-config. At this
stage the system should boot to a local console from which you have the stage the system should boot to a local console from which you have the
means to perform basic tasks needed to configure and install the system. means to perform basic tasks needed to configure and install the system.
This is as minimal as a system can possibly get, and its arguably not
really usable yet in a traditional sense yet. Still, if you want minimal,
this is minimal and the rest you could reasonably do yourself as sysadmin.
- **Stage 2** - lite system. This stage produces the Raspbian-Lite image. It - **Stage 2** - lite system. This stage produces the Raspberry Pi OS Lite image.
installs some optimized memory functions, sets timezone and charmap Stage 2 installs some optimized memory functions, sets timezone and charmap
defaults, installs fake-hwclock and ntp, wireless LAN and bluetooth support, defaults, installs fake-hwclock and ntp, wireless LAN and bluetooth support,
dphys-swapfile, and other basics for managing the hardware. It also dphys-swapfile, and other basics for managing the hardware. It also
creates necessary groups and gives the pi user access to sudo and the creates necessary groups and gives the pi user access to sudo and the
standard console hardware permission groups. standard console hardware permission groups.
There are a few tools that may not make a whole lot of sense here for Note: Raspberry Pi OS Lite contains a number of tools for development,
development purposes on a minimal system such as basic Python and Lua including `Python`, `Lua` and the `build-essential` package. If you are
packages as well as the `build-essential` package. They are lumped right creating an image to deploy in products, be sure to remove extraneous development
in with more essential packages presently, though they need not be with tools before deployment.
pi-gen. These are understandable for Raspbian's target audience, but if
you were looking for something between truly minimal and Raspbian-Lite,
here's where you start trimming.
- **Stage 3** - desktop system. Here's where you get the full desktop system - **Stage 3** - desktop system. Here's where you get the full desktop system
with X11 and LXDE, web browsers, git for development, Raspbian custom UI with X11 and LXDE, web browsers, git for development, Raspberry Pi OS custom UI
enhancements, etc. This is a base desktop system, with some development enhancements, etc. This is a base desktop system, with some development
tools installed. tools installed.
- **Stage 4** - Normal Raspbian image. System meant to fit on a 4GB card. This is the - **Stage 4** - Normal Raspberry Pi OS image. System meant to fit on a 4GB card.
stage that installs most things that make Raspbian friendly to new This is the stage that installs most things that make Raspberry Pi OS friendly
users like system documentation. to new users - e.g. system documentation.
- **Stage 5** - The Raspbian Full image. More development - **Stage 5** - The Raspberry Pi OS Full image. More development
tools, an email client, learning tools like Scratch, specialized packages tools, an email client, learning tools like Scratch, specialized packages
like sonic-pi, office productivity, etc. like sonic-pi, office productivity, etc.
@ -402,7 +375,7 @@ to `./stage2` (if building a minimal system).
```bash ```bash
# Example for building a lite system # Example for building a lite system
echo "IMG_NAME='Raspbian'" > config echo "IMG_NAME='raspios'" > config
touch ./stage3/SKIP ./stage4/SKIP ./stage5/SKIP touch ./stage3/SKIP ./stage4/SKIP ./stage5/SKIP
touch ./stage4/SKIP_IMAGES ./stage5/SKIP_IMAGES touch ./stage4/SKIP_IMAGES ./stage5/SKIP_IMAGES
sudo ./build.sh # or ./build-docker.sh sudo ./build.sh # or ./build-docker.sh
@ -429,71 +402,6 @@ follows:
* Once you're happy with the image you can remove the SKIP_IMAGES files and * Once you're happy with the image you can remove the SKIP_IMAGES files and
export your image to test export your image to test
# Regarding Qcow2 image building
### Get infos about the image in use
If you issue the two commands shown in the example below in a second command shell while a build
is running you can find out, which network block device is currently being used and which qcow2 image
is bound to it.
Example:
```bash
root@build-machine:~/$ lsblk | grep nbd
nbd1 43:32 0 10G 0 disk
├─nbd1p1 43:33 0 10G 0 part
└─nbd1p1 253:0 0 10G 0 part
root@build-machine:~/$ ps xa | grep qemu-nbd
2392 pts/6 S+ 0:00 grep --color=auto qemu-nbd
31294 ? Ssl 0:12 qemu-nbd --discard=unmap -c /dev/nbd1 image-stage4.qcow2
```
Here you can see, that the qcow2 image `image-stage4.qcow2` is currently connected to `/dev/nbd1` with
the associated partition map `/dev/mapper/nbd1p1`. Don't worry that `lsblk` shows two entries. It is totally fine, because the device map is accessible via `/dev/mapper/nbd1p1` and also via `/dev/dm-0`. This is all part of the device mapper functionality of the kernel. See `dmsetup` for further information.
### Mount a qcow2 image
If you want to examine the content of a a single stage, you can simply mount the qcow2 image found in the `WORK_DIR` directory with the tool `./imagetool.sh`.
See `./imagetool.sh -h` for further details on how to use it.
### Disconnect an image if something went wrong
It can happen, that your build stops in case of an error. Normally `./build.sh` should handle image disconnection appropriately, but in rare cases, especially during a Docker build, this may not work as expected. If that happens, starting a new build will fail and you may have to disconnect the image and/or device yourself.
A typical message indicating that there are some orphaned device mapper entries is this:
```
Failed to set NBD socket
Disconnect client, due to: Unexpected end-of-file before all bytes were read
```
If that happens go through the following steps:
1. First, check if the image is somehow mounted to a directory entry and umount it as you would any other block device, like i.e. a hard disk or USB stick.
2. Second, to disconnect an image from `qemu-nbd`, the QEMU Disk Network Block Device Server, issue the following command (be sure to change the device name to the one actually used):
```bash
sudo qemu-nbd -d /dev/nbd1
```
Note: if you use Docker build, normally no active `qemu-nbd` process exists anymore as it will be terminated when the Docker container stops.
3. To disconnect a device partition map from the network block device, execute:
```bash
sudo kpartx -d /dev/nbd1
or
sudo ./imagetool.sh --cleanup
```
Note: The `imagetool.sh` command will cleanup any /dev/nbdX that is not connected to a running `qemu-nbd` daemon. Be careful if you use network block devices for other tasks utilizing NBDs on your build machine as well.
Now you should be able to start a new build without running into troubles again. Most of the time, especially when using Docker build, you will only need no. 3 to get everything up and running again.
# Troubleshooting # Troubleshooting
## `64 Bit Systems` ## `64 Bit Systems`

3
build-docker.sh
View File

@ -135,9 +135,6 @@ time ${DOCKER} run \
$DOCKER_CMDLINE_PRE \ $DOCKER_CMDLINE_PRE \
--name "${DOCKER_CMDLINE_NAME}" \ --name "${DOCKER_CMDLINE_NAME}" \
--privileged \ --privileged \
--cap-add=ALL \
-v /dev:/dev \
-v /lib/modules:/lib/modules \
${PIGEN_DOCKER_OPTS} \ ${PIGEN_DOCKER_OPTS} \
--volume "${CONFIG_FILE}":/config:ro \ --volume "${CONFIG_FILE}":/config:ro \
-e "GIT_HASH=${GIT_HASH}" \ -e "GIT_HASH=${GIT_HASH}" \

123
build.sh
View File

@ -23,11 +23,6 @@ EOF
on_chroot << EOF on_chroot << EOF
apt-get -o Acquire::Retries=3 install --no-install-recommends -y $PACKAGES apt-get -o Acquire::Retries=3 install --no-install-recommends -y $PACKAGES
EOF EOF
if [ "${USE_QCOW2}" = "1" ]; then
on_chroot << EOF
apt-get clean
EOF
fi
fi fi
log "End ${SUB_STAGE_DIR}/${i}-packages-nr" log "End ${SUB_STAGE_DIR}/${i}-packages-nr"
fi fi
@ -38,11 +33,6 @@ EOF
on_chroot << EOF on_chroot << EOF
apt-get -o Acquire::Retries=3 install -y $PACKAGES apt-get -o Acquire::Retries=3 install -y $PACKAGES
EOF EOF
if [ "${USE_QCOW2}" = "1" ]; then
on_chroot << EOF
apt-get clean
EOF
fi
fi fi
log "End ${SUB_STAGE_DIR}/${i}-packages" log "End ${SUB_STAGE_DIR}/${i}-packages"
fi fi
@ -99,16 +89,7 @@ run_stage(){
STAGE_WORK_DIR="${WORK_DIR}/${STAGE}" STAGE_WORK_DIR="${WORK_DIR}/${STAGE}"
ROOTFS_DIR="${STAGE_WORK_DIR}"/rootfs ROOTFS_DIR="${STAGE_WORK_DIR}"/rootfs
if [ "${USE_QCOW2}" = "1" ]; then
if [ ! -f SKIP ]; then
load_qimage
fi
else
# make sure we are not umounting during export-image stage
if [ "${USE_QCOW2}" = "0" ] && [ "${NO_PRERUN_QCOW2}" = "0" ]; then
unmount "${WORK_DIR}/${STAGE}" unmount "${WORK_DIR}/${STAGE}"
fi
fi
if [ ! -f SKIP_IMAGES ]; then if [ ! -f SKIP_IMAGES ]; then
if [ -f "${STAGE_DIR}/EXPORT_IMAGE" ]; then if [ -f "${STAGE_DIR}/EXPORT_IMAGE" ]; then
@ -116,7 +97,7 @@ run_stage(){
fi fi
fi fi
if [ ! -f SKIP ]; then if [ ! -f SKIP ]; then
if [ "${CLEAN}" = "1" ] && [ "${USE_QCOW2}" = "0" ] ; then if [ "${CLEAN}" = "1" ]; then
if [ -d "${ROOTFS_DIR}" ]; then if [ -d "${ROOTFS_DIR}" ]; then
rm -rf "${ROOTFS_DIR}" rm -rf "${ROOTFS_DIR}"
fi fi
@ -133,14 +114,7 @@ run_stage(){
done done
fi fi
if [ "${USE_QCOW2}" = "1" ]; then
unload_qimage
else
# make sure we are not umounting during export-image stage
if [ "${USE_QCOW2}" = "0" ] && [ "${NO_PRERUN_QCOW2}" = "0" ]; then
unmount "${WORK_DIR}/${STAGE}" unmount "${WORK_DIR}/${STAGE}"
fi
fi
PREV_STAGE="${STAGE}" PREV_STAGE="${STAGE}"
PREV_STAGE_DIR="${STAGE_DIR}" PREV_STAGE_DIR="${STAGE_DIR}"
@ -184,10 +158,7 @@ do
done done
term() { term() {
if [ "${USE_QCOW2}" = "1" ]; then true; #TODO: Cleanup
log "Unloading image"
unload_qimage
fi
} }
trap term EXIT INT TERM trap term EXIT INT TERM
@ -270,18 +241,6 @@ source "${SCRIPT_DIR}/common"
# shellcheck source=scripts/dependencies_check # shellcheck source=scripts/dependencies_check
source "${SCRIPT_DIR}/dependencies_check" source "${SCRIPT_DIR}/dependencies_check"
export NO_PRERUN_QCOW2="${NO_PRERUN_QCOW2:-1}"
export USE_QCOW2="${USE_QCOW2:-0}"
export BASE_QCOW2_SIZE=${BASE_QCOW2_SIZE:-12G}
source "${SCRIPT_DIR}/qcow2_handling"
if [ "${USE_QCOW2}" = "1" ]; then
NO_PRERUN_QCOW2=1
else
NO_PRERUN_QCOW2=0
fi
export NO_PRERUN_QCOW2="${NO_PRERUN_QCOW2:-1}"
if [ "$SETFCAP" != "1" ]; then if [ "$SETFCAP" != "1" ]; then
export CAPSH_ARG="--drop=cap_setfcap" export CAPSH_ARG="--drop=cap_setfcap"
fi fi
@ -336,98 +295,22 @@ for EXPORT_DIR in ${EXPORT_DIRS}; do
# shellcheck source=/dev/null # shellcheck source=/dev/null
source "${EXPORT_DIR}/EXPORT_IMAGE" source "${EXPORT_DIR}/EXPORT_IMAGE"
EXPORT_ROOTFS_DIR=${WORK_DIR}/$(basename "${EXPORT_DIR}")/rootfs EXPORT_ROOTFS_DIR=${WORK_DIR}/$(basename "${EXPORT_DIR}")/rootfs
if [ "${USE_QCOW2}" = "1" ]; then
USE_QCOW2=0
EXPORT_NAME="${IMG_FILENAME}${IMG_SUFFIX}"
echo "------------------------------------------------------------------------"
echo "Running export stage for ${EXPORT_NAME}"
rm -f "${WORK_DIR}/export-image/${EXPORT_NAME}.img" || true
rm -f "${WORK_DIR}/export-image/${EXPORT_NAME}.qcow2" || true
rm -f "${WORK_DIR}/${EXPORT_NAME}.img" || true
rm -f "${WORK_DIR}/${EXPORT_NAME}.qcow2" || true
EXPORT_STAGE=$(basename "${EXPORT_DIR}")
for s in $STAGE_LIST; do
TMP_LIST=${TMP_LIST:+$TMP_LIST }$(basename "${s}")
done
FIRST_STAGE=${TMP_LIST%% *}
FIRST_IMAGE="image-${FIRST_STAGE}.qcow2"
pushd "${WORK_DIR}" > /dev/null
echo "Creating new base "${EXPORT_NAME}.qcow2" from ${FIRST_IMAGE}"
cp "./${FIRST_IMAGE}" "${EXPORT_NAME}.qcow2"
ARR=($TMP_LIST)
# rebase stage images to new export base
for CURR_STAGE in "${ARR[@]}"; do
if [ "${CURR_STAGE}" = "${FIRST_STAGE}" ]; then
PREV_IMG="${EXPORT_NAME}"
continue
fi
echo "Rebasing image-${CURR_STAGE}.qcow2 onto ${PREV_IMG}.qcow2"
qemu-img rebase -f qcow2 -u -b ${PREV_IMG}.qcow2 image-${CURR_STAGE}.qcow2
if [ "${CURR_STAGE}" = "${EXPORT_STAGE}" ]; then
break
fi
PREV_IMG="image-${CURR_STAGE}"
done
# commit current export stage into base export image
echo "Committing image-${EXPORT_STAGE}.qcow2 to ${EXPORT_NAME}.qcow2"
qemu-img commit -f qcow2 -p -b "${EXPORT_NAME}.qcow2" image-${EXPORT_STAGE}.qcow2
# rebase stage images back to original first stage for easy re-run
for CURR_STAGE in "${ARR[@]}"; do
if [ "${CURR_STAGE}" = "${FIRST_STAGE}" ]; then
PREV_IMG="image-${CURR_STAGE}"
continue
fi
echo "Rebasing back image-${CURR_STAGE}.qcow2 onto ${PREV_IMG}.qcow2"
qemu-img rebase -f qcow2 -u -b ${PREV_IMG}.qcow2 image-${CURR_STAGE}.qcow2
if [ "${CURR_STAGE}" = "${EXPORT_STAGE}" ]; then
break
fi
PREV_IMG="image-${CURR_STAGE}"
done
popd > /dev/null
mkdir -p "${WORK_DIR}/export-image/rootfs"
mv "${WORK_DIR}/${EXPORT_NAME}.qcow2" "${WORK_DIR}/export-image/"
echo "Mounting image ${WORK_DIR}/export-image/${EXPORT_NAME}.qcow2 to rootfs ${WORK_DIR}/export-image/rootfs"
mount_qimage "${WORK_DIR}/export-image/${EXPORT_NAME}.qcow2" "${WORK_DIR}/export-image/rootfs"
CLEAN=0
run_stage run_stage
CLEAN=1
USE_QCOW2=1
else
run_stage
fi
if [ "${USE_QEMU}" != "1" ]; then if [ "${USE_QEMU}" != "1" ]; then
if [ -e "${EXPORT_DIR}/EXPORT_NOOBS" ]; then if [ -e "${EXPORT_DIR}/EXPORT_NOOBS" ]; then
# shellcheck source=/dev/null # shellcheck source=/dev/null
source "${EXPORT_DIR}/EXPORT_NOOBS" source "${EXPORT_DIR}/EXPORT_NOOBS"
STAGE_DIR="${BASE_DIR}/export-noobs" STAGE_DIR="${BASE_DIR}/export-noobs"
if [ "${USE_QCOW2}" = "1" ]; then
USE_QCOW2=0
run_stage run_stage
USE_QCOW2=1
else
run_stage
fi
fi fi
fi fi
done done
if [ -x postrun.sh ]; then if [ -x "${BASE_DIR}/postrun.sh" ]; then
log "Begin postrun.sh" log "Begin postrun.sh"
cd "${BASE_DIR}" cd "${BASE_DIR}"
./postrun.sh ./postrun.sh
log "End postrun.sh" log "End postrun.sh"
fi fi
if [ "${USE_QCOW2}" = "1" ]; then
unload_qimage
fi
log "End ${BASE_DIR}" log "End ${BASE_DIR}"

2
depends
View File

@ -17,7 +17,5 @@ file
git git
lsmod:kmod lsmod:kmod
bc bc
qemu-nbd:qemu-utils
kpartx
gpg gpg
pigz pigz

4
export-image/04-set-partuuid/00-run.sh
View File

@ -1,7 +1,5 @@
#!/bin/bash -e #!/bin/bash -e
if [ "${NO_PRERUN_QCOW2}" = "0" ]; then
IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img"
IMGID="$(dd if="${IMG_FILE}" skip=440 bs=1 count=4 2>/dev/null | xxd -e | cut -f 2 -d' ')" IMGID="$(dd if="${IMG_FILE}" skip=440 bs=1 count=4 2>/dev/null | xxd -e | cut -f 2 -d' ')"
@ -13,5 +11,3 @@ if [ "${NO_PRERUN_QCOW2}" = "0" ]; then
sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab" sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab"
sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/boot/firmware/cmdline.txt" sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/boot/firmware/cmdline.txt"
fi

19
export-image/05-finalise/01-run.sh
View File

@ -83,24 +83,17 @@ cp "$ROOTFS_DIR/etc/rpi-issue" "$INFO_FILE"
dpkg -l --root "$ROOTFS_DIR" dpkg -l --root "$ROOTFS_DIR"
} >> "$INFO_FILE" } >> "$INFO_FILE"
mkdir -p "${DEPLOY_DIR}"
rm -f "${DEPLOY_DIR}/${ARCHIVE_FILENAME}${IMG_SUFFIX}.*"
rm -f "${DEPLOY_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img"
mv "$INFO_FILE" "$DEPLOY_DIR/"
if [ "${USE_QCOW2}" = "0" ] && [ "${NO_PRERUN_QCOW2}" = "0" ]; then
ROOT_DEV="$(mount | grep "${ROOTFS_DIR} " | cut -f1 -d' ')" ROOT_DEV="$(mount | grep "${ROOTFS_DIR} " | cut -f1 -d' ')"
unmount "${ROOTFS_DIR}" unmount "${ROOTFS_DIR}"
zerofree "${ROOT_DEV}" zerofree "${ROOT_DEV}"
unmount_image "${IMG_FILE}" unmount_image "${IMG_FILE}"
else
unload_qimage mkdir -p "${DEPLOY_DIR}"
make_bootable_image "${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.qcow2" "$IMG_FILE"
fi rm -f "${DEPLOY_DIR}/${ARCHIVE_FILENAME}${IMG_SUFFIX}.*"
rm -f "${DEPLOY_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img"
case "${DEPLOY_COMPRESSION}" in case "${DEPLOY_COMPRESSION}" in
zip) zip)
@ -121,3 +114,5 @@ none | *)
cp "$IMG_FILE" "$DEPLOY_DIR/" cp "$IMG_FILE" "$DEPLOY_DIR/"
;; ;;
esac esac
cp "$INFO_FILE" "$DEPLOY_DIR/"

2
export-image/prerun.sh
View File

@ -1,6 +1,5 @@
#!/bin/bash -e #!/bin/bash -e
if [ "${NO_PRERUN_QCOW2}" = "0" ]; then
IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img"
unmount_image "${IMG_FILE}" unmount_image "${IMG_FILE}"
@ -65,4 +64,3 @@ if [ "${NO_PRERUN_QCOW2}" = "0" ]; then
rsync -aHAXx --exclude /var/cache/apt/archives --exclude /boot/firmware "${EXPORT_ROOTFS_DIR}/" "${ROOTFS_DIR}/" rsync -aHAXx --exclude /var/cache/apt/archives --exclude /boot/firmware "${EXPORT_ROOTFS_DIR}/" "${ROOTFS_DIR}/"
rsync -rtx "${EXPORT_ROOTFS_DIR}/boot/firmware/" "${ROOTFS_DIR}/boot/firmware/" rsync -rtx "${EXPORT_ROOTFS_DIR}/boot/firmware/" "${ROOTFS_DIR}/boot/firmware/"
fi

4
export-noobs/00-release/00-run.sh
View File

@ -41,8 +41,4 @@ sed "${NOOBS_DIR}/os.json" -i -e "s|KERNEL|$(cat "${STAGE_WORK_DIR}/kernel_versi
sed "${NOOBS_DIR}/release_notes.txt" -i -e "s|UNRELEASED|${IMG_DATE}|" sed "${NOOBS_DIR}/release_notes.txt" -i -e "s|UNRELEASED|${IMG_DATE}|"
if [ "${USE_QCOW2}" = "1" ]; then
mv "${NOOBS_DIR}" "${DEPLOY_DIR}/"
else
cp -a "${NOOBS_DIR}" "${DEPLOY_DIR}/" cp -a "${NOOBS_DIR}" "${DEPLOY_DIR}/"
fi

4
export-noobs/prerun.sh
View File

@ -41,8 +41,4 @@ bsdtar --numeric-owner --format gnutar -C "${STAGE_WORK_DIR}/rootfs/boot" -cpf -
umount "${STAGE_WORK_DIR}/rootfs/boot" umount "${STAGE_WORK_DIR}/rootfs/boot"
bsdtar --numeric-owner --format gnutar -C "${STAGE_WORK_DIR}/rootfs" --one-file-system -cpf - . | xz -T0 > "${NOOBS_DIR}/root.tar.xz" bsdtar --numeric-owner --format gnutar -C "${STAGE_WORK_DIR}/rootfs" --one-file-system -cpf - . | xz -T0 > "${NOOBS_DIR}/root.tar.xz"
if [ "${USE_QCOW2}" = "1" ]; then
rm "$ROOTFS_DIR/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service"
fi
unmount_image "${IMG_FILE}" unmount_image "${IMG_FILE}"

114
imagetool.sh
View File

@ -1,114 +0,0 @@
#!/bin/bash
if [ "$(id -u)" != "0" ]; then
echo "Please run as root" 1>&2
exit 1
fi
progname=$(basename $0)
function usage()
{
cat << HEREDOC
Usage:
Mount Image : $progname [--mount] [--image-name <path to qcow2 image>] [--mount-point <mount point>]
Umount Image: $progname [--umount] [--mount-point <mount point>]
Cleanup NBD : $progname [--cleanup]
arguments:
-h, --help show this help message and exit
-c, --cleanup cleanup orphaned device mappings
-m, --mount mount image
-u, --umount umount image
-i, --image-name path to qcow2 image
-p, --mount-point mount point for image
This tool will use /dev/nbd1 as default for mounting an image. If you want to use another device, execute like this:
NBD_DEV=/dev/nbd2 ./$progname --mount --image-name <your image> --mount-point <your path>
HEREDOC
}
MOUNT=0
UMOUNT=0
IMAGE=""
MOUNTPOINT=""
nbd_cleanup() {
DEVS="$(lsblk | grep nbd | grep disk | cut -d" " -f1)"
if [ ! -z "${DEVS}" ]; then
for d in $DEVS; do
if [ ! -z "${d}" ]; then
QDEV="$(ps xa | grep $d | grep -v grep)"
if [ -z "${QDEV}" ]; then
kpartx -d /dev/$d && echo "Unconnected device map removed: /dev/$d"
fi
fi
done
fi
}
# As long as there is at least one more argument, keep looping
while [[ $# -gt 0 ]]; do
key="$1"
case "$key" in
-h|--help)
usage
exit
;;
-c|--cleanup)
nbd_cleanup
;;
-m|--mount)
MOUNT=1
;;
-u|--umount)
UMOUNT=1
;;
-i|--image-name)
shift
IMAGE="$1"
;;
-p|--mount-point)
shift
MOUNTPOINT="$1"
;;
*)
echo "Unknown option '$key'"
usage
exit
;;
esac
# Shift after checking all the cases to get the next option
shift
done
if [ "${MOUNT}" = "1" ] && [ "${UMOUNT}" = "1" ]; then
usage
echo "Concurrent mount options not possible."
exit
fi
if [ "${MOUNT}" = "1" ] && ([ -z "${IMAGE}" ] || [ -z "${MOUNTPOINT}" ]); then
usage
echo "Can not mount image. Image path and/or mount point missing."
exit
fi
if [ "${UMOUNT}" = "1" ] && [ -z "${MOUNTPOINT}" ]; then
usage
echo "Can not umount. Mount point parameter missing."
exit
fi
export NBD_DEV="${NBD_DEV:-/dev/nbd1}"
export MAP_BOOT_DEV=/dev/mapper/nbd1p1
export MAP_ROOT_DEV=/dev/mapper/nbd1p2
source scripts/qcow2_handling
if [ "${MOUNT}" = "1" ]; then
mount_qimage "${IMAGE}" "${MOUNTPOINT}"
elif [ "${UMOUNT}" = "1" ]; then
umount_qimage "${MOUNTPOINT}"
fi

2
scripts/common
View File

@ -123,5 +123,7 @@ ensure_loopdev_partitions() {
mknod "/dev/$partition" b "${majmin%:*}" "${majmin#*:}" mknod "/dev/$partition" b "${majmin%:*}" "${majmin#*:}"
fi fi
done done
command -v udevadm >/dev/null 2>&1 || return 0
udevadm settle 10
} }
export -f ensure_loopdev_partitions export -f ensure_loopdev_partitions

256
scripts/qcow2_handling
View File

@ -1,256 +0,0 @@
#!/bin/bash
# QCOW2 Routines
export CURRENT_IMAGE
export CURRENT_MOUNTPOINT
export NBD_DEV
export MAP_BOOT_DEV
export MAP_ROOT_DEV
# set in build.sh
# should be fairly enough for the beginning
# overwrite here by uncommenting following lines
# BASE_QCOW2_SIZE=12G
# find and initialize free block device nodes
init_nbd() {
modprobe nbd max_part=16
if [ -z "${NBD_DEV}" ]; then
for x in /sys/class/block/nbd* ; do
S=`cat $x/size`
if [ "$S" == "0" ] ; then
NBD_DEV=/dev/$(basename $x)
MAP_BOOT_DEV=/dev/mapper/$(basename $x)p1
MAP_ROOT_DEV=/dev/mapper/$(basename $x)p2
break
fi
done
fi
}
export -f init_nbd
# connect image to block device
connect_blkdev() {
init_nbd
qemu-nbd --discard=unmap -c $NBD_DEV "$1"
sync
kpartx -as $NBD_DEV
sync
CURRENT_IMAGE="$1"
}
export -f connect_blkdev
# disconnect image from block device
disconnect_blkdev() {
kpartx -d $NBD_DEV
qemu-nbd -d $NBD_DEV
NBD_DEV=
MAP_BOOT_DEV=
MAP_ROOT_DEV=
CURRENT_IMAGE=
}
export -f disconnect_blkdev
# mount qcow2 image: mount_image <image file> <mountpoint>
mount_qimage() {
connect_blkdev "$1"
mount -v -t ext4 $MAP_ROOT_DEV "$2"
mkdir -p "${ROOTFS_DIR}/boot"
mount -v -t vfat $MAP_BOOT_DEV "$2/boot"
CURRENT_MOUNTPOINT="$2"
}
export -f mount_qimage
# umount qcow2 image: umount_image <current mountpoint>
umount_qimage() {
sync
#umount "$1/boot"
while mount | grep -q "$1"; do
local LOCS
LOCS=$(mount | grep "$1" | cut -f 3 -d ' ' | sort -r)
for loc in $LOCS; do
echo "$loc"
while mountpoint -q "$loc" && ! umount "$loc"; do
sleep 0.1
done
done
done
CURRENT_MOUNTPOINT=
disconnect_blkdev
}
export -f umount_qimage
# create base image / backing image / mount image
load_qimage() {
if [ -z "${CURRENT_MOUNTPOINT}" ]; then
if [ ! -d "${ROOTFS_DIR}" ]; then
mkdir -p "${ROOTFS_DIR}";
fi
if [ "${CLEAN}" = "1" ] && [ -f "${WORK_DIR}/image-${STAGE}.qcow2" ]; then
rm -f "${WORK_DIR}/image-${STAGE}.qcow2";
fi
if [ ! -f "${WORK_DIR}/image-${STAGE}.qcow2" ]; then
pushd ${WORK_DIR} > /dev/null
init_nbd
if [ -z "${PREV_STAGE}" ]; then
echo "Creating base image: image-${STAGE}.qcow2"
# -o preallocation=falloc
qemu-img create -f qcow2 image-${STAGE}.qcow2 $BASE_QCOW2_SIZE
sync
qemu-nbd --discard=unmap -c $NBD_DEV image-${STAGE}.qcow2
sync
sfdisk $NBD_DEV << EOF
4MiB,250MiB,c,*
254MiB,,83;
EOF
sync
kpartx -as $NBD_DEV
mkdosfs -n boot -F 32 -s 4 -v $MAP_BOOT_DEV
mkfs.ext4 -L rootfs -O "^huge_file,^64bit" $MAP_ROOT_DEV
sync
else
if [ ! -f "${WORK_DIR}/image-${PREV_STAGE}.qcow2" ]; then
exit 1;
fi
echo "Creating backing image: image-${STAGE}.qcow2 <- ${WORK_DIR}/image-${PREV_STAGE}.qcow2"
qemu-img create -f qcow2 \
-o backing_file=${WORK_DIR}/image-${PREV_STAGE}.qcow2 \
${WORK_DIR}/image-${STAGE}.qcow2
sync
qemu-nbd --discard=unmap -c $NBD_DEV image-${STAGE}.qcow2
sync
kpartx -as $NBD_DEV
fi
mount -v -t ext4 $MAP_ROOT_DEV "${ROOTFS_DIR}"
mkdir -p "${ROOTFS_DIR}/boot"
mount -v -t vfat $MAP_BOOT_DEV "${ROOTFS_DIR}/boot"
CURRENT_IMAGE=${WORK_DIR}/image-${STAGE}.qcow2
CURRENT_MOUNTPOINT=${ROOTFS_DIR}
popd > /dev/null
else
mount_qimage "${WORK_DIR}/image-${STAGE}.qcow2" "${ROOTFS_DIR}"
fi
echo "Current image in use: ${CURRENT_IMAGE} (MP: ${CURRENT_MOUNTPOINT})"
fi
}
export -f load_qimage
# umount current image and refresh mount point env var
unload_qimage() {
if [ ! -z "${CURRENT_MOUNTPOINT}" ]; then
fstrim -v "${CURRENT_MOUNTPOINT}" || true
umount_qimage "${CURRENT_MOUNTPOINT}"
fi
}
export -f unload_qimage
# based on: https://github.com/SirLagz/RaspberryPi-ImgAutoSizer
# helper function for make_bootable_image, do not call directly
function resize_qcow2() {
if [ -z "$CALL_FROM_MBI" ]; then
echo "resize_qcow2: cannot be called directly, use make_bootable_image instead"
return 1
fi
# ROOT_MARGIN=$((800*1024*1024))
ROOT_MARGIN=$((1*1024*1024))
PARTED_OUT=`parted -s -m "$NBD_DEV" unit B print`
PART_NO=`echo "$PARTED_OUT" | grep ext4 | awk -F: ' { print $1 } '`
PART_START=`echo "$PARTED_OUT" | grep ext4 | awk -F: ' { print substr($2,1,length($2)-1) } '`
e2fsck -y -f $MAP_ROOT_DEV || true
DATA_SIZE=`resize2fs -P $MAP_ROOT_DEV | awk -F': ' ' { print $2 } '`
BLOCK_SIZE=$(dumpe2fs -h $MAP_ROOT_DEV | grep 'Block size' | awk -F': ' ' { print $2 }')
BLOCK_SIZE=${BLOCK_SIZE// /}
let DATA_SIZE=$DATA_SIZE+$ROOT_MARGIN/$BLOCK_SIZE
resize2fs -p $MAP_ROOT_DEV $DATA_SIZE
sleep 1
let PART_NEW_SIZE=$DATA_SIZE*$BLOCK_SIZE
let PART_NEW_END=$PART_START+$PART_NEW_SIZE
ACT1=`parted -s "$NBD_DEV" rm 2`
ACT2=`parted -s "$NBD_DEV" unit B mkpart primary $PART_START $PART_NEW_END`
NEW_IMG_SIZE=`parted -s -m "$NBD_DEV" unit B print free | tail -1 | awk -F: ' { print substr($2,1,length($2)-1) } '`
}
export -f resize_qcow2
# create raw img from qcow2: make_bootable_image <in.qcow2> <out.img>
function make_bootable_image() {
EXPORT_QCOW2="$1"
EXPORT_IMAGE="$2"
echo "Connect block device to source qcow2"
connect_blkdev "${EXPORT_QCOW2}"
echo "Resize fs and partition"
CALL_FROM_MBI=1
resize_qcow2
sync
CALL_FROM_MBI=
echo "Disconnect block device"
disconnect_blkdev
if [ -z "$NEW_IMG_SIZE" ]; then
echo "NEW_IMG_SIZE could not be calculated, cannot process image. Exit."
exit 1
fi
echo "Shrinking qcow2 image"
qemu-img resize --shrink "${EXPORT_QCOW2}" $NEW_IMG_SIZE
sync
echo "Convert qcow2 to raw image"
qemu-img convert -f qcow2 -O raw "${EXPORT_QCOW2}" "${EXPORT_IMAGE}"
sync
echo "Get PARTUUIDs from image"
IMGID="$(blkid -o value -s PTUUID "${EXPORT_IMAGE}")"
BOOT_PARTUUID="${IMGID}-01"
echo "Boot: $BOOT_PARTUUID"
ROOT_PARTUUID="${IMGID}-02"
echo "Root: $ROOT_PARTUUID"
echo "Mount image"
MOUNTROOT=${WORK_DIR}/tmpimage
mkdir -p $MOUNTROOT
MOUNTPT=$MOUNTROOT
PARTITION=2
mount "${EXPORT_IMAGE}" "$MOUNTPT" -o loop,offset=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*start=[ ]*//' | sed 's/,.*//'` * 512 ],sizelimit=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*size=[ ]*//' | sed 's/,.*//'` * 512 ] || exit 1
MOUNTPT=$MOUNTROOT/boot
PARTITION=1
mount "${EXPORT_IMAGE}" "$MOUNTPT" -o loop,offset=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*start=[ ]*//' | sed 's/,.*//'` * 512 ],sizelimit=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*size=[ ]*//' | sed 's/,.*//'` * 512 ] || exit 1
if [ ! -d "${MOUNTROOT}/root" ]; then
echo "Image damaged or not mounted. Exit."
exit 1
fi
echo "Setup PARTUUIDs"
if [ ! -z "$BOOT_PARTUUID" ] && [ ! -z "$ROOT_PARTUUID" ]; then
echo "Set UUIDs to make it bootable"
sed -i "s/BOOTDEV/PARTUUID=${BOOT_PARTUUID}/" "${MOUNTROOT}/etc/fstab"
sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${MOUNTROOT}/etc/fstab"
sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${MOUNTROOT}/boot/cmdline.txt"
fi
echo "Umount image"
sync
umount "${MOUNTROOT}/boot" || exit 1
umount "${MOUNTROOT}" || exit 1
echo "Remove qcow2 export image"
rm -f "${EXPORT_QCOW2}"
}
export -f make_bootable_image

2
stage0/prerun.sh
View File

@ -5,6 +5,6 @@ if [ "$RELEASE" != "bookworm" ]; then
echo " Please check the relevant README.md section." echo " Please check the relevant README.md section."
fi fi
if [ ! -d "${ROOTFS_DIR}" ] || [ "${USE_QCOW2}" = "1" ]; then if [ ! -d "${ROOTFS_DIR}" ]; then
bootstrap ${RELEASE} "${ROOTFS_DIR}" http://deb.debian.org/debian/ bootstrap ${RELEASE} "${ROOTFS_DIR}" http://deb.debian.org/debian/
fi fi