From 69037ce9fe9abe51cc6510971b3fb307745b1ef9 Mon Sep 17 00:00:00 2001 From: Tom Dewey <146750643+tdewey-rpi@users.noreply.github.com> Date: Mon, 19 Feb 2024 14:32:58 +0000 Subject: [PATCH 1/5] Update README.md (#755) Remove references to 'Raspbian', and instead refer to 'Raspberry Pi OS'. Additionally, a first pass of editing for conciseness. Replace bookworm instructions with bullseye --- README.md | 88 ++++++++++++++++++++++++++----------------------------- 1 file changed, 41 insertions(+), 47 deletions(-) diff --git a/README.md b/README.md index 2fdb65e..e19b2b3 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,18 @@ # pi-gen -Tool used to create Raspberry Pi OS images. (Previously known as Raspbian). +Tool used to create Raspberry Pi OS images, and custom images based on Raspberry Pi OS, +which was in turn derived from the Raspbian project. +**Note**: Raspberry Pi OS 32 bit images are based primarily on Raspbian, while +Raspberry Pi OS 64 bit images are based primarily on Debian. ## Dependencies -pi-gen runs on Debian-based operating systems. Currently it is only supported on -either Debian Buster or Ubuntu Xenial and is known to have issues building on -earlier releases of these systems. On other Linux distributions it may be possible -to use the Docker build described below. +pi-gen runs on Debian-based operating systems released after 2017, and we +always advise you use the latest OS for security reasons. + +On other Linux distributions it may be possible to use the Docker build described +below. To install the required dependencies for `pi-gen` you should run: @@ -50,17 +54,16 @@ The following environment variables are supported: * `IMG_NAME` **required** (Default: unset) - The name of the image to build with the current stage directories. Setting - `IMG_NAME=Raspbian` is logical for an unmodified RPi-Distro/pi-gen build, - but you should use something else for a customized version. Export files - in stages may add suffixes to `IMG_NAME`. + The name of the image to build with the current stage directories. Use this + variable to set the root name of your OS, eg `IMG_NAME=Frobulator`. + Export files in stages may add suffixes to `IMG_NAME`. * `PI_GEN_RELEASE` (Default: `Raspberry Pi reference`) The release name to use in `/etc/issue.txt`. The default should only be used for official Raspberry Pi builds. -* `USE_QCOW2` **EXPERIMENTAL** (Default: `0` ) + * `USE_QCOW2` **EXPERIMENTAL** (Default: `0` ) Instead of using traditional way of building the rootfs of every stage in single subdirectories and copying over the previous one to the next one, @@ -73,14 +76,12 @@ The following environment variables are supported: * `BASE_QCOW2_SIZE` (Default: 12G) Size of the virtual qcow2 disk. - Note: it will not actually use that much of space at once but defines the - maximum size of the virtual disk. If you change the build process by adding - a lot of bigger packages or additional build stages, it can be necessary to - increase the value because the virtual disk can run out of space like a normal - hard drive would. + Note: This is a maximum size - not a guaranteed allocation size. You may need + to increase the default size if you are building a particularly full image, containing + many packages or pre-built artefacts. - **CAUTION:** Although the qcow2 build mechanism will run fine inside Docker, it can happen - that the network block device is not disconnected correctly after the Docker process has + **CAUTION:** Although the qcow2 build mechanism will run inside Docker, there is a known issue + where the network block device is not disconnected correctly after the Docker process has ended abnormally. In that case see [Disconnect an image if something went wrong](#Disconnect-an-image-if-something-went-wrong) * `RELEASE` (Default: bookworm) @@ -89,7 +90,7 @@ The following environment variables are supported: Debian release. However, since different releases will have different sets of packages available, you'll need to either modify your stages accordingly, or checkout the appropriate branch. For example, if you'd like to build a - `buster` image, you should do so from the `buster` branch. + `bullseye` image, you should do so from the `bullseye` branch. * `APT_PROXY` (Default: unset) @@ -231,10 +232,10 @@ The following environment variables are supported: If set, then instead of working through the numeric stages in order, this list will be followed. For example setting to `"stage0 stage1 mystage stage2"` will run the contents of `mystage` before stage2. Note that quotes are needed around the list. An absolute or relative path can be given for stages outside the pi-gen directory. -A simple example for building Raspbian: +A simple example for building Raspberry Pi OS: ```bash -IMG_NAME='Raspbian' +IMG_NAME='raspios' ``` The config file can also be specified on the command line as an argument the `build.sh` or `build-docker.sh` scripts. @@ -249,17 +250,17 @@ This is parsed after `config` so can be used to override values set there. The following process is followed to build images: - * Loop through all of the stage directories in alphanumeric order + * Interate through all of the stage directories in alphanumeric order - * Move on to the next directory if this stage directory contains a file called + * Bypass a stage directory if it contains a file called "SKIP" * Run the script ```prerun.sh``` which is generally just used to copy the build directory between stages. - * In each stage directory loop through each subdirectory and then run each of the - install scripts it contains, again in alphanumeric order. These need to be named - with a two digit padded number at the beginning. + * In each stage directory iterate through each subdirectory and then run each of the + install scripts it contains, again in alphanumeric order. **These need to be named + with a two digit padded number at the beginning.** There are a number of different files and directories which can be used to control different parts of the build process: @@ -294,7 +295,7 @@ It is recommended to examine build.sh for finer details. Docker can be used to perform the build inside a container. This partially isolates the build from the host system, and allows using the script on non-debian based -systems (e.g. Fedora Linux). The isolate is not complete due to the need to use +systems (e.g. Fedora Linux). The isolation is not complete due to the need to use some kernel level services for arm emulation (binfmt) and loop devices (losetup). To build: @@ -307,7 +308,7 @@ vi config # Edit your config file. See above. If everything goes well, your finished image will be in the `deploy/` folder. You can then remove the build container with `docker rm -v pigen_work` -If something breaks along the line, you can edit the corresponding scripts, and +If you encounter errors during the build, you can edit the corresponding scripts, and continue: ```bash @@ -351,43 +352,36 @@ maintenance and allows for more easy customization. `debootstrap`, which creates a minimal filesystem suitable for use as a base.tgz on Debian systems. This stage also configures apt settings and installs `raspberrypi-bootloader` which is missed by debootstrap. The - minimal core is installed but not configured, and the system will not quite - boot yet. + minimal core is installed but not configured. As a result, this stage will not boot. - **Stage 1** - truly minimal system. This stage makes the system bootable by installing system files like `/etc/fstab`, configures the bootloader, makes the network operable, and installs packages like raspi-config. At this stage the system should boot to a local console from which you have the means to perform basic tasks needed to configure and install the system. - This is as minimal as a system can possibly get, and its arguably not - really usable yet in a traditional sense yet. Still, if you want minimal, - this is minimal and the rest you could reasonably do yourself as sysadmin. - - **Stage 2** - lite system. This stage produces the Raspbian-Lite image. It - installs some optimized memory functions, sets timezone and charmap + - **Stage 2** - lite system. This stage produces the Raspberry Pi OS Lite image. + Stage 2 installs some optimized memory functions, sets timezone and charmap defaults, installs fake-hwclock and ntp, wireless LAN and bluetooth support, dphys-swapfile, and other basics for managing the hardware. It also creates necessary groups and gives the pi user access to sudo and the standard console hardware permission groups. - There are a few tools that may not make a whole lot of sense here for - development purposes on a minimal system such as basic Python and Lua - packages as well as the `build-essential` package. They are lumped right - in with more essential packages presently, though they need not be with - pi-gen. These are understandable for Raspbian's target audience, but if - you were looking for something between truly minimal and Raspbian-Lite, - here's where you start trimming. + Note: Raspberry Pi OS Lite contains a number of tools for development, + including `Python`, `Lua` and the `build-essential` package. If you are + creating an image to deploy in products, be sure to remove extraneous development + tools before deployment. - **Stage 3** - desktop system. Here's where you get the full desktop system - with X11 and LXDE, web browsers, git for development, Raspbian custom UI + with X11 and LXDE, web browsers, git for development, Raspberry Pi OS custom UI enhancements, etc. This is a base desktop system, with some development tools installed. - - **Stage 4** - Normal Raspbian image. System meant to fit on a 4GB card. This is the - stage that installs most things that make Raspbian friendly to new - users like system documentation. + - **Stage 4** - Normal Raspberry Pi OS image. System meant to fit on a 4GB card. + This is the stage that installs most things that make Raspberry Pi OS friendly + to new users - e.g. system documentation. - - **Stage 5** - The Raspbian Full image. More development + - **Stage 5** - The Raspberry Pi OS Full image. More development tools, an email client, learning tools like Scratch, specialized packages like sonic-pi, office productivity, etc. @@ -402,7 +396,7 @@ to `./stage2` (if building a minimal system). ```bash # Example for building a lite system -echo "IMG_NAME='Raspbian'" > config +echo "IMG_NAME='raspios'" > config touch ./stage3/SKIP ./stage4/SKIP ./stage5/SKIP touch ./stage4/SKIP_IMAGES ./stage5/SKIP_IMAGES sudo ./build.sh # or ./build-docker.sh From 502e5fa0e3e1342ac7eedaea964b40ec55d7adb4 Mon Sep 17 00:00:00 2001 From: Mark Spatz Date: Thu, 22 Feb 2024 23:19:10 -0500 Subject: [PATCH 2/5] Remove QCOW2 build mechanism (#648) --- Dockerfile | 2 +- README.md | 88 +-------- build-docker.sh | 3 - build.sh | 131 +------------ depends | 2 - export-image/04-set-partuuid/00-run.sh | 18 +- export-image/05-finalise/01-run.sh | 23 +-- export-image/prerun.sh | 104 +++++----- export-noobs/00-release/00-run.sh | 6 +- export-noobs/prerun.sh | 4 - imagetool.sh | 114 ----------- scripts/qcow2_handling | 256 ------------------------- stage0/prerun.sh | 2 +- 13 files changed, 78 insertions(+), 675 deletions(-) delete mode 100755 imagetool.sh delete mode 100644 scripts/qcow2_handling diff --git a/Dockerfile b/Dockerfile index 2a5d8fe..673f756 100644 --- a/Dockerfile +++ b/Dockerfile @@ -8,7 +8,7 @@ RUN apt-get -y update && \ git vim parted \ quilt coreutils qemu-user-static debootstrap zerofree zip dosfstools \ libarchive-tools libcap2-bin rsync grep udev xz-utils curl xxd file kmod bc\ - binfmt-support ca-certificates qemu-utils kpartx fdisk gpg pigz\ + binfmt-support ca-certificates fdisk gpg pigz\ && rm -rf /var/lib/apt/lists/* COPY . /pi-gen/ diff --git a/README.md b/README.md index e19b2b3..3529770 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ To install the required dependencies for `pi-gen` you should run: ```bash apt-get install coreutils quilt parted qemu-user-static debootstrap zerofree zip \ dosfstools libarchive-tools libcap2-bin grep rsync xz-utils file git curl bc \ -qemu-utils kpartx gpg pigz +gpg pigz ``` The file `depends` contains a list of tools needed. The format of this @@ -63,27 +63,6 @@ The following environment variables are supported: The release name to use in `/etc/issue.txt`. The default should only be used for official Raspberry Pi builds. - * `USE_QCOW2` **EXPERIMENTAL** (Default: `0` ) - - Instead of using traditional way of building the rootfs of every stage in - single subdirectories and copying over the previous one to the next one, - qcow2 based virtual disks with backing images are used in every stage. - This speeds up the build process and reduces overall space consumption - significantly. - - Additional optional parameters regarding qcow2 build: - - * `BASE_QCOW2_SIZE` (Default: 12G) - - Size of the virtual qcow2 disk. - Note: This is a maximum size - not a guaranteed allocation size. You may need - to increase the default size if you are building a particularly full image, containing - many packages or pre-built artefacts. - - **CAUTION:** Although the qcow2 build mechanism will run inside Docker, there is a known issue - where the network block device is not disconnected correctly after the Docker process has - ended abnormally. In that case see [Disconnect an image if something went wrong](#Disconnect-an-image-if-something-went-wrong) - * `RELEASE` (Default: bookworm) The release version to build images against. Valid values are any supported @@ -423,71 +402,6 @@ follows: * Once you're happy with the image you can remove the SKIP_IMAGES files and export your image to test -# Regarding Qcow2 image building - -### Get infos about the image in use - -If you issue the two commands shown in the example below in a second command shell while a build -is running you can find out, which network block device is currently being used and which qcow2 image -is bound to it. - -Example: - -```bash -root@build-machine:~/$ lsblk | grep nbd -nbd1 43:32 0 10G 0 disk -├─nbd1p1 43:33 0 10G 0 part -└─nbd1p1 253:0 0 10G 0 part - -root@build-machine:~/$ ps xa | grep qemu-nbd - 2392 pts/6 S+ 0:00 grep --color=auto qemu-nbd -31294 ? Ssl 0:12 qemu-nbd --discard=unmap -c /dev/nbd1 image-stage4.qcow2 -``` - -Here you can see, that the qcow2 image `image-stage4.qcow2` is currently connected to `/dev/nbd1` with -the associated partition map `/dev/mapper/nbd1p1`. Don't worry that `lsblk` shows two entries. It is totally fine, because the device map is accessible via `/dev/mapper/nbd1p1` and also via `/dev/dm-0`. This is all part of the device mapper functionality of the kernel. See `dmsetup` for further information. - -### Mount a qcow2 image - -If you want to examine the content of a a single stage, you can simply mount the qcow2 image found in the `WORK_DIR` directory with the tool `./imagetool.sh`. - -See `./imagetool.sh -h` for further details on how to use it. - -### Disconnect an image if something went wrong - -It can happen, that your build stops in case of an error. Normally `./build.sh` should handle image disconnection appropriately, but in rare cases, especially during a Docker build, this may not work as expected. If that happens, starting a new build will fail and you may have to disconnect the image and/or device yourself. - -A typical message indicating that there are some orphaned device mapper entries is this: - -``` -Failed to set NBD socket -Disconnect client, due to: Unexpected end-of-file before all bytes were read -``` - -If that happens go through the following steps: - -1. First, check if the image is somehow mounted to a directory entry and umount it as you would any other block device, like i.e. a hard disk or USB stick. - -2. Second, to disconnect an image from `qemu-nbd`, the QEMU Disk Network Block Device Server, issue the following command (be sure to change the device name to the one actually used): - - ```bash - sudo qemu-nbd -d /dev/nbd1 - ``` - - Note: if you use Docker build, normally no active `qemu-nbd` process exists anymore as it will be terminated when the Docker container stops. - -3. To disconnect a device partition map from the network block device, execute: - - ```bash - sudo kpartx -d /dev/nbd1 - or - sudo ./imagetool.sh --cleanup - ``` - - Note: The `imagetool.sh` command will cleanup any /dev/nbdX that is not connected to a running `qemu-nbd` daemon. Be careful if you use network block devices for other tasks utilizing NBDs on your build machine as well. - -Now you should be able to start a new build without running into troubles again. Most of the time, especially when using Docker build, you will only need no. 3 to get everything up and running again. - # Troubleshooting ## `64 Bit Systems` diff --git a/build-docker.sh b/build-docker.sh index 83e76a5..41e341f 100755 --- a/build-docker.sh +++ b/build-docker.sh @@ -144,9 +144,6 @@ time ${DOCKER} run \ $DOCKER_CMDLINE_PRE \ --name "${DOCKER_CMDLINE_NAME}" \ --privileged \ - --cap-add=ALL \ - -v /dev:/dev \ - -v /lib/modules:/lib/modules \ ${PIGEN_DOCKER_OPTS} \ --volume "${CONFIG_FILE}":/config:ro \ -e "GIT_HASH=${GIT_HASH}" \ diff --git a/build.sh b/build.sh index 7d7bc40..f34adfc 100755 --- a/build.sh +++ b/build.sh @@ -23,11 +23,6 @@ EOF on_chroot << EOF apt-get -o Acquire::Retries=3 install --no-install-recommends -y $PACKAGES EOF - if [ "${USE_QCOW2}" = "1" ]; then - on_chroot << EOF -apt-get clean -EOF - fi fi log "End ${SUB_STAGE_DIR}/${i}-packages-nr" fi @@ -38,11 +33,6 @@ EOF on_chroot << EOF apt-get -o Acquire::Retries=3 install -y $PACKAGES EOF - if [ "${USE_QCOW2}" = "1" ]; then - on_chroot << EOF -apt-get clean -EOF - fi fi log "End ${SUB_STAGE_DIR}/${i}-packages" fi @@ -99,16 +89,7 @@ run_stage(){ STAGE_WORK_DIR="${WORK_DIR}/${STAGE}" ROOTFS_DIR="${STAGE_WORK_DIR}"/rootfs - if [ "${USE_QCOW2}" = "1" ]; then - if [ ! -f SKIP ]; then - load_qimage - fi - else - # make sure we are not umounting during export-image stage - if [ "${USE_QCOW2}" = "0" ] && [ "${NO_PRERUN_QCOW2}" = "0" ]; then - unmount "${WORK_DIR}/${STAGE}" - fi - fi + unmount "${WORK_DIR}/${STAGE}" if [ ! -f SKIP_IMAGES ]; then if [ -f "${STAGE_DIR}/EXPORT_IMAGE" ]; then @@ -116,7 +97,7 @@ run_stage(){ fi fi if [ ! -f SKIP ]; then - if [ "${CLEAN}" = "1" ] && [ "${USE_QCOW2}" = "0" ] ; then + if [ "${CLEAN}" = "1" ]; then if [ -d "${ROOTFS_DIR}" ]; then rm -rf "${ROOTFS_DIR}" fi @@ -133,14 +114,7 @@ run_stage(){ done fi - if [ "${USE_QCOW2}" = "1" ]; then - unload_qimage - else - # make sure we are not umounting during export-image stage - if [ "${USE_QCOW2}" = "0" ] && [ "${NO_PRERUN_QCOW2}" = "0" ]; then - unmount "${WORK_DIR}/${STAGE}" - fi - fi + unmount "${WORK_DIR}/${STAGE}" PREV_STAGE="${STAGE}" PREV_STAGE_DIR="${STAGE_DIR}" @@ -184,10 +158,7 @@ do done term() { - if [ "${USE_QCOW2}" = "1" ]; then - log "Unloading image" - unload_qimage - fi + true; #TODO: Cleanup } trap term EXIT INT TERM @@ -270,18 +241,6 @@ source "${SCRIPT_DIR}/common" # shellcheck source=scripts/dependencies_check source "${SCRIPT_DIR}/dependencies_check" -export NO_PRERUN_QCOW2="${NO_PRERUN_QCOW2:-1}" -export USE_QCOW2="${USE_QCOW2:-0}" -export BASE_QCOW2_SIZE=${BASE_QCOW2_SIZE:-12G} -source "${SCRIPT_DIR}/qcow2_handling" -if [ "${USE_QCOW2}" = "1" ]; then - NO_PRERUN_QCOW2=1 -else - NO_PRERUN_QCOW2=0 -fi - -export NO_PRERUN_QCOW2="${NO_PRERUN_QCOW2:-1}" - if [ "$SETFCAP" != "1" ]; then export CAPSH_ARG="--drop=cap_setfcap" fi @@ -336,98 +295,22 @@ for EXPORT_DIR in ${EXPORT_DIRS}; do # shellcheck source=/dev/null source "${EXPORT_DIR}/EXPORT_IMAGE" EXPORT_ROOTFS_DIR=${WORK_DIR}/$(basename "${EXPORT_DIR}")/rootfs - if [ "${USE_QCOW2}" = "1" ]; then - USE_QCOW2=0 - EXPORT_NAME="${IMG_FILENAME}${IMG_SUFFIX}" - echo "------------------------------------------------------------------------" - echo "Running export stage for ${EXPORT_NAME}" - rm -f "${WORK_DIR}/export-image/${EXPORT_NAME}.img" || true - rm -f "${WORK_DIR}/export-image/${EXPORT_NAME}.qcow2" || true - rm -f "${WORK_DIR}/${EXPORT_NAME}.img" || true - rm -f "${WORK_DIR}/${EXPORT_NAME}.qcow2" || true - EXPORT_STAGE=$(basename "${EXPORT_DIR}") - for s in $STAGE_LIST; do - TMP_LIST=${TMP_LIST:+$TMP_LIST }$(basename "${s}") - done - FIRST_STAGE=${TMP_LIST%% *} - FIRST_IMAGE="image-${FIRST_STAGE}.qcow2" - - pushd "${WORK_DIR}" > /dev/null - echo "Creating new base "${EXPORT_NAME}.qcow2" from ${FIRST_IMAGE}" - cp "./${FIRST_IMAGE}" "${EXPORT_NAME}.qcow2" - - ARR=($TMP_LIST) - # rebase stage images to new export base - for CURR_STAGE in "${ARR[@]}"; do - if [ "${CURR_STAGE}" = "${FIRST_STAGE}" ]; then - PREV_IMG="${EXPORT_NAME}" - continue - fi - echo "Rebasing image-${CURR_STAGE}.qcow2 onto ${PREV_IMG}.qcow2" - qemu-img rebase -f qcow2 -u -b ${PREV_IMG}.qcow2 image-${CURR_STAGE}.qcow2 - if [ "${CURR_STAGE}" = "${EXPORT_STAGE}" ]; then - break - fi - PREV_IMG="image-${CURR_STAGE}" - done - - # commit current export stage into base export image - echo "Committing image-${EXPORT_STAGE}.qcow2 to ${EXPORT_NAME}.qcow2" - qemu-img commit -f qcow2 -p -b "${EXPORT_NAME}.qcow2" image-${EXPORT_STAGE}.qcow2 - - # rebase stage images back to original first stage for easy re-run - for CURR_STAGE in "${ARR[@]}"; do - if [ "${CURR_STAGE}" = "${FIRST_STAGE}" ]; then - PREV_IMG="image-${CURR_STAGE}" - continue - fi - echo "Rebasing back image-${CURR_STAGE}.qcow2 onto ${PREV_IMG}.qcow2" - qemu-img rebase -f qcow2 -u -b ${PREV_IMG}.qcow2 image-${CURR_STAGE}.qcow2 - if [ "${CURR_STAGE}" = "${EXPORT_STAGE}" ]; then - break - fi - PREV_IMG="image-${CURR_STAGE}" - done - popd > /dev/null - - mkdir -p "${WORK_DIR}/export-image/rootfs" - mv "${WORK_DIR}/${EXPORT_NAME}.qcow2" "${WORK_DIR}/export-image/" - echo "Mounting image ${WORK_DIR}/export-image/${EXPORT_NAME}.qcow2 to rootfs ${WORK_DIR}/export-image/rootfs" - mount_qimage "${WORK_DIR}/export-image/${EXPORT_NAME}.qcow2" "${WORK_DIR}/export-image/rootfs" - - CLEAN=0 - run_stage - CLEAN=1 - USE_QCOW2=1 - - else - run_stage - fi + run_stage if [ "${USE_QEMU}" != "1" ]; then if [ -e "${EXPORT_DIR}/EXPORT_NOOBS" ]; then # shellcheck source=/dev/null source "${EXPORT_DIR}/EXPORT_NOOBS" STAGE_DIR="${BASE_DIR}/export-noobs" - if [ "${USE_QCOW2}" = "1" ]; then - USE_QCOW2=0 - run_stage - USE_QCOW2=1 - else - run_stage - fi + run_stage fi fi done -if [ -x postrun.sh ]; then +if [ -x "${BASE_DIR}/postrun.sh" ]; then log "Begin postrun.sh" cd "${BASE_DIR}" ./postrun.sh log "End postrun.sh" fi -if [ "${USE_QCOW2}" = "1" ]; then - unload_qimage -fi - log "End ${BASE_DIR}" diff --git a/depends b/depends index db88171..10fcdd4 100644 --- a/depends +++ b/depends @@ -17,7 +17,5 @@ file git lsmod:kmod bc -qemu-nbd:qemu-utils -kpartx gpg pigz diff --git a/export-image/04-set-partuuid/00-run.sh b/export-image/04-set-partuuid/00-run.sh index 2694295..99500f3 100755 --- a/export-image/04-set-partuuid/00-run.sh +++ b/export-image/04-set-partuuid/00-run.sh @@ -1,17 +1,13 @@ #!/bin/bash -e -if [ "${NO_PRERUN_QCOW2}" = "0" ]; then +IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" - IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" +IMGID="$(dd if="${IMG_FILE}" skip=440 bs=1 count=4 2>/dev/null | xxd -e | cut -f 2 -d' ')" - IMGID="$(dd if="${IMG_FILE}" skip=440 bs=1 count=4 2>/dev/null | xxd -e | cut -f 2 -d' ')" +BOOT_PARTUUID="${IMGID}-01" +ROOT_PARTUUID="${IMGID}-02" - BOOT_PARTUUID="${IMGID}-01" - ROOT_PARTUUID="${IMGID}-02" - - sed -i "s/BOOTDEV/PARTUUID=${BOOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab" - sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab" - - sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/boot/firmware/cmdline.txt" -fi +sed -i "s/BOOTDEV/PARTUUID=${BOOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab" +sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/etc/fstab" +sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${ROOTFS_DIR}/boot/firmware/cmdline.txt" diff --git a/export-image/05-finalise/01-run.sh b/export-image/05-finalise/01-run.sh index 3b56b20..8310e87 100755 --- a/export-image/05-finalise/01-run.sh +++ b/export-image/05-finalise/01-run.sh @@ -83,25 +83,18 @@ cp "$ROOTFS_DIR/etc/rpi-issue" "$INFO_FILE" dpkg -l --root "$ROOTFS_DIR" } >> "$INFO_FILE" +ROOT_DEV="$(mount | grep "${ROOTFS_DIR} " | cut -f1 -d' ')" + +unmount "${ROOTFS_DIR}" +zerofree "${ROOT_DEV}" + +unmount_image "${IMG_FILE}" + mkdir -p "${DEPLOY_DIR}" rm -f "${DEPLOY_DIR}/${ARCHIVE_FILENAME}${IMG_SUFFIX}.*" rm -f "${DEPLOY_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" -mv "$INFO_FILE" "$DEPLOY_DIR/" - -if [ "${USE_QCOW2}" = "0" ] && [ "${NO_PRERUN_QCOW2}" = "0" ]; then - ROOT_DEV="$(mount | grep "${ROOTFS_DIR} " | cut -f1 -d' ')" - - unmount "${ROOTFS_DIR}" - zerofree "${ROOT_DEV}" - - unmount_image "${IMG_FILE}" -else - unload_qimage - make_bootable_image "${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.qcow2" "$IMG_FILE" -fi - case "${DEPLOY_COMPRESSION}" in zip) pushd "${STAGE_WORK_DIR}" > /dev/null @@ -121,3 +114,5 @@ none | *) cp "$IMG_FILE" "$DEPLOY_DIR/" ;; esac + +cp "$INFO_FILE" "$DEPLOY_DIR/" diff --git a/export-image/prerun.sh b/export-image/prerun.sh index d042c6a..b95fe36 100755 --- a/export-image/prerun.sh +++ b/export-image/prerun.sh @@ -1,68 +1,66 @@ #!/bin/bash -e -if [ "${NO_PRERUN_QCOW2}" = "0" ]; then - IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" +IMG_FILE="${STAGE_WORK_DIR}/${IMG_FILENAME}${IMG_SUFFIX}.img" - unmount_image "${IMG_FILE}" +unmount_image "${IMG_FILE}" - rm -f "${IMG_FILE}" +rm -f "${IMG_FILE}" - rm -rf "${ROOTFS_DIR}" - mkdir -p "${ROOTFS_DIR}" +rm -rf "${ROOTFS_DIR}" +mkdir -p "${ROOTFS_DIR}" - BOOT_SIZE="$((512 * 1024 * 1024))" - ROOT_SIZE=$(du --apparent-size -s "${EXPORT_ROOTFS_DIR}" --exclude var/cache/apt/archives --exclude boot/firmware --block-size=1 | cut -f 1) +BOOT_SIZE="$((512 * 1024 * 1024))" +ROOT_SIZE=$(du --apparent-size -s "${EXPORT_ROOTFS_DIR}" --exclude var/cache/apt/archives --exclude boot/firmware --block-size=1 | cut -f 1) - # All partition sizes and starts will be aligned to this size - ALIGN="$((4 * 1024 * 1024))" - # Add this much space to the calculated file size. This allows for - # some overhead (since actual space usage is usually rounded up to the - # filesystem block size) and gives some free space on the resulting - # image. - ROOT_MARGIN="$(echo "($ROOT_SIZE * 0.2 + 200 * 1024 * 1024) / 1" | bc)" +# All partition sizes and starts will be aligned to this size +ALIGN="$((4 * 1024 * 1024))" +# Add this much space to the calculated file size. This allows for +# some overhead (since actual space usage is usually rounded up to the +# filesystem block size) and gives some free space on the resulting +# image. +ROOT_MARGIN="$(echo "($ROOT_SIZE * 0.2 + 200 * 1024 * 1024) / 1" | bc)" - BOOT_PART_START=$((ALIGN)) - BOOT_PART_SIZE=$(((BOOT_SIZE + ALIGN - 1) / ALIGN * ALIGN)) - ROOT_PART_START=$((BOOT_PART_START + BOOT_PART_SIZE)) - ROOT_PART_SIZE=$(((ROOT_SIZE + ROOT_MARGIN + ALIGN - 1) / ALIGN * ALIGN)) - IMG_SIZE=$((BOOT_PART_START + BOOT_PART_SIZE + ROOT_PART_SIZE)) +BOOT_PART_START=$((ALIGN)) +BOOT_PART_SIZE=$(((BOOT_SIZE + ALIGN - 1) / ALIGN * ALIGN)) +ROOT_PART_START=$((BOOT_PART_START + BOOT_PART_SIZE)) +ROOT_PART_SIZE=$(((ROOT_SIZE + ROOT_MARGIN + ALIGN - 1) / ALIGN * ALIGN)) +IMG_SIZE=$((BOOT_PART_START + BOOT_PART_SIZE + ROOT_PART_SIZE)) - truncate -s "${IMG_SIZE}" "${IMG_FILE}" +truncate -s "${IMG_SIZE}" "${IMG_FILE}" - parted --script "${IMG_FILE}" mklabel msdos - parted --script "${IMG_FILE}" unit B mkpart primary fat32 "${BOOT_PART_START}" "$((BOOT_PART_START + BOOT_PART_SIZE - 1))" - parted --script "${IMG_FILE}" unit B mkpart primary ext4 "${ROOT_PART_START}" "$((ROOT_PART_START + ROOT_PART_SIZE - 1))" +parted --script "${IMG_FILE}" mklabel msdos +parted --script "${IMG_FILE}" unit B mkpart primary fat32 "${BOOT_PART_START}" "$((BOOT_PART_START + BOOT_PART_SIZE - 1))" +parted --script "${IMG_FILE}" unit B mkpart primary ext4 "${ROOT_PART_START}" "$((ROOT_PART_START + ROOT_PART_SIZE - 1))" - echo "Creating loop device..." - cnt=0 - until ensure_next_loopdev && LOOP_DEV="$(losetup --show --find --partscan "$IMG_FILE")"; do - if [ $cnt -lt 5 ]; then - cnt=$((cnt + 1)) - echo "Error in losetup. Retrying..." - sleep 5 - else - echo "ERROR: losetup failed; exiting" - exit 1 - fi - done - - ensure_loopdev_partitions "$LOOP_DEV" - BOOT_DEV="${LOOP_DEV}p1" - ROOT_DEV="${LOOP_DEV}p2" - - ROOT_FEATURES="^huge_file" - for FEATURE in 64bit; do - if grep -q "$FEATURE" /etc/mke2fs.conf; then - ROOT_FEATURES="^$FEATURE,$ROOT_FEATURES" +echo "Creating loop device..." +cnt=0 +until ensure_next_loopdev && LOOP_DEV="$(losetup --show --find --partscan "$IMG_FILE")"; do + if [ $cnt -lt 5 ]; then + cnt=$((cnt + 1)) + echo "Error in losetup. Retrying..." + sleep 5 + else + echo "ERROR: losetup failed; exiting" + exit 1 fi - done - mkdosfs -n bootfs -F 32 -s 4 -v "$BOOT_DEV" > /dev/null - mkfs.ext4 -L rootfs -O "$ROOT_FEATURES" "$ROOT_DEV" > /dev/null +done - mount -v "$ROOT_DEV" "${ROOTFS_DIR}" -t ext4 - mkdir -p "${ROOTFS_DIR}/boot/firmware" - mount -v "$BOOT_DEV" "${ROOTFS_DIR}/boot/firmware" -t vfat +ensure_loopdev_partitions "$LOOP_DEV" +BOOT_DEV="${LOOP_DEV}p1" +ROOT_DEV="${LOOP_DEV}p2" - rsync -aHAXx --exclude /var/cache/apt/archives --exclude /boot/firmware "${EXPORT_ROOTFS_DIR}/" "${ROOTFS_DIR}/" - rsync -rtx "${EXPORT_ROOTFS_DIR}/boot/firmware/" "${ROOTFS_DIR}/boot/firmware/" +ROOT_FEATURES="^huge_file" +for FEATURE in 64bit; do +if grep -q "$FEATURE" /etc/mke2fs.conf; then + ROOT_FEATURES="^$FEATURE,$ROOT_FEATURES" fi +done +mkdosfs -n bootfs -F 32 -s 4 -v "$BOOT_DEV" > /dev/null +mkfs.ext4 -L rootfs -O "$ROOT_FEATURES" "$ROOT_DEV" > /dev/null + +mount -v "$ROOT_DEV" "${ROOTFS_DIR}" -t ext4 +mkdir -p "${ROOTFS_DIR}/boot/firmware" +mount -v "$BOOT_DEV" "${ROOTFS_DIR}/boot/firmware" -t vfat + +rsync -aHAXx --exclude /var/cache/apt/archives --exclude /boot/firmware "${EXPORT_ROOTFS_DIR}/" "${ROOTFS_DIR}/" +rsync -rtx "${EXPORT_ROOTFS_DIR}/boot/firmware/" "${ROOTFS_DIR}/boot/firmware/" diff --git a/export-noobs/00-release/00-run.sh b/export-noobs/00-release/00-run.sh index bfaea9f..513cbfd 100755 --- a/export-noobs/00-release/00-run.sh +++ b/export-noobs/00-release/00-run.sh @@ -41,8 +41,4 @@ sed "${NOOBS_DIR}/os.json" -i -e "s|KERNEL|$(cat "${STAGE_WORK_DIR}/kernel_versi sed "${NOOBS_DIR}/release_notes.txt" -i -e "s|UNRELEASED|${IMG_DATE}|" -if [ "${USE_QCOW2}" = "1" ]; then - mv "${NOOBS_DIR}" "${DEPLOY_DIR}/" -else - cp -a "${NOOBS_DIR}" "${DEPLOY_DIR}/" -fi +cp -a "${NOOBS_DIR}" "${DEPLOY_DIR}/" diff --git a/export-noobs/prerun.sh b/export-noobs/prerun.sh index 80cbb8e..8e55c9e 100755 --- a/export-noobs/prerun.sh +++ b/export-noobs/prerun.sh @@ -41,8 +41,4 @@ bsdtar --numeric-owner --format gnutar -C "${STAGE_WORK_DIR}/rootfs/boot" -cpf - umount "${STAGE_WORK_DIR}/rootfs/boot" bsdtar --numeric-owner --format gnutar -C "${STAGE_WORK_DIR}/rootfs" --one-file-system -cpf - . | xz -T0 > "${NOOBS_DIR}/root.tar.xz" -if [ "${USE_QCOW2}" = "1" ]; then - rm "$ROOTFS_DIR/etc/systemd/system/multi-user.target.wants/apply_noobs_os_config.service" -fi - unmount_image "${IMG_FILE}" diff --git a/imagetool.sh b/imagetool.sh deleted file mode 100755 index 002b50b..0000000 --- a/imagetool.sh +++ /dev/null @@ -1,114 +0,0 @@ -#!/bin/bash - -if [ "$(id -u)" != "0" ]; then - echo "Please run as root" 1>&2 - exit 1 -fi - -progname=$(basename $0) - -function usage() -{ - cat << HEREDOC - -Usage: - Mount Image : $progname [--mount] [--image-name ] [--mount-point ] - Umount Image: $progname [--umount] [--mount-point ] - Cleanup NBD : $progname [--cleanup] - - arguments: - -h, --help show this help message and exit - -c, --cleanup cleanup orphaned device mappings - -m, --mount mount image - -u, --umount umount image - -i, --image-name path to qcow2 image - -p, --mount-point mount point for image - - This tool will use /dev/nbd1 as default for mounting an image. If you want to use another device, execute like this: - NBD_DEV=/dev/nbd2 ./$progname --mount --image-name --mount-point - -HEREDOC -} - -MOUNT=0 -UMOUNT=0 -IMAGE="" -MOUNTPOINT="" - -nbd_cleanup() { - DEVS="$(lsblk | grep nbd | grep disk | cut -d" " -f1)" - if [ ! -z "${DEVS}" ]; then - for d in $DEVS; do - if [ ! -z "${d}" ]; then - QDEV="$(ps xa | grep $d | grep -v grep)" - if [ -z "${QDEV}" ]; then - kpartx -d /dev/$d && echo "Unconnected device map removed: /dev/$d" - fi - fi - done - fi -} - -# As long as there is at least one more argument, keep looping -while [[ $# -gt 0 ]]; do - key="$1" - case "$key" in - -h|--help) - usage - exit - ;; - -c|--cleanup) - nbd_cleanup - ;; - -m|--mount) - MOUNT=1 - ;; - -u|--umount) - UMOUNT=1 - ;; - -i|--image-name) - shift - IMAGE="$1" - ;; - -p|--mount-point) - shift - MOUNTPOINT="$1" - ;; - *) - echo "Unknown option '$key'" - usage - exit - ;; - esac - # Shift after checking all the cases to get the next option - shift -done - -if [ "${MOUNT}" = "1" ] && [ "${UMOUNT}" = "1" ]; then - usage - echo "Concurrent mount options not possible." - exit -fi - -if [ "${MOUNT}" = "1" ] && ([ -z "${IMAGE}" ] || [ -z "${MOUNTPOINT}" ]); then - usage - echo "Can not mount image. Image path and/or mount point missing." - exit -fi - -if [ "${UMOUNT}" = "1" ] && [ -z "${MOUNTPOINT}" ]; then - usage - echo "Can not umount. Mount point parameter missing." - exit -fi - -export NBD_DEV="${NBD_DEV:-/dev/nbd1}" -export MAP_BOOT_DEV=/dev/mapper/nbd1p1 -export MAP_ROOT_DEV=/dev/mapper/nbd1p2 -source scripts/qcow2_handling - -if [ "${MOUNT}" = "1" ]; then - mount_qimage "${IMAGE}" "${MOUNTPOINT}" -elif [ "${UMOUNT}" = "1" ]; then - umount_qimage "${MOUNTPOINT}" -fi diff --git a/scripts/qcow2_handling b/scripts/qcow2_handling deleted file mode 100644 index 66708e7..0000000 --- a/scripts/qcow2_handling +++ /dev/null @@ -1,256 +0,0 @@ -#!/bin/bash - -# QCOW2 Routines - -export CURRENT_IMAGE -export CURRENT_MOUNTPOINT - -export NBD_DEV -export MAP_BOOT_DEV -export MAP_ROOT_DEV - -# set in build.sh -# should be fairly enough for the beginning -# overwrite here by uncommenting following lines -# BASE_QCOW2_SIZE=12G - -# find and initialize free block device nodes -init_nbd() { - modprobe nbd max_part=16 - if [ -z "${NBD_DEV}" ]; then - for x in /sys/class/block/nbd* ; do - S=`cat $x/size` - if [ "$S" == "0" ] ; then - NBD_DEV=/dev/$(basename $x) - MAP_BOOT_DEV=/dev/mapper/$(basename $x)p1 - MAP_ROOT_DEV=/dev/mapper/$(basename $x)p2 - break - fi - done - fi -} -export -f init_nbd - -# connect image to block device -connect_blkdev() { - init_nbd - qemu-nbd --discard=unmap -c $NBD_DEV "$1" - sync - kpartx -as $NBD_DEV - sync - CURRENT_IMAGE="$1" -} -export -f connect_blkdev - -# disconnect image from block device -disconnect_blkdev() { - kpartx -d $NBD_DEV - qemu-nbd -d $NBD_DEV - NBD_DEV= - MAP_BOOT_DEV= - MAP_ROOT_DEV= - CURRENT_IMAGE= -} -export -f disconnect_blkdev - -# mount qcow2 image: mount_image -mount_qimage() { - connect_blkdev "$1" - mount -v -t ext4 $MAP_ROOT_DEV "$2" - mkdir -p "${ROOTFS_DIR}/boot" - mount -v -t vfat $MAP_BOOT_DEV "$2/boot" - CURRENT_MOUNTPOINT="$2" -} -export -f mount_qimage - -# umount qcow2 image: umount_image -umount_qimage() { - sync - #umount "$1/boot" - while mount | grep -q "$1"; do - local LOCS - LOCS=$(mount | grep "$1" | cut -f 3 -d ' ' | sort -r) - for loc in $LOCS; do - echo "$loc" - while mountpoint -q "$loc" && ! umount "$loc"; do - sleep 0.1 - done - done - done - CURRENT_MOUNTPOINT= - disconnect_blkdev -} -export -f umount_qimage - -# create base image / backing image / mount image -load_qimage() { - if [ -z "${CURRENT_MOUNTPOINT}" ]; then - if [ ! -d "${ROOTFS_DIR}" ]; then - mkdir -p "${ROOTFS_DIR}"; - fi - - if [ "${CLEAN}" = "1" ] && [ -f "${WORK_DIR}/image-${STAGE}.qcow2" ]; then - rm -f "${WORK_DIR}/image-${STAGE}.qcow2"; - fi - - if [ ! -f "${WORK_DIR}/image-${STAGE}.qcow2" ]; then - pushd ${WORK_DIR} > /dev/null - init_nbd - if [ -z "${PREV_STAGE}" ]; then - echo "Creating base image: image-${STAGE}.qcow2" - # -o preallocation=falloc - qemu-img create -f qcow2 image-${STAGE}.qcow2 $BASE_QCOW2_SIZE - sync - qemu-nbd --discard=unmap -c $NBD_DEV image-${STAGE}.qcow2 - sync - sfdisk $NBD_DEV << EOF -4MiB,250MiB,c,* -254MiB,,83; -EOF - sync - kpartx -as $NBD_DEV - mkdosfs -n boot -F 32 -s 4 -v $MAP_BOOT_DEV - mkfs.ext4 -L rootfs -O "^huge_file,^64bit" $MAP_ROOT_DEV - sync - else - if [ ! -f "${WORK_DIR}/image-${PREV_STAGE}.qcow2" ]; then - exit 1; - fi - echo "Creating backing image: image-${STAGE}.qcow2 <- ${WORK_DIR}/image-${PREV_STAGE}.qcow2" - qemu-img create -f qcow2 \ - -o backing_file=${WORK_DIR}/image-${PREV_STAGE}.qcow2 \ - ${WORK_DIR}/image-${STAGE}.qcow2 - sync - qemu-nbd --discard=unmap -c $NBD_DEV image-${STAGE}.qcow2 - sync - kpartx -as $NBD_DEV - fi - - mount -v -t ext4 $MAP_ROOT_DEV "${ROOTFS_DIR}" - mkdir -p "${ROOTFS_DIR}/boot" - mount -v -t vfat $MAP_BOOT_DEV "${ROOTFS_DIR}/boot" - CURRENT_IMAGE=${WORK_DIR}/image-${STAGE}.qcow2 - CURRENT_MOUNTPOINT=${ROOTFS_DIR} - popd > /dev/null - else - mount_qimage "${WORK_DIR}/image-${STAGE}.qcow2" "${ROOTFS_DIR}" - fi - echo "Current image in use: ${CURRENT_IMAGE} (MP: ${CURRENT_MOUNTPOINT})" - fi -} -export -f load_qimage - -# umount current image and refresh mount point env var -unload_qimage() { - if [ ! -z "${CURRENT_MOUNTPOINT}" ]; then - fstrim -v "${CURRENT_MOUNTPOINT}" || true - umount_qimage "${CURRENT_MOUNTPOINT}" - fi -} -export -f unload_qimage - -# based on: https://github.com/SirLagz/RaspberryPi-ImgAutoSizer -# helper function for make_bootable_image, do not call directly -function resize_qcow2() { - if [ -z "$CALL_FROM_MBI" ]; then - echo "resize_qcow2: cannot be called directly, use make_bootable_image instead" - return 1 - fi - - # ROOT_MARGIN=$((800*1024*1024)) - ROOT_MARGIN=$((1*1024*1024)) - PARTED_OUT=`parted -s -m "$NBD_DEV" unit B print` - PART_NO=`echo "$PARTED_OUT" | grep ext4 | awk -F: ' { print $1 } '` - PART_START=`echo "$PARTED_OUT" | grep ext4 | awk -F: ' { print substr($2,1,length($2)-1) } '` - - e2fsck -y -f $MAP_ROOT_DEV || true - - DATA_SIZE=`resize2fs -P $MAP_ROOT_DEV | awk -F': ' ' { print $2 } '` - BLOCK_SIZE=$(dumpe2fs -h $MAP_ROOT_DEV | grep 'Block size' | awk -F': ' ' { print $2 }') - BLOCK_SIZE=${BLOCK_SIZE// /} - - let DATA_SIZE=$DATA_SIZE+$ROOT_MARGIN/$BLOCK_SIZE - resize2fs -p $MAP_ROOT_DEV $DATA_SIZE - sleep 1 - - let PART_NEW_SIZE=$DATA_SIZE*$BLOCK_SIZE - let PART_NEW_END=$PART_START+$PART_NEW_SIZE - ACT1=`parted -s "$NBD_DEV" rm 2` - ACT2=`parted -s "$NBD_DEV" unit B mkpart primary $PART_START $PART_NEW_END` - NEW_IMG_SIZE=`parted -s -m "$NBD_DEV" unit B print free | tail -1 | awk -F: ' { print substr($2,1,length($2)-1) } '` -} -export -f resize_qcow2 - -# create raw img from qcow2: make_bootable_image -function make_bootable_image() { - - EXPORT_QCOW2="$1" - EXPORT_IMAGE="$2" - - echo "Connect block device to source qcow2" - connect_blkdev "${EXPORT_QCOW2}" - - echo "Resize fs and partition" - CALL_FROM_MBI=1 - resize_qcow2 - sync - CALL_FROM_MBI= - - echo "Disconnect block device" - disconnect_blkdev - - if [ -z "$NEW_IMG_SIZE" ]; then - echo "NEW_IMG_SIZE could not be calculated, cannot process image. Exit." - exit 1 - fi - - echo "Shrinking qcow2 image" - qemu-img resize --shrink "${EXPORT_QCOW2}" $NEW_IMG_SIZE - sync - - echo "Convert qcow2 to raw image" - qemu-img convert -f qcow2 -O raw "${EXPORT_QCOW2}" "${EXPORT_IMAGE}" - sync - - echo "Get PARTUUIDs from image" - IMGID="$(blkid -o value -s PTUUID "${EXPORT_IMAGE}")" - - BOOT_PARTUUID="${IMGID}-01" - echo "Boot: $BOOT_PARTUUID" - ROOT_PARTUUID="${IMGID}-02" - echo "Root: $ROOT_PARTUUID" - - echo "Mount image" - MOUNTROOT=${WORK_DIR}/tmpimage - mkdir -p $MOUNTROOT - - MOUNTPT=$MOUNTROOT - PARTITION=2 - mount "${EXPORT_IMAGE}" "$MOUNTPT" -o loop,offset=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*start=[ ]*//' | sed 's/,.*//'` * 512 ],sizelimit=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*size=[ ]*//' | sed 's/,.*//'` * 512 ] || exit 1 - - MOUNTPT=$MOUNTROOT/boot - PARTITION=1 - mount "${EXPORT_IMAGE}" "$MOUNTPT" -o loop,offset=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*start=[ ]*//' | sed 's/,.*//'` * 512 ],sizelimit=$[ `/sbin/sfdisk -d "${EXPORT_IMAGE}" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*size=[ ]*//' | sed 's/,.*//'` * 512 ] || exit 1 - - if [ ! -d "${MOUNTROOT}/root" ]; then - echo "Image damaged or not mounted. Exit." - exit 1 - fi - - echo "Setup PARTUUIDs" - if [ ! -z "$BOOT_PARTUUID" ] && [ ! -z "$ROOT_PARTUUID" ]; then - echo "Set UUIDs to make it bootable" - sed -i "s/BOOTDEV/PARTUUID=${BOOT_PARTUUID}/" "${MOUNTROOT}/etc/fstab" - sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${MOUNTROOT}/etc/fstab" - sed -i "s/ROOTDEV/PARTUUID=${ROOT_PARTUUID}/" "${MOUNTROOT}/boot/cmdline.txt" - fi - - echo "Umount image" - sync - umount "${MOUNTROOT}/boot" || exit 1 - umount "${MOUNTROOT}" || exit 1 - - echo "Remove qcow2 export image" - rm -f "${EXPORT_QCOW2}" -} -export -f make_bootable_image diff --git a/stage0/prerun.sh b/stage0/prerun.sh index 0252071..5f0bd2c 100755 --- a/stage0/prerun.sh +++ b/stage0/prerun.sh @@ -5,6 +5,6 @@ if [ "$RELEASE" != "bookworm" ]; then echo " Please check the relevant README.md section." fi -if [ ! -d "${ROOTFS_DIR}" ] || [ "${USE_QCOW2}" = "1" ]; then +if [ ! -d "${ROOTFS_DIR}" ]; then bootstrap ${RELEASE} "${ROOTFS_DIR}" http://raspbian.raspberrypi.com/raspbian/ fi From 94cbf476a57bfc7483d1ce08f2ddbe28e4949ffe Mon Sep 17 00:00:00 2001 From: Serge Schneider Date: Wed, 14 Feb 2024 13:25:42 +0000 Subject: [PATCH 3/5] Create .gitlab-ci.yml file --- .gitlab-ci.yml | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 .gitlab-ci.yml diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000..32f2688 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,4 @@ +include: + - project: serge/pi-gen + ref: ci + file: 'pi-gen.yml' From c029146a132b613f538b1368fd0706cc93fd88b9 Mon Sep 17 00:00:00 2001 From: Serge Schneider Date: Thu, 15 Feb 2024 21:27:39 +0000 Subject: [PATCH 4/5] ensure_loopdev_partitions: let udev settle --- scripts/common | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/common b/scripts/common index cda5551..6150f2b 100644 --- a/scripts/common +++ b/scripts/common @@ -122,5 +122,7 @@ ensure_loopdev_partitions() { mknod "/dev/$partition" b "${majmin%:*}" "${majmin#*:}" fi done + command -v udevadm >/dev/null 2>&1 || return 0 + udevadm settle 10 } export -f ensure_loopdev_partitions From 82e5642ad16453f47fbbaeb11f8b63d4ccf17d81 Mon Sep 17 00:00:00 2001 From: Christian Kuhtz Date: Mon, 26 Feb 2024 06:12:43 -0800 Subject: [PATCH 5/5] README.md: Add xxd dependency (#750) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3529770..136e892 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ To install the required dependencies for `pi-gen` you should run: ```bash apt-get install coreutils quilt parted qemu-user-static debootstrap zerofree zip \ dosfstools libarchive-tools libcap2-bin grep rsync xz-utils file git curl bc \ -gpg pigz +gpg pigz xxd ``` The file `depends` contains a list of tools needed. The format of this