raspberrypi: Proper A/B kernel Updates and Atomic Boot Partition Updates #12
Comments
|
Hi @om26er, You should be able to achieve this using: Btw meta-mender-raspberrypi for Mender A/B updates could be used as a reference example for such feature. Best regards, Leon |
|
Hi, On raspberry pi, the kernel and devicetree is stored on boot partition, which is not included in update process. The update process using rauc only updates rootfs. So even if you include kernel image and devicetree files in the rootfs, rasperry won't use it after update. Maybe it is a good first step which could be later on extended about copying these files to the boot partition, maybe using a post update hook. But it is not a recommended way because the update is not atomic. So if there is a problem during update, the raspberry pi will not boot again at all. Also there is no fallback to previously known good kernel image. What I think is the best way is to implement boot-mbr-switch method described in the rauc documentation here, which consists of two boot partitions stored on the card and switching between them by manipulating partition table. Which is atomic and therefore more safe. This way, we can also implement a fallback in case of boot fail which is probably the most important reason. Because otherwise the only change of fixing the pi is to reflash the sd card or at least the boot partition. The setup should not be hard, the only thing I have a problem with is where to store the u-boot environment file. This file is currently on the /boot partition, so we need to move it out, somewhere to the bare unallocated space of the sd card. Because we will have two different boot partitions. U-boot can also save two copies of this file to be sure that we won't use a broken one on boot. |
|
I found some useful information on this page as well https://github.com/mendersoftware/meta-mender/tree/master/meta-mender-raspberrypi#meta-mender-raspberrypi It seems mender only supports kernel updates (by putting the kernel on the rootfs) and DTB updates are not supported. For our project, being able to update the kernel and DTB is very important so we'd explore more and see what could be done. |
|
maybe if @timemaster5 can find a solution to this stuff, it'll make life easier for us ;-) |
|
Problem with me is that I don't have any time right now. I am working on a project where I must deliver something under the deadline. So for now I end up with extensive research on this topic, and I hope someone else will be able to take over and do the implementation. If not, I will definitely get back at some point, because I would like this feature implemented too, but currently can't say when. From what I know, I am pretty sure the required changes are:
Region start values, corresponding partition sizes and alignments changes in wic file, and in the patch could be based on the suggested layout from the rauc documentation. And that's probably it. Sorry I can't help more at this point |
|
@timemaster5 thank you for the valuable feedback and sharing all these details. I believe this will be very useful in future. Thanks, |
|
You are welcome, I hope I will have time for the actual integration soon :) Anyway, just to let you know. I just successfully updated the device in a remote location in a similar way to what was originally suggested @leon-anavi. I uploaded a rootfs image, did the update of rootfs image and then manually copied over a new content of /boot partition. Even though it is not my favourite way, it went ok and I rebooted to the new kernel and device tree. But there are no guarantees. So I would consider this way as best-effort only. |
|
@timemaster5 thanks for your pointers. This is on my TODO list for this week as well. I got some help from the RAUC upstream as well and they say they have successfully update RPi boot partition as well. |
|
Oh great, if you could share some other information on this topic from other channels it would be appreciated |
|
I have this stuff mostly implemented. One thing I need to mention is that we only need a single entry for bootloader slot in the system.conf The main thing to figure out was to create a vfat image based on the files listed in ${IMAGE_BOOT_FILES} variable. |
|
So the only missing thing for me currently is that the |
|
Here is the fdisk output. Before Flash the image After flashing |
Thank you, could you share your .wks file and maybe some other info on how to build that vfat image? |
This seems to me like the patch has not been applied. Maybe Out of the curiosity, what RaspberryPi do you use? For example, I use Compute module 3 with custom defconfig for the u-boot. So this patch wouldn't work me out of the box. |
And this is really cool :) Also, your region starts and offset values looks good to me. They don't seem to collide with the u-boot env file defined in the patch. |
|
Here is what I did to write a vfat image file, these functions are called through |
|
@timemaster5 you are right, it seems the patch wasn't really applied. Once I did apply the patch (with a minor fix). I no longer see |
|
Also here is how my patch to u-boot looks like, maybe something wrong with that ? |
|
Also tried with the original patch (with minor fix) and that also produced the same result |
|
So it seems I needed to change |
|
@om26er have a look how |
you're too fast, I will try to respond to the rest during the day :) But this one should be easy, instead of With this change, rauc should be able to work. |
|
Anyway, I need to change my statement above, and maybe we need to change the title of this issue. According to @om26er 's findings, this is not an A/B deployment as I thought. This is "only" an atomic update of /boot partition. This is not bad as long as we are sure the kernel and rootfs combination works together after the update. It "only" ensures the update doesn't crash the system on fail (power outage, oomkiller, ...). So It should be good enough for production OTA upgrades because no machine should receive an untested update. If we want a real A/B deployment of /boot partition, we would need this bit, which already works for @om26er to atomically update u-boot only, and then let u-boot load the kernel and DT from either bootA or bootB slot. |
So you are re-creating boot partition as a file which you are delivering together with rootfs. |
|
@timemaster5 I am creating a rauc bundle manually by generating a manifest and putting that into the rootfs. The bootloader update is then applied afterwards as recommended here https://rauc.readthedocs.io/en/latest/advanced.html#considerations-when-updating-the-bootloader |
|
Thank you for pointing this out. I haven't seen this part of the documentation. I believe, that it is not applicable to our case. They are talking specifically about updating the bootloader (u-boot for us). Which is something totally different from what are we doing here. We are updating the whole boot partition which consists of the bootloader, kernel image and device tree plus some device-specific configuration. If there is a problem with any of those, the system will not boot up. In their case, they expect the system can boot even with an older version of the bootloader. The user is then to check whether the update was successful or not because it might be important for some other part of the update process or recovery. I've seen this many times - I had to upgrade a bootloader first, then I was able to update to the newest system, it is a quite common example, but not really related to the RaspberryPi ecosystem. |
I think there are multiple things to consider here. The /boot partition contains the follow at least
The raspberrypi expects much of the above in the "first" partition, so I am not sure if we can really do A/B redundancy separate from the updating /boot Mender loads the kernel from the rootfs, and we might be able to do that as well. But the dtb and other stuff basically has to stay on the first partition https://github.com/mendersoftware/meta-mender/tree/master/meta-mender-raspberrypi#meta-mender-raspberrypi Thoughts ? |
|
The latest raspberry pi EEPROM it seems support a way for A/B partitioning https://www.raspberrypi.org/forums/viewtopic.php?t=297354#p1791107 -- Using that approach could remove the need of u-boot all together. RAUC would have to be patched to support that though. |
Good to hear from you @om26er But with these recent findings of [REBOOT_FLAGS] (raspberrypi/linux@7576667) the situation is different and more optimistic, this could be a good solution for us really. Removal of uBoot probably depends on personal preferences, it could still be useful for some other tasks than switching rootfs. But I agree that for a basic idea it should not be needed with this new fw. |
|
I have Here is what I had to do
|
Note that, using tryboot technique, one could update the OS in the background and then tell the user to reboot using our script that calls |
|
hi @om26er, very nice possibility to update the boot partition !! 🎉 How does your rauc config look like and how does the boot slot detected? |
|
I am also interested in using the tryboot feature of the Raspberry bootloader for A/B switching instead of U-Boot. |
|
I'm very interested in this also. @om26er care to share some more details? |
|
Also interested :) |
|
I think I've found a solution myself. Its not road tested and has obvious flaws, but the basics seem to work and can update/switch root images. Aside from @om26er's custom WIC plugin above, I also wrote a custom bootloader backend (defined in [handlers] in system.conf) and a post-install script that handles the editing/copying/moving of config.txt/tryboot.txt Current shortcomings is that I'm not sure how to mark a slot good or bad, expect in the case where the system is correctly booting from tryboot.txt, which implies a good slot, and tryboot.txt then becomes config.txt. |
Add instructions how to move the Linux kernel to the rootfs on Raspberry Pi and to update with a RAUC bundle rauc#12 Signed-off-by: Leon Anavi <[email protected]>
|
Hi, I created a GitHub pull request which modifies meta-rauc-raspberrypi/README.rst and adds instructions how to move the Linux kernel to the rootfs partition and after that to update it as part of the RAUC bundle. Please note that the dtb is not updated in the described method. Please let me know if there are comments. After merging the GitHub pull request I am planning to close this issue. Best regards, Leon |
Add instructions how to move the Linux kernel to the rootfs on Raspberry Pi and to update with a RAUC bundle rauc#12 Signed-off-by: Leon Anavi <[email protected]>
|
For anyone interested, I can also share the implementation I had for RPI before this layer was created: https://github.com/ejoerns/meta-rauc-demo-wip/tree/gatesgarth-rpi/meta-rauc-raspberrypi IMHO having the kernel as part of the rootfs is the simplest solution anyway. Having the boot partition updatable in an atomic manner should be part of this layer, too. I remember having struggled with this, too. My solution back then was: https://github.com/ejoerns/meta-rauc-demo-wip/blob/gatesgarth-rpi/meta-rauc-raspberrypi/recipes-core/rauc/files/raspberrypi3/system.conf Maybe using a proper wic plugin for the boot partition generation is even better (as suggested above). Haven't used 'tryboot' back then but having this as a custom (or code-implemented?) bootloader backend would be interesting, too, I guess. It's fine for me, too, to close this Issue first of all and keep it in mind as a reference once someone wants to implement a solution in a PR. I'll modify the title (as already suggested) to better match the scope of the rest of the discussion. |
ejoerns
changed the title
|
Hello, for what it worth, I have an implementation here using tryboot and autobot.txt. I have a buildroot br2-external example and an oe layer is on its way. And also a c implementation in RAUC as a new bootchooser. |
|
The implementation is self-documented in the bash script. If anyone wants to have a look at it, reviews are welcome, especially about making a final implementation in RAUC. It can be discussed here, at least for now. That custom backend is very similar to EFI as there is no boot-attempt counter, and there is a "volatile" flag to switch A/B at next boot. RAUC status does not guess the status of the non-booted slot. It could use the presence of the tryboot.txt file. Also, I wonder if the persistent RAUC The kernel update is performed using a bundle post installation hook to save the device of the rootfs slot being updated, and to append the rootfs= to cmdline.txt after the boot slot update. See the commit. |
Currently we have been using
meta-rauc-communityand it works fine, the A/B partitioning scheme and OTA updates work as well (we use RAUC).Since there is only a single partition for
boot, I wonder how to implement OTA updates for the kernel ?The text was updated successfully, but these errors were encountered: