Kernel panic after update from 3.8.8 to 3.9.1

[VortexNexus]

Hello,

I've been using PINN since years now and its a really great solution, always improved in a very good way. I never had any problem during installation nor update until today.

On a fully functional Raspberry PI 3B v1.2 (so quite an old one for sure) running PINN 3.8.8, I have been notified a new version was available, and as usual I applied the upgrade. After that, PINN didn't start any more, displaying a Kernel panic message. Additionally, the "os" sub-directory was empty, but I usually put there the image(s) of the installed system(s) as I am currently doing tests often resulting in a dysfunctional bookworm system, and so I need to be able to replace it from scratch (thanks to PINN functionalities).

Some details to reproduce:

• Raspberry PI 3B v1.2 with official power supply.
• SD card 32Gb (RECOVERY: 1.86Gb, SETTINGS: 128Mb, boot64: 512Mb, root64: 8Gb, data: ~11Gb, boot640: 512Mb, root640: 8Gb)
• PINN 3.8.8
• 2x RasPIOS (64 bits) - bookworm from 2024-03-15 - original kernel 6.6.20 upgraded to 6.6.28

To solve my problem, I re-formatted the RECOVERY partition, restored PINN 3.8.8 files and its "os" sub-directory content, updated the recovery.cmdline file (essentially removing the "runinstaller" argument, and adding the "no_update"). Then, PINN and the two RasPIOS systems were able to start again normally.

I know PINN is currently on its way to gain full compatibility with the PI 5. So perhaps something is simply wrong in my setup (using an old PI 3B). In that case, just let me know about it.

Thanks a lot for your work.

[procount]

Sorry the upgrade didn't quite work out for you.
The new version is quite a bit bigger to accommodate the Pi5 kernel and I struggled to get it to fit (hence the change in wallpaper for some cases). I tried hard to account for the change in firmware names etc, but a few corner cases slipped through. So for ease in 3.9.1 I just wipe the recovery partition and install the new version. The only thing that is preserved are the settings from recovery.cmdline to the new cmdline.txt. This is why your /os folder contents disappeared.

You already have an expanded recovery partition, so I'm surprised it failed (unless your /os folder was totally full!).
You could repeat your repair process using PINN 3.9.1 instead of PINN 3.8.8 and see if that works (if not, just go back to 3.8.8). Make sure you have enough free space for it (maybe delete the /os folder temporarily and restore afterwards)
3.9.1 is essentially 3.8.8 PLUS the Pi5 kernel and overlay files so there shouldn't be any reason for the kernel panic on a PI3. Maybe it was just an installation failure of some sort 🤷‍♂️. Let me know if it's still a problem though.

Glad to hear you are making the most of PINN's functionalities, though! 😁

[VortexNexus]

My recovery partition has about 700Mb free space even with the /os sub-directory containing RasPIOS images. So I don't think the partition space was causing the problem.

As you proposed, I retried my repair process using PINN 3.9.1 version and modifying the /cmdline.txt file (thanks to you pointing the name change). As you expected, everything went well. But now I don't know what went wrong the first time, and I don't dare let the update happen on my other raspberry pi.

You surely noticed the unnatural data partition size. After OSes installation (through PINN), I used GParted to move and resize all partitions in the way I need for my tests. But I didn't "inform" PINN about those changes in any way. Do you think this could be the origin of the update problem? If so, what should I do so that PINN can deal with the changes during the update process?

Additionally, do you plan from now on to force this /os sub-directory "clean up" on every PINN's update? This would be very annoying as I don't have physical access to every raspberry pi I use, and keeping the original OSes images on the same SD card occasionally saved my life ;-) ...

[procount]

I can't imagine what happened, then. The upgrade process normally works fine. From 3.8.8 I added a 'preupdate' script feature so that I could adapt the upgrade process. This was necessary this time to account for the filename changes and the lack of disk space. In addition, I save the boot partition contents to a temporary folder before upgrading so I can rollback if it fails due to lack of disk space.
Hmm, a thought has just occurred to me that this might have caused your failure as you have a large /os folder.
Could you perhaps test that hypothesis? If you restore 3.8.8 with and without your /os folder populated and see if that makes a difference to the upgrade to 3.9.1?
Provided you have local access to your RPi to recover if it fails again, you should be ok as the /settings partition and all your OSes are left untouched.
To be extra safe, just take a backup copy of your /os, config.txt and cmdline.txt/recovery.cmdline files before upgrading.
BTW - it shouldn't be necessary to reformat the RECOVERY partition - just delete all the files on it before unzipping pinn-lite.zip.

Moving and resizing the partitions should not have any effect on PINN. PINN only cares about the order of the partitions and depending on the OS, the partition label, partuuid or uuid. It does not care about the size or location of the partition.
PINN 3.9.1 now has the ability to specify the partition sizes when you install the OSes (similar to Matt's webpage) so that should make life easier for you!

I shall have to rethink the "clean up" of the boot partition on the next upgrade. I had overlooked that as I am stopping providing a full NOOBS image with some OSes stored in the /os folder as it is too time consuming to produce. For users who have already enlarged their recovery partition, this should not be necessary anyway.

[VortexNexus]

I made the tests you asked for, with and without my /os folder populated.
Just to remember, here are the steps I applied:
(hardware: empty 64Gb SD card, Raspberry Pi 3B rev1.2, Raspberry Pi 3B+ rev1.3)

• create one fat32 partition of 10Gb at the beginning of the SD card (gparted)
• extract PINN 3.8.8 files to the partition (without any configuration change)
• depending on the test, copy my RasPiOS folder to the /os folder of the partition (1.15 Gb)
• start the Pi, wait for PINN's update notification and apply it immediatly.
• wait for PINN's restart after the update has been applied, and check Pi state.

In conclusion, your feeling seems to be right. When the /os folder is empty, everything goes well. But when there is something in the /os folder, the update process ends on a kernel panic message:

Beside this, I have bad news: I tried to update to PINN 3.9.1 on a Raspberry Pi 4 (Model B Rev 1.4). I tried through the normal PINN 3.8.8 update process, and also manually replacing the files in the recovery partition. This Pi has its OSes images on a USB key, and it does not have any data in the /os folder. However there is a reserve=+800 cmdline argument, and so the recovery partition has enough free space.
But I've not been able to start PINN 3.9.1, only obtaining the following lines:

So I manually made a roll-back to PINN 3.8.8.

I don't have any Raspberry Pi 5 at the moment, so I couldn't give it a try.

Hope all this will give you a little help.

[procount]

Thanks for doing these tests. I think that confirms my suspicion and the the kernel panic is understandable.
The issue with your 4B looks like something else related to the display.
Do you have a monitor plugged into the 2nd HDMI port instead of the 1st one next to the USB-C power input?

EDIT: Have you modified config.txt or pinn_init?
What happens if you rename pinn_init?
e.g. mv pinn_init pinn_init.bak

[VortexNexus]

Effectively, changing the HDMI port allowed a successful update process to 3.9.1, like expected as /os folder is empty.

No I didn't do any change in PINN's configuration, except in the recovery.cmdline file (adding allowed parameters like vncshare and others). Those changes where already effective for PINN 3.8.8.

I found that those kind of messages are displayed by an old feature called RPI safe mode, only still supported by NOOBS (and possibly by inheritors like PINN), that can be activated by shunting GPIO pins 5 and 6, to be able to edit config.txt and cmdline.txt with vi in a busybox shell.
My pi is a media server boxed in an Argon ONE M.2 Case, but as far as I can see, no GPIO pin is in use.

So, that's right: my HDMI connection was not "as expected", but falling in emergency mode seems to me a little extreme :-) and not really expectable as PINN 3.8.8 is not affected by the misconnection. Nevertheless, thanks to you for your really appreciate help and work.

[procount]

Wow, that was totally unexpected. I don't think it is a failure of the upgrade process, but a failure of 3.9.1 to startup when using HDMI2 instead of HDMI1.

It's not from RPi Safe mode, but when the Pi4 came along with 2 HDMI ports, I tried to detect which one is in use and switch PINN to using that one. First time I have seen such a failure. I will see if I can replicate it.

[procount]

This is indeed an issue caused by the renaming of the recovery files.
This was necessary as the Pi5 does not support this alternate load mechanism, but the loss of HDMI2 use was totally unexpected.
The names of the firmware files seem to change their behaviour.
I have reached out to see if I can get some help fixing this.
Meanwhile, don't use HDMI2 !

[procount]

Fortunately, we have a quick solution! 😄
Please add the following to config.txt, I suggest just above the [pi4] section:

[HDMI1]
hdmi_force_hotplug=1

This is done automatically in recovery.elf, but when it's renamed to start.elf, it isn't.

[Pivan78]

Same problem as above.
But on Raspberry Pi 4 2GB working normally.
If I installed PINN older version to Raspberry Pi 400, working well.
But after update got error:

ioctl FBIOPUT_CON2FBMAP: Invalid argument
ioctl FBIOPUT_CON2FBMAP: Invalid argument
Recovery application crashed
Starting shell
sh: can't acces tty; job control turned off
/sys/class/block #

After exit - kernel panic...
Hardware name BCM2711

Doesn't matter if is fresh install or update from older version. Or if boot from SDcard or USB.
Old version working without issues.

[procount]

Did you modify config.txt as above?

[Pivan78]

Did you modify config.txt as above?

Yup. But not help in my case. I stay with previous version for now.

[procount]

Are you using the secondary HDMI port?
Does it fail if you use the other HDMI port?

[procount]

I've tested the change on my Pi400 and it works fine. Please check you modified it correctly.
I am pushing this change up in v3.9.2 shortly.

[procount]

@VortexNexus - The v3.9.2 upgrade should now work with your expanded recovery partition and populated /os folder.

[dbjones-clanjones]

Although I had been aware of PINN for a little while, but never had gotten around to using it, probably as I use all my Pi 4b's headless for server apps and my Pi 400 has gathered dust for the last year as I had to undertake multiple house moves. However, I want to be able to use to try multiple different operating systems such as the latest Ubuntu and Fedora, without messing with my Raspberry Pi OS install all from the same Samsung T1 500gb SSD drive.
So I began trying yesterday, but have run into problems. I started with PINN 3.9.1 and I am receiving the same Kernal panic error when I create 3.9.1 from Raspberry Pi Imager on a Raspberry Pi 400, using a Samsung T1 500gb SSD drive (also tried using Sandisk 256gb USB 3.1 key). It booted successfully from the same drive(s) using PINN 3.8.8. Some (or possibly all of) these attempts had 2 HDMI monitors connected. I'll try again and if problem persists will create a separate issue. But wanted to log that the Kernal Panic had happen on my Raspberry Pi 400.

Tried again with 3.9.2 and the issue is no longer occurring.

[procount]

Glad to hear 3.9.2 fixed your issues.
But I'd like to understand your 3.9.1 issue a little more as 3.9.2 fixed 2 bugs.

You said you started with 3.9.1 and got a "kernel panic" after flashing it directly from Rpi Imager. If so, that would be a different issue because the kernel panic mentioned here was due to an upgrade failure from 3.8.8. "Kernel panic" is actually mentioned on the screen.

But you also mentioned you tried 3.8.8. If you upgraded from there to 3.9.1, the kernel panic would occur due to additional files on the recovery partition like in the /os folder..

The 2nd bug fixed in 3.9.2 concerns using the 2nd hdmi port, but I thought it would only happen if you only had hdmi1 connectrd. Switching to hdmi0 would solve that issue. Not sure it would occur if you had both connected at the same time. This was not a kernel panic.

[VortexNexus]

@procount:
Sorry for the late answer, and for the the bad news, but it seems this issue cannot be closed at the moment :-) ...

I made the tests you asked me for again, trying to update from PINN 3.8.8 to 3.9.2. But it seems that this requires PINN to update first to 3.9.1, falling into the same problem of the kernel panic.
In addition, and this is my bad as I didn't noticed it during my first tests, it appears that when os folder has any content, the update process from 3.8.8 to 3.9.1 clears its content but also clears the cmdline.txt file content ... Now we have the real origin of the kernel panic fallback. And, bad news again, the update process from 3.9.1 to 3.9.2 keeps the os folder (thanks a lot for the modifications you made), but clears the cmdline.txt file content too, and so fallbacks again into kernel panic. This appears logic to me as I imagine you didn't change anything in 3.9.2's cmdline.txt handling, so the problem has been inherited.
On the other hand, when os folder is empty, everything goes well in the chained update processes, so there are good news too: no unexpected modifications' side effect :-)

[procount]

Hopefully the update to 3.9.2 fixes things now.

[VortexNexus]

Mmh, not sure to understand what you mean. As explained in my previous post, 3.9.1 and 3.9.2 updates both left the cmdline.txt file without any content ...

I thought it was possible to update from PINN 3.8.8 directly to 3.9.2 using the "Ignore" button on PINN's 3.8.8 update confirmation dialog. So I tried, but it seems it has the same behavior as the "No" button, doing nothing.

Now, imho, for anyone having PINN 3.8.8 installed with a non empty os folder, auto-update is not possible anymore, leaving only 2 choices:

• remain sticked to PINN 3.8.8, adding the "no_update" flag in the recovery.cmdline file.
• having physical access to the raspberry pi and its SD card, update PINN to 3.9.2 by manually replacing all the files except the os folder, and adding all needed custom parameters in the cmdline.txt file.

Do you see another option?

[procount]

Assuming you have 3.8.8 installed, please try the following:

1. Download https://sourceforge.net/projects/pinn/files/pinn64/pinn-lite.zip/download
2. With PINN booted, copy this file to /tmp. (This is a temporary folder, so if you turn off your Pi or reboot it, it will be lost)
   You can do this several ways: a) transfer it via USB stick. You'll need to do it from an SSH shell or the recovery cmd shell (login with root/raspberry). Or b) my favourite using scp: scp pinn-lite.img root@<PINN-ip-address>:/tmp (replace <PINN-ip-address> with the IP address of your Pi)
3. Go to PINN's maintenance menu
4. tick the checkbox next to PINN and select reinstall. This will reinstall PINN from the 3.9.2 pinn-lite.zip file in the /tmp folder, thus hopefully bypassing your problems with 3.9.1. 🤞

[VortexNexus]

I've given it a try, but unfortunately, that didn't work.

Remarks:

• both file transfer options need ssh server to be active in the running PINN system.
• PINN's FTP server doesn't seem to understand SFTP protocol, which is the default one for distros using a recent OpenSSH release. So -O (uppercase letter "o") option need to be added to scp's cmd line.
• the downloaded file is named pinn-lite.zip, but your given scp cmd looks for pinn-lite.img. So I changed the cmd line for the ".zip" version.

At the end of the transfer process, I checked through SSH that the archive file was really present in the remote /tmp folder. So, having all set up, I went to the maintenance menu, toggled the PINN's line check box, and clicked the "Reinstall" button. The PINN's update dialog showed up. A click on the Details button showed the 3.9.1 change log ... Anyway, I thought the data have been downloaded from internet, so I started the update ... and obtained the now well-known kernel panic message 😄.

So it seems the update process don't care about an already downloaded ".zip" file in the /tmp folder. Or perhaps, do I have to rename the ".zip" file into ".img"? Seems not so obvious ...

[iNgeon]

Have a Raspberry PI4 4GB Model B.
Everything was working fine. PINN version p3.8.7 was installed
PINN asked for update on boot. Did first updgrade and PINN upgraded to p3.8.8
Rebooted. PINN asked for update again. PINN updated to p3.9.2
Raspberry rebooted with console on screen, crash-failed with last line item. end Kernel Panic
Formatted SD card and started from scratch iow wiped all other partitions and only left a 16GB FAT32 for PINN to copy.
Downloaded latest version of v.3.8.8 pinn,.zip 2024-02-19, 3.4GB
SD format. Again PINN asked for update to p3.9.2
Raspberry rebooted with console on screen, crash-failed with last line item. end Kernel Panic
Note: Also tried both HDMIO and HDMI1

Attached

screenshot

[procount]

Unfortunately the log doesn't help me much. It says it can't mount the rootfs, which is actually an initramfs so I'm confused.
As you have wiped your card, why don't you run RPi Imager and install PINN 3.9.2 directly from there onto your SD card? You'll find PINN under the Misc Utility Images.

[iNgeon]

Can't seem to find a log ? Does PINN dumb one somewhere on the disk ?
Will try loading 3.9.2 direct, was hoping to share the full log, tried capturing on video via phone but just a blur.

edit: Okay loaded onto SD card, 3.9.2 worked by installing the Lite version, still would like to pull full log on upgrade fail, i have some other PI4 i'd like to upgrade PINN without having to redo everything...

[procount]

There is the kernel dmesg log which you screenshot a photo of, and PINN has its own log in /tmp/debug.
Unfortunately, neither of these are saved if you lose power, so in the case of a kernel panic you can't get at them.
If the upgrade fails to the command prompt, then you can usually access the logs before you remove the power.

The kernel panics during the development of the upgrade process were due to additional files in the /os folder, but that was resolved and you've tried an upgrade from a default 3.8.8 version (which worked for me) so this is something new. Is there anything out of the ordinary with your setup? - any unusual hardware attached, for example, or additional files stored on PINN's recovery partition other than those supplied?

Something to remember in future, is that if an upgrade fails, it will only really affect the recovery partition, so there's no need to completely start from scratch and wipe your OS partitions. In that case, the best way to proceed is to wipe only the RECOVERY partition and then unzip pinn-lite.zip (v3.8.8) onto it. When you boot PINN, there will be a prompt to delete all the partitions, but you should skip that to keep your existing OSes. (You can avoid that risk altogether by deleting the 'runinstaller' option from cmdline.txt to be doubly sure).
3.9.2 is really only the merger of the PI5 beta with the other Rpi models. The only thing you're getting extra is the adjustable partition sizes, so if you're happy without that, you can stick with 3.8.8 until this is resolved. The other major change is the recommended recovery partition size has doubled to 128MB to account for the new kernel. 3.9.2 will just about fit in 64MB but it is a tight squeeze. The upgrade process tries to make sure it will fit (e.g. by using the smaller blue wallpaper) and will rollback to the old version if it won't, but maybe something on your system is breaking it.

In order to resolve this, seeing as there are no logs, I think it would be necessary to have an image of your recovery partition so I can debug this myself (provided you haven't filled your /os folder with MBs of files!). So before you upgrade another one, please take an image of just the RECOVERY partition (e.g. use dd) and a copy of the output of fdisk -l so I can construct an identical image BEFORE upgrading. (Ask me if you don't know how to do this). NOTE: if the upgrade fails, this image would be useful to restore your card back to where it was beforehand anyway as an alternative to reinstalling 3.8.8.

At some point, if you are upgrading several PINN installations, you should really expand the RECOVERY partition otherwise future PINN upgrades may not fit.

1. Selecting the runinstaller option is one way to do this, but the downside is it will wipe all your OSes. One way to overcome this is to use PINN to backup each OS to a USB stick first. Install PINN 3.9.2 to a new SD card of similar or larger size and then reinstall all your backed -up OSes from USB. This is a bit long-winded and will take a while, but it keeps your original card intact until you have verified your backups have restored correctly. If not, you still have your original SD card as a backup! If it works, you can reuse your original SD card for the next SD card to upgrade.
2. Another way is to use Gparted to move and resize your partitions. Your PINN SD card must not be mounted to do this so you will need to do it on another computer, or by running another OS. To minimise the disk moving required (assuming a typical installation), I would shrink P7 by 64MB, move it right, Move P6 right, then move P5 right. This will leave you with a gap at the start of the extended partition P2. Now you can shrink P2 by moving the start of it right 64MB, and finally you can expand P1 to fill the gap. (It's a but like one of those sliding puzzle games!) Also you can do this on a clone of your SD card, so if something goes wrong, you still have the original SD card.

[VortexNexus]

@procount : as I pointed it in a previous post, the update process to 3.9.2 clears the content of the cmdline.txt file. I'm not sure you noticed it. For me, this is the reason why the rootfs cannot be mounted (the system has no info about which mounting point must be used), and also the reason why the system falls in the emergency "kernel panic" mode.

[procount]

@VortexNexus - yes I noticed your comment and changed the procedure. The upgrade process (should) now copy recovery.cmdline (or cmdline.txt) to /tmp first, then restores it after upgrading. Rather than reinstalling a totally new version, I maintain the old version and modify the few changes that are needed for 3.9.2.
I didn't experience any failures after implementing this, but if yours is still failing, then clearly some use case is still missing and it needs relooking at.
@iNgeon - If you get another failure again, please check if cmdline.txt is present after the upgrade.