Migration guide to convert Uyuni installation from RPM to container setup?

[mayrstefan]

With the release of 2024.05 we got two announcements that will affect existing Uyuni installations:

1. the container version of Uyuni is now ready for prime time
2. there will be only one more RPM release: 2024.06

So what are the supported migration or upgrade paths to switch an existing installation from RPM to the new container setup? Do we have a migration guide that we can follow?

[mayrstefan]

I'm aware of

• https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/server-intro-uyuni.html#container-deployment/suma/migrate-uyuni-to-a-container.adoc which does not contain any information at all.
• https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/container-deployment/uyuni/migrate-uyuni-to-a-container.html which does explain the difference between the kubernetes and podman variants and has no information regarding the verions that can be used for that migration.

[mayrstefan]

And some more confusion: https://youtu.be/0eiQIDTGY-U?t=88 uses Leap 15.5 and 15.6 while the docs in https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/container-deployment/uyuni/opensuse-leap-micro-deployment.html use only Leap Micro 5.5

[rjmateus]

For now we, are only supporting the migration or existing servers to a new machine. The reason is that we didn't had the time to test an in-place migration yet.
The tested scenario is deploy a new machine with opensuse leap micro 5.5, install the new tools, configure ssh connection between new machine and new one, and run the migrate command. All data should be migrated to the new containerized version.
After that you only need to change DNS to point to the new machine and al should be working with all the data.

The migration was tested for uyuni running on podman and kubernetes.

Thank you for point out the documentation, I also think it's not clear enough, and we will work on improving it.

[sbluhm]

@rjmateus , got one more question on this for testing.
Am I correct to assume that the migration is non-destructive/intrusive? I.e. Can I run the migration multiple times with the same result?
Otherwise it would require cloning the prod server first for testing.

[rjmateus]

Migration is non-destructive. With migration from 4.3 to 5.0, the old machine will stay untouch. If something going wrong in the migration, you can just revert the DNS and have the old suma working.
Then you can migrate again, how many times you want.

[mayrstefan]

So if I understand this correctly we can use the migration command to upgrade from a 2024.03 legacy to a 2024.05 container setup in one step. No need to upgrade the legacy setup first and migration in a second step. Although I think I would prefer doing the two-step approach so differ which issues come from the version upgrade and which come from the new container setup.

When the migration is done - can we change the Uyuni master name on the clients to get rid of the old DNS name or will it have to stick around forever? I would like to transfer all clients to a generic alias which is easier to move around between servers. Keeping the old server name would complicate things for us.

[rjmateus]

You can migrate from uyuni version 2024.03 to a containerized one, with no need for an intermediate step.
The current approach is to migrate to a new server and adjust the DNS to redirect the traffic from the old server to the new one. After this one should be able to change server DNS, as it was able in the past.
However, the script to change the DNS has not been migrated yet, so it will not work as is. Updating the script is in our plans, but I cannot say when it will be ready (we hope it will be soon).

[ppanon2022]

Does the migration script handle migrating imported/custom signed cert/key pairs for the server if they're in use? If not, do you have some guidance on how to modify settings in the container and deploying the associated CA chain certs? Other customizations of concern would be the salt pillars for copying GPG keys according to https://www.uyuni-project.org/uyuni-docs/en/uyuni/client-configuration/gpg-keys.html#_user_defined_gpg_keys (which currently doesn't seem to actually work for us even in an rpm installation), generating new management GPG keys for signing repo data in the containerized model https://www.uyuni-project.org/uyuni-docs/en/uyuni/administration/repo-metadata.html. How does that get done in the container environment? When can we expect updated documentation for doing that?

For anything that isn't supported by the migration script, I'm going to guess that we would use mgrctl term, similar to what you've indicated for the PAM setup in #8675 (which we also use), but are there any gotchas?

I'm also not clear on how updates are supposed to happen in the containerized environment. If we just update the container, but we've had to make these changes in the container overlay, won't they get lost with the container update? Having to redo all these changes manually with each new container release/update would be a big step backwards.

[mbussolotto]

GPG keys should re-added using:

mgradm gpg add [URL]... [flags]

For anything that isn't supported by the migration script, I'm going to guess that we would use mgrctl term

correct, but the migration should already migrate eveything (except the gpg key). Let us know if anything is missing.

I'm also not clear on how updates are supposed to happen in the containerized environment.

First you'd need to upgrade mgradm, then you can run mgradm upgrade command. The command will automatically get the new image and run the upgrade script.

If we just update the container, but we've had to make these changes in the container overlay, won't they get lost with the container update?

All the folders containing configuration are persisted, so the changes aren't lost during upgrade. You can find the list of the persisted folder here https://www.uyuni-project.org/uyuni-docs/en/uyuni/installation-and-upgrade/container-management/persistent-container-volumes.html .

mgradm provides command to applies changes in non-persisted folder. By now we identifies:

Administrator tools:
  distribution Distributions management
  gpg          Manage GPG keys for 3rd party repositories

So there's no changes that needs to be re-applied after container restart, as you said it'd be a big step backwards: let us know if you find any of this problem and we'll provide a solution (it'd be provide a mgradm command to avoid it)

[ppanon2022]

Sorry, I should have made it clearer that gpg key distribution wasn't even working for us in our current RPM install. But, after writing that, I also found out I didn't notice that the spacewalk-backend* packages didn't get updated on the last update, so its possible incompatibilities between those older packages and the current version of salt server could be the cause of that issue. I suppose greater atomicity during the upgrade to avoid that scenario is one of the benefits of the containerized model.

However, I'm concerned that there may not be enough time to fully test the containerized Uyuni with only 2 months of overlap between it being greenlit for production use and the retirement of the rpm-based release. I've had enough issues with the AppStream switch (which would be a fantastic improvement I would be gushing over if we hadn't had the spacewalk-backend issues - kudos really), and earlier upgrade issues in the last 6 months, that we're leery of being an early adopter on switching to the container release. If everybody takes the same approach then you're not going to find a lot of issues until the last minute in July after the cut-off of the rpm release. Especially with it being summer months and Uyuni admins going on vacations. I realize you don't want to maintain dual development streams for the two releases any longer than absolutely necessary because the extra workload reduces developer time available for app feature progress, but that schedule seems very aggressive to me given its timing.

For example, I didn't see anything in your reply on how to handle web site certificates (server and proxy) signed by our private/corporate CA. We also use a modified wrapper script in the bootstrap subdir around the generated bootstrap script(s), which deploys our corporate CA certs, re-generates machine-ids in VMs prior to bootstrapping (in case they're a clone), identifies the O/S flavour and other details to select the appropriate generated bootstrap script (and associated activation key) to run. These make it much easier to have users bootstrap new builds into Uyuni by giving them a one-size-fits-all command. We've also developed some custom metric reports for use with spacewalk-report on the reporting database (and which use a non-packaged csv2html executable and mutt to send the results via email). We're also using some 3rd party salt formulas for configuring firewalls, snmp agents, and other functions and only some of them can be done using Uyuni's salt GUI facilities - some of them need the underlying salt installation. And those are just the few customizations we've made for supporting our business processes that I remembered on short notice. We're also looking at starting using Cobbler for O/S deployment and would probably want to optimize that process a bit as well with secondary/chained scripts that are also likely to need persistence in the container environment.

Therefore, my impression from the limited scope of your response is that, as-is, we would need to re-deploy most or all of those changes after each container upgrade, because nobody's discussed making those kinds of customizations specific to local needs in discussion groups since they're relatively straightforward for a good Linux admin. I expect there are a number of other sites out there with different customizations that would be similarly affected.

Thanks to [mayrstefan] for opening this discussion.

[mbussolotto]

Thanks for providing all these informations.
About CA certificates and custom salt formula, they will be automatically migrated by mgradm migrate podman ..., so no additional steps for you.

We've also developed some custom metric reports for use with spacewalk-report on the reporting database

I'm not completely sure about this, I'm going to verify if also this part will be automatically migrated, but I guess so

[mbussolotto]

I confirm also the reports are automatically migrated by mgradm migrate podman ...

[digdilem]

I think ppanon2022 makes some very valid points, all of which I share. The timescale seems very short for all of those reasons, plus the complexity of what may be quite evolved installations. Migration scripts in my experience rarely manage to resolve every single issue without fault, so it seems likely there will be a lot of testing and issues arising.

That might mean several migration attempts for each user, then if they do hit problems, having to restore from backups, report the issue and wait for the dev team to fix. It feels unrealistic for that to happen within one month at any time of the year, but especially during the northern hemisphere's summer break.

Admission: I hadn't actually realised until now that this container change would be mandatory and that Uyuni would only be available as a container option going forwards, so perhaps I hadn't given it my full attention. But this makes me uneasy and not every admin will have familiarity with the tools and methods going forwards. I would have expected some quite big notifications to make users aware of this, and at least a six month transition period.

[admd]

We have a news today in community hour around rpm based Uyuni lifecycle, tune in!

[mayrstefan]

It is good to read we have some more time for the migration. But I'm still irritated by the target OS: the updated slide at https://youtu.be/SA-UDtE97hc?t=304 still uses classic Leap while the docs and @rjmateus mention only Leap Micro

[rjmateus]

@mayrstefan we are recommending the usage of Leap Micro because is what we are using in our CI to test the containerized deployment. However one can use another OS to deploy uyuni, but we are not testing in there.
With the containerized version I'm not expecting issues when running on a different OS version, as long the podman and netavark version are compatible.

[mayrstefan]

@rjmateus why is this not reflected on those slides and will the migration script also work on classic Leap or only with Leap Micro? This is really confusing me

[rjmateus]

@mayrstefan I'm not sure about what that slide means. maybe @deneb-alpha or @raulillo82 can give more insides.
I'm not sure if on the left side it refers to the base container image used. The managing tools can be installed in other OS's then just leap micro, and migration with mgradm should work