Ugrade base system to Ubuntu 24.04 #1067
Conversation
This should have been done right after the 1.5 release, so let's do it now.
The NodeJs package manager (NPM) is no longer automatically installed when we install NodeJS, so we ask for it explicitly. We need it to install Obographviz. Ideally it should be possible to install both NPM and Obographviz in the builder image and then to transfer only Obographviz to the final ODKFull image, thereby reducing the clutter in ODKFull, but this will need further investigation.
This is a runtime dependency of Konclude (even the statically compiled one that we use on x86_64). It was probably automatically pulled by another package on Ubuntu 22.04, but it is not on 22.04, so we need to ask for it explicitly.
We have now officially entered the era where Python developers don't even try anymore to maintain a coherent system, and where using virtual environments is supposed to be the standard way to install Python programs without breaking anything... Isn't that great? Fuck it, we are _not_ going to create a virtual environment just for the OBO Dashboard. If the OBO Dashboard dependencies end up breaking the rest of our Python packages, we might as well just create a separate Docker image just for the dashboard.
The odk.py script has some issues when we run it under Python 3.12, we fix them here.
Installing virtualenv in the builder image has nafarious consequences when we try to later install Python packages. That's because the Ubuntu package for virtualenv automatically installs platformdirs version 2.5.1, which then prevents us from installing the platformdirs 4.x that we need as a dependency for some of our packages. The only reason we had virtualenv in the builder image was that we use it to run the update-constraints workflow (in which we try to install all our Python packages in a virtualenv). So here, we 1) remove virtualenv from the builder image; 2) amend the update-constraints.sh script to make it install virtualenv itself.
gouttegd
changed the title
There was a problem hiding this comment.
Some system packages that were previously implicitly installed now need to be explicitly asked for (e.g. libpcre3, needed for Konclude, or npm, needed to install Obographviz).
Ok.
The odk.py script has some issues with Python 3.12 that need fixing.
I assume this is primarity the dataclass field issue, right? So all fixed?
| @@ -95,7 +97,7 @@ COPY scripts/obodash /tools | |||
| RUN chmod +x /tools/obodash && \ | |||
| git clone --depth 1 https://github.com/OBOFoundry/OBO-Dashboard.git && \ | |||
| cd OBO-Dashboard && \ | |||
| python3 -m pip install -r requirements.txt && \ | |||
| python3 -m pip install -r requirements.txt --break-system-packages && \ | |||
There was a problem hiding this comment.
The OBO Dashboard must now be installed with --break-system-packages, otherwise pip will flatly refuse to install it system-wide.
I wonder if this is very suboptimal and we should, instead, install OBO dashboard as a pypi package and install it the usual way? Why is --break-system-packages not needed in docker/builder/Dockerfile?
There was a problem hiding this comment.
What are the break system packages? I can update to the latest version, if possible.
There was a problem hiding this comment.
I wonder if this is very suboptimal and we should, instead, install OBO dashboard as a pypi package and install it the usual way?
We could, but then we would need actual releases of the OBO Dashboard package whenever we want to update it in the ODK. For now, the OBO Dashboard is effectively working in a “rolling release” fashion: it doesn’t do releases and we always install it from the tip of the master branch. That would need to change.
There was a problem hiding this comment.
What are the break system packages?
What do you mean? We are not breaking anything. --break-system-packages is merely the name of the option that forces pip to accept an instruction to install a package in the system-wide site-packages directory – without that option, it flatly refuses to do, because in Ubuntu 24.04 the system-wide directory is considered to be the private garden of the distribution’s own package system (APT).
The option is named like this as a rather condescending way of telling end-users that they should not try to install anything in the system-wide directory, because if they do they run the risk of introducing a conflict between packages that have been installed there by the distribution (packages installed with apt-get install python3-mypackage) and packages installed with pip.
But in a Docker image, we mostly don’t care about that risk, since when the image is built, the system-wide directory is in effect read-only. Once we are done installing whatever we need in it, nobody will ever modify that directory ever again.
There was a problem hiding this comment.
That would need to change.
I can create a release right now.
There was a problem hiding this comment.
Why is
--break-system-packagesnot needed indocker/builder/Dockerfile?
Because in docker/builder/Dockerfile we are not installing Python packages to the system-wide directory: we are installing them to a staging area, from where we copy them to the ODKLite/ODKFull images.
| @@ -787,7 +787,7 @@ class Generator(object): | |||
| """ | |||
|
|
|||
| ## TODO: consider merging Generator and ExecutionContext? | |||
| context : ExecutionContext = ExecutionContext() | |||
| context : ExecutionContext = field(default_factory=ExecutionContext) | |||
There was a problem hiding this comment.
Is this because of the newer dataclass version in the newer python?
There was a problem hiding this comment.
Yes.
The other fix that the odk.py script needed was the removal of a bogus escape sequence in one of the messages it displays to the user (\@).
| @@ -27,8 +27,7 @@ RUN apt-get update && \ | |||
| rustc \ | |||
| cargo \ | |||
| python3-dev \ | |||
| python3-pip \ | |||
| python3-virtualenv | |||
There was a problem hiding this comment.
We can no longer have virtualenv in the builder image, as it messes with the installation of our Python packages.
What about pipelines based on ODK that require the ability to install a virtual environment before running system? Seems quite fundamental a thing to remove from the image.. How does it "mess"?
There was a problem hiding this comment.
We are talking about the builder image here! This does not remove anything from the final ODKFull!
Having virtualenv installed in the builder image creates a mess because
the Ubuntu package for virtualenv automatically installs platformdirs version 2.5.1, which then prevents us from installing the platformdirs 4.x that we need as a dependency for some of our packages.
| @@ -97,7 +97,7 @@ COPY scripts/obodash /tools | |||
| RUN chmod +x /tools/obodash && \ | |||
There was a problem hiding this comment.
Fuck it, we are not going to create a virtual environment just for the
OBO Dashboard. If the OBO Dashboard dependencies end up breaking the
rest of our Python packages, we might as well just create a separate
Docker image just for the dashboard.
Your commit messages are great, but it's important to avoid using offensive language in them. Thank you!
There was a problem hiding this comment.
Sorry but I won’t take lessons in how to write commit messages from folks who can’t be bothered to write decent commit messages themselves.
If people complain about an occasional profanity in one of my commit messages, at least I will know that there are some people who read them.
There was a problem hiding this comment.
Correct me if I am wrong but I dont think this was meant as a lesson, more like a statement of affectedness.
I also prefer to avoid strong language in the spirit of conduct (as the Covenant Code of Conduct suggests), I had endless discussions of misconduct in some other projects which just distract from our common goals.
There was a problem hiding this comment.
Correct me if I am wrong
My pleasure.
I dont think this was meant as a lesson, more like a statement of affectedness.
A “statement of affectedness” would have been something like, “I felt offended by the presence of that F-word, I don’t like reading such things when I am reviewing a PR. I’d appreciate if you abstained to do that“ or similar.
“It’s important to avoid using offensive language in [commit messages]” is absolutely a lesson in how one should write a commit message – one that I find hard to accept coming from someone who I can’t remember having ever seen write a single decent commit message that actually explains what they are trying to do.
as the Covenant Code of Conduct suggests
The ODK does not have a code of conduct. Maybe it should (though I’d much rather have a “code on how to write efficient commit messages so that future contributors don’t have to guess what the f... the developers had in mind when they committed that change”, but it’s OK, we can have different priorities), but currently it does not.
Even if it had, according to the most popular code of conducts in use out there, the correct response to the occasional use of inappropriate language is a “private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate” – not a public rebuke.
I had endless discussions of misconduct in some other projects which just distract from our common goals
Fine. I hereby publicly apologise for having expressed my state of mind, as to why I believe installing the OBO Dashboard in a virtual environment just to circumvent pip’s default behaviour would be a unnecessary complication, using a 6-letters profanity instead of a more elaborate, professional, and polite phrasing. I did that for the sake of efficiency, but I didn’t consider that indeed people might not want to read a F-word when they are busy reviewing a PR.
I hope this will end the current distraction, and I will temporarily withdraw for all ODK maintenance to be sure you are not distracted again.
This PR updates the base system of all our images to the latest LTR version of Ubuntu, 24.04.
libpcre3, needed for Konclude, ornpm, needed to install Obographviz).--break-system-packages, otherwisepipwill flatly refuse to install it system-wide.virtualenvin the builder image, as it messes with the installation of our Python packages.odk.pyscript has some issues with Python 3.12 that need fixing.closes #1008