Upgrade horror, security issues

[frankystone]

Since we have several security issues for django and third party apps, this is
about upgrading...

Current status of main software:

• OS:
  • Ubuntu 18.04.6 (Bionic), supported until April 2023
    • Highest available python version as package: 3.6.7 https://packages.ubuntu.com/bionic-updates/python3
  • Next LTS release is Ubuntu 20.04 (Focal), supported until April 2025
    • Highest available python version as package: 3.8 https://packages.ubuntu.com/focal/python3
• Python/virtualenvironment:
  • The virtual environment is made with python version 3.6.8, EOL
• Django:
  • Version 1.11.29, EOL
  • Latest official supported LTS version is 3.2 (support until April 2024)
  • Next LTS release will start in April 2023
  • Django Release timeline (source):

Assuming one wants to upgrade to Django 3.2:

Although Django 3.2 supports python 3.6 some third party apps needs a higher
python or django version. Here's a list of used apps/programs in the virtualenvironment
and which versions they need if upgrading (?=haven't found info about):

Installed in virt_env	latest Version	Needed python version	Supported django version	Note
django	3.2 (LTS)	>= 3.6		currently security issue
beautifulsoup4	4.10.0	>= 3.0	independent
django_haystack	3.1.1	3.5 - 3.9	all
django-messages			compatible until 2.2	Not maintained anymore, see below
django-nocaptcha-recaptcha				Deprecated, see below
django-star-ratings	0.9.2	3.6 - 3.9	>= 2.2	can't be updated in current virt_env django version!
dj-pagination	2.5.0	?	>= 1.8
django-registration	3.2	3.6 - 3.9	2.2 - 3.2	can't be updated in current virt_env but security issue
django-tagging	0.5.0	>= 3.5	>= 1.11
django-filter	21.1	3.6 - 3.8	2.2 - 3.2	can't be updated in current virt_env but security issue
gunicorn	20.1.0	> 3.5	independent
lxml	4.7.1	> 3.5	independent
Markdown	3.3.6	3.6 - 3.10	independent
mysqlclient	2.1.0	>= 3.5	independent
numpy	1.22.1	>= 3.8	independent	can't be updated in current virt_env but security issue
Pillow	9.0.1	>= 3.7	independent	can't be updated in current virt_env but security issue
pydot	1.4.2	>= 3.4	independent
python-magic	0.4.25	>= 3.6	independent
Sphinx	4.4.0	>= 3.6	independent
docutils	0.18.1	>= 3.5	independent
Whoosh	2.7.4	?	independent

There are some more apps which will be installed as dependencies. Omitting them.

Most problematic seems to be django-messages which was last updated on October 2019. Maybe one can find a replacement on the django package index.

django-nocaptcha-recaptcha says it can be exchanged with django-recaptcha

Conclusion

The first step onto upgrading the security related apps/programs is creating a new virtualenvironment made with a higher python version.

Sticking on the server OS Ubuntu 18.04 means one can't create a new virtualenvironment because Ubuntu 18.04 does not provide a package for a higher python version. A possible solution is to compile an other python version on the server and create the new virtualenvironment with this new python version.

Upgrading the server OS to Ubuntu 20.04 one can create a new virtualenvironment with python version 3.8 which is the lowest version for numpy.

Any thoughts about this?

PS: Writing this doesn't mean i can do the upgrade

[Noordfrees]

Upgrading the server OS to Ubuntu 20.04 one can create a new virtualenvironment with python version 3.8 which is the lowest version for numpy.

When we do a big upgrade already, is there a reason not to wait another three months and then upgrade straight to 22.04? That's a bigger leap but I assume it would also be longer until the same problem arises again.

[frankystone]

According to the Ubuntu upgrade Introduction this needs two runs of upgrades: First upgrade to 20.04, then upgrade again to 22.04. Personally i have no experience with ubuntu upgrades at all... so i can't tell if this works without trouble.
@stonerl ? @SirVer ?

[matthiakl]

I think waiting for 22.04 is reasonable. If the configuration setup is not overly complex a fresh install might be better than two upgrade steps, because it looks like most of the packages needs to be replaced anyway. But this depends on the knowledge what has been done on the server (and I don't know anything about it ;-).
How urgent are the security issues? Is it possible to update django in the current environment and removing some (minor) functionalities to mitigated those issues?

Edit: It looks like django-messages can be easily updated to support django 3+ (see arneb/django-messages#149 or will-turner123/django-messages@6b0319c)

[frankystone]

How urgent are the security issues? Is it possible to update django in the current environment and removing some (minor) functionalities to mitigated those issues?

Didn't looked into the deep, but some of the security issues are not urgent for us. The problematic security issues in regard to python are:

• numpy: As far i can see numpy is only used in test_map.py which is probably not up to date anyway (not used in production)
• Pillow is used in several places (users profile, pybb, screenshots, wiki) but i can't say if we are affected by the security issue.

All other issues should be fixable by updating django.

So in result the only problematic one with regard to security is Pillow, which needs python version 3.7.

Edit: It looks like django-messages can be easily updated to support django 3+ (see arneb/django-messages#149 or will-turner123/django-messages@6b0319c)

django-messages has another culprit: Since a specific commit the use of the notification app is hardcoded into django-messages. The notification-app is used for emailing and we maintain an own copy of the original because of reasons. See this bug report

Upgrading django and all other apps is a lot of work. I did that two times in the past... it will take several weeks/months. And i don't know if i find the time and motivation to do this a third time (beside i have the feeling to get more and more stupid). Anyway: If someone wants to do this he needs the target python version to start working on this upgrading. Since Pillow is the problematic one, the target python version should be at least 3.7.

I think in general it should be considered to detach the virtual environment stuff from the OS stuff as pointed out in the first post (sticking on Ubuntu 18.04). The only downside is maybe that upgrading the OS gets probably more complicated, i don't know.

[SirVer]

Thanks for kicking of this discussion @frankystone, also excellent rundown of the current situation of the homepage package.

My experience says that ubuntu updates are easy and non critical, so I would not have qualms just running two lsb updates back to back. We have nightly backups of the server (which I would ensure work before doing the update).

As for the running of the website: It might be easier for us to run the gunicorn process within a docker container on the server. That way, its environment is decoupled from the OS and incremental updates should be easier. I have not worked with django in docker, but I am almost certain that this is the recommended way of doing it these days anyways. For example I found https://docs.docker.com/samples/django/, and https://testdriven.io/blog/dockerizing-django-with-postgres-gunicorn-and-nginx/ which seems to be close to the setup we would want to run.

[frankystone]

Thanks @SirVer ....

I am not familiar with docker, just tested it a bit in the last days with modified examples you gave. The main problem using docker for me is (for now):

1. Each docker container is decoupled from the host system, but our website needs access to the widelands binaries which are lying on the host system. E.g. the encyclopedia needs access to wl_map_object_info and the documentation needs access to extract_rst.py. I guess just mounting the folder with the widelands executable will not work because executing binaries will need much more than just a docker container containing python (think about the need for xvfb).
2. Although i read a lot of whether to use a virtualenvironment within docker or not, i get no clear meaning. Some say yes, some say no.
3. Developing the addon-server is done without docker. Is it possible to develop the website stuff in the same way?

So using docker puts currently a lot of questions in my brain...

[SirVer]

Re your points:

1. This is indeed a good question. One way could be to mount /usr/lib, /usr/local/lib and /usr/local/bin into the docker container, but that might create other issues. Ideally we could push static binaries into the docker container.
2. My stance is that it does not buy you anything to use a virtualenv in a docker container, if you want a different environment, you just rebuild the container.
3. Yes, it is always possible to work without docker. but it means that everything in the end has to work on the same computer, i.e. if we need different python versions, java runtimes and/or ubuntu releases for our individual tools, we have to somehow hack this environment together on the computer. With docker, every tool can have its own environment.

Docker is (a little bit of) upfront cost which will pay off I think.

[klaus-halfmann]

Docker is no Magic and solid choice (actually doing this on my Job in some openshift environment every day).
Decoupling is the basic idea of Docker and will harden the service in terms of security etc.
It is possible to allow a docker container to access well defined parts of the hosting environment.
I could give you some docker training but my time is currently quite limited.

[SirVer]

This discussion has died down a bit, but it becomes a bit more pressing now - 18.04 enters security fixes only mode. We should update soon. @hessenfarmer Any plans on how we should coordinate this update?

[hessenfarmer]

Not from my side as I am neither a Linux nor Server expert. I believe @frankystone and @janus are the server guys. And you of course

[stonerl]

@SirVer

You should make a backup of the VM first, in case the update screws something up and we need to reverse to the prior state.

Then I recommend a 18.04->20.04->22.04 upgrade.

I have some spare time at the end of August. So I could take care if this.

I just need some briefing on the changes e.g. the add-on server since I haven't had much time to follow the development lately.

[frankystone]

@hessenfarmer I am not a server guy. Working on the server makes me nervous and some times i am close to a heart attack ;)

Sorry that i didn't mentioned it earlier: I have no interest in learning docker. And from what i know @janus isn't interested in docker also. We may can do the changes to upgrade the website related stuff if it is independent from docker. From my side possibly in the next winter time. The only thing what is needed is the target python version.

[SirVer]

@stonerl @frankystone Thanks for volunteering. I do not think we need docker per se, I am just very short on time these days. @stonerl it would be great if you could do the server update. I have nightly backups via rsync of the server, so we should be good to go any time.

[SirVer]

I am planning to do the update myself on the weekend of 1st & 2nd October. @hessenfarmer Could you announce this on the website? It will likely be down for long stretches of time during the weekend, so will be the add on server & the game server.

[hessenfarmer]

Announcement is on the website

[blotter]

#384

[SirVer]

So, this seems done now. I did the following things:

• I upgraded the server to 20.04 LTS, and then once again to 22.04 LTS.
• I upgraded the website to use django 2.2. Current django is 4.0, but it seemed daunting to upgrade 2 full versions, so I made only one version jump - which was also necessary, since django 1 does not work with python 3.10, and that is the python that ships with 22.04 LTS (Port to Python 3.10 and Django 2.2  #384)
• I replaced the virtual envs for wlwebsite & alpha on the server with new ones & copied the media folder over.
• @frankystone and @blotter helped testing the changes, we found a couple of bugs that we fixed.
• I made sure that ./manage.py test is green again. It was not in a long time and I found a lot of really outdated tests. (Server upgrade part2 #387)
• I upgraded the remaining security relevant dependencies. I checked the differences and they seemed not consequential to us, i.e. we use very little of them anyways. So this upgrade feels fairly safe (Upgrade django-regstration and django-filter. #388).
• I checked that the metaserver is still running and reachable.
• I ensured the PPA for widelands is still working, so that the binaries can update themselves.

I did not test the extension server, since I have no knowledge how to.

Overall, the website seems to be running stable again, however this was quite a few changes in a short amount of time, so I expect that we will have more problems going forward. Please keep your eyes peeled and inform me about any bugs you encounter.

[frankystone]

You are unbelievable… so many changes in a few hours…

Thanks a lot!

[SirVer]

It was actually ~10 hours, so not that slow. I just put my back in :)

[frankystone]

Just saw we are now on Django 2.2.28 which is already EOL… so when do you find some more few hours to upgrade to django 3.2? ;)

[SirVer]

@frankystone I fear upgrading to Django 3 (or even 4, since this is out) is much more work.

[hessenfarmer]

many thanks for doing this hard work

[hessenfarmer]

Ooops, I don't know whether this is related but I can't login to the Metaserver anymore. (I changed my password to test this)

[hessenfarmer]

here is the log for trying to connect to the addon server
stdout.txt

[Noordfrees]

[Wed Oct 05 13:31:07 CEST 2022 @ Client#000635] Username: hessenfarmer
[Wed Oct 05 13:31:07 CEST 2022 @ Client#000635] Widelands: 1.1~git25864 (b301eaa@master)
[Wed Oct 05 13:31:07 CEST 2022 @ Client#000635] ERROR: java.lang.IllegalArgumentException: Illegal base64 character 27
java.lang.IllegalArgumentException: Illegal base64 character 27
        at java.base/java.util.Base64$Decoder.decode0(Base64.java:847)
        at java.base/java.util.Base64$Decoder.decode(Base64.java:566)
        at java.base/java.util.Base64$Decoder.decode(Base64.java:589)
        at wl.server.ClientThread.run(ClientThread.java:113)
        at java.base/java.lang.Thread.run(Thread.java:833)

Perhaps Base64 encoding for newly changed passwords is broken on the website now?

[Noordfrees]

Yep, the database now stores new ggz passwords as

b'foobarbaz'

instead of

foobarbaz

That's a bug/regression in the website

[SirVer]

@Noordfrees great find. @hessenfarmer thanks about being persistent about the issue. I fixed the root cause in #390 and I updated the database to have only correctly encoded WLGGZ password fields. The code is deployed already again. Could you test again, please?

The rootcause was that base64 now encodes into bytes instead of a string and we need to convert to a string manually. I had similar issues with the notifications module that I fixed the other day. I grepped the code base and believe this particular issue will not bite us anywhere else.

[hessenfarmer]

fix confirmed. thanks a lot

[frankystone]

The addon-server has to be restarted. @Noordfrees ?

[klaus-halfmann]

Thanks for all the work!

[frankystone]

The search functionality does not work correct. I guess the indexes have to be reinitialized. I would be glad if someone else can do this?

Probably this can be fixed this way:

• Enable virtualenvironment
• run ./manage.py rebuild_index

this will take round about 10 to 15 Minutes (if i remember correctly)

Reference: https://django-haystack.readthedocs.io/en/master/management_commands.html#rebuild-index

[SirVer]

still running....

[SirVer]

Giving up on waiting for now. I checked that maps show up in the search - that has been reindexed already. It currently indexes 35859 Posts.... I'll check in tomorrow.

[SirVer]

... so my brain is working weird, but it just informed me that 35859 posts/(13 years of website being in service ×365 days in a year) ~ 8 posts per day. We have posted 8 times per day in our forums over the last 13 years. Quite active all things considered...

Would be interesting to see a plot of activity over time.

[SirVer]

oh... it's done now. Took ~ 1hour.

Can somebody check if everything is working ok now?

real 56m9.682s
user 37m50.089s
sys 19m25.982s

[frankystone]

Looks my guess about the needed time was terrible wrong :-D

The search works again. Many thanks!

[frankystone]

Since there is a docker_compose.yml in the source:

Does the website now runs in a docker container?

[SirVer]

@frankystone no, that was just a useful way for me to test the website against actual data - it says so also in the comment in the file. I did not change how the website is running and is deployed, i.e. a naked virtualenv as before.

[SirVer]

Overall, I tried to change as little as possible.

[Noordfrees]

Another regression: Replying to news comments seems to be broken. In this thread I tried to post my comment as a reply to the first comment but the Reply button doesn't do anything.

[SirVer]

@Noordfrees I can confirm that this in not working. However it is working due to the javascript function not being defined. I have no explanation how this could come from my change.

I will not have time to look into this for a few weeks at least. If there are any takers for this bug, I'd appreciate that.

[blotter]

https://github.com/widelands/widelands-website/blob/master/threadedcomments/templates/threadedcomments/inlines/reply_to.js#L31

• "{% csrf_token %}" -> + '{% csrf_token %}'

ich glaube das sollte es beheben. bei mir im debug tat es das :)

tante edit:
ich habe es auf alpha probiert. tut was es soll.

[frankystone]

Looks like your changes are also applied to the productive website.

I'll commit this change.