Migration to Debian 12

[fgalan]

• Code changes
• Check unit test in Debian 12 locally
• Check ftest in Debian 12 locally (except the usual 3001_mqtt/mqtt_subscription_multibroker.test glith)
• Dockerfile (test build locally) (docker build -t prueba --build-arg GIT_REV_ORION=hardening/debian12-migration .) - 224ace3
• Check valgrind run in Debian 12 locally - detail here
• Check coverage in Debian 12 locally - problems found doing this, but not related with this PR istelf (moved to issue ftest coverage functionality is broken #4418), good enough with this result
• CI Dockerfile (test build locally) - 2d870f7
• Test after rebuilding CI with the changes in CI Dockerfile (use manual build on dockerhub) (*) - at c3c827d tests are passing fine
• Documentation
• CNR
• (post merge) relaunch tests in PR [WIP] MongoDB 5.0 #4331, PR MongoDB 6.0 #4332 and PR [WIP] MongoDB 7.0 #4414 to check regression in Debian 12

(*) Up to this point, regression in this PR commits is done using preexisting Debian 11 based CI image

[fgalan]

At commit b65186d, valgrind pass seems ok:

test/valgrind/valgrindTestSuite.sh
lun 04 sep 2023 18:37:35 CEST
Run tests 0 to 0
Test 001/1133: 0000_bad_requests/exit ............................................................................................................... OK (.07 seconds)
Test 002/1133: 0000_bad_requests/pagination_error_in_uri_params ..................................................................................... OK (11.04 seconds)
Test 003/1133: 0000_cli/bool_option_with_value ...................................................................................................... Not apt for valgrind - skipped
Test 004/1133: 0000_cli/command_line_options ........................................................................................................ Not apt for valgrind - skipped
Test 005/1133: 0000_cli/tracelevel_without_logLevel_as_DEBUG ........................................................................................ Not apt for valgrind - skipped
Test 006/1133: 0000_cli/version ..................................................................................................................... Not apt for valgrind - skipped
Test 007/1133: 0000_content_related_headers/accept_fail_01 .......................................................................................... OK (8.89 seconds)
Test 008/1133: 0000_content_related_headers/accept_fail ............................................................................................. OK (12.06 seconds)
Test 009/1133: 0000_content_related_headers/missing_content_type_header ............................................................................. OK (10.55 seconds)
Test 010/1133: 0000_deprecated_checkings/disable_ngsi_cli ........................................................................................... OK (9.47 seconds)
Test 011/1133: 0000_deprecated_checkings/geoquery_circle_deprecated ................................................................................. OK (15.94 seconds)
Test 012/1133: 0000_deprecated_checkings/get_entity_dates_with_options .............................................................................. OK (47.74 seconds) (no leak but ftest error 1)
Test 013/1133: 0000_deprecated_checkings/legacy_geoquery_bad_coords ................................................................................. OK (12.05 seconds)
Test 014/1133: 0000_deprecated_checkings/legacy_geoquery_circle_json ................................................................................ OK (16.65 seconds)
Test 015/1133: 0000_deprecated_checkings/legacy_geoquery_polygon_json ............................................................................... OK (17.42 seconds)
Test 016/1133: 0000_deprecated_checkings/legacy_location_no_actual_location_change .................................................................. OK (11.07 seconds)
Test 017/1133: 0000_deprecated_checkings/legacy_wgs84 ............................................................................................... OK (15.70 seconds)
Test 018/1133: 0000_deprecated_checkings/log_deprecate_warning_dynamic .............................................................................. OK (10.67 seconds)
Test 019/1133: 0000_deprecated_checkings/log_deprecate_warning ...................................................................................... OK (11.65 seconds)
Test 020/1133: 0000_deprecated_checkings/metadata_id_as_regular_metadata ............................................................................ OK (12.06 seconds)
Test 021/1133: 0000_deprecated_checkings/metadata_id_duplicate_error ................................................................................ OK (11.11 seconds)
Test 022/1133: 0000_deprecated_checkings/uppercase_action_types_in_ngsiv2_batch_update .............................................................. OK (13.17 seconds)
Test 023/1133: 0000_deprecated_checkings/xml_support ................................................................................................ OK (8.75 seconds)
Test 024/1133: 0000_ipv6_support/ipv4_only .......................................................................................................... OK (.05 seconds)
Test 025/1133: 0000_ipv6_support/ipv6_only .......................................................................................................... OK (.07 seconds)
Test 026/1133: 0000_json_parse/jsonParseGetNotifyContext ............................................................................................ OK (9.15 seconds)
Test 027/1133: 0000_json_parse/jsonParsePostQueryContext ............................................................................................ OK (11.23 seconds)
Test 028/1133: 0000_large_requests/large_query_context .............................................................................................. OK (10.81 seconds)
Test 029/1133: 0000_large_requests/large_update_context ............................................................................................. OK (9.18 seconds)
...

A full pass has not been done, given it takes a long time, it will be done by GitActions once this PR gets merged.

[fgalan]

$ make coverage_unit_test
...
Overall coverage rate:
  lines......: 21.4% (4888 of 22891 lines)
  functions..: 41.9% (753 of 1799 functions)

The actual results (pretty low) is not relevant (this is due we are no relying in unit test too much since MongoDB driver changed time ago in Orion 3.0.0). What is relevant is that the coverage tooling works.

[fgalan]

Not actually related with this PR, but found while I was investigating make coverage issues.

NTC (informative)

[fgalan]

At commit 2d870f7 first run is with preexisting Debian 11 image

Let's build now the new Debian 12 based image in dockerhub and see how it goes...

[fgalan]

After some problem with dockerhub, the new image was finally build from commit 2d870f7 in this branch:

Launching new pass of tests...

[fgalan]

Fails in cases/1705_csub_cache_objects/csub_cache_objects_errors.test due to curl CLI has changed:

Debian 11:

$ curl 'http://localhost:9999/v2/entities?q=a==12;    '
{"error":"BadRequest","description":"empty q-item"}

$ curl --version
curl 7.74.0 (x86_64-pc-linux-gnu) libcurl/7.74.0 OpenSSL/1.1.1n zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.9.0 nghttp2/1.43.0 librtmp/2.3
Release-Date: 2020-12-09
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

Debian 12:

$ curl 'http://localhost:9999/v2/entities?q=a==12;    '
curl: (3) URL using bad/illegal format or missing URL

$ curl --version
curl 7.88.1 (x86_64-pc-linux-gnu) libcurl/7.88.1 OpenSSL/3.0.9 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 (+libidn2/2.3.3) libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3
Release-Date: 2023-02-20
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Thus, it seem curl has changed its behaviour among versions...

I have posted to SOF about this: https://stackoverflow.com/questions/77045755/avoid-curl-to-check-for-malformed-urls

[fgalan]

@fisuda, several .md files in the English version have been modified in this PR. I have partially modified some Japanese .md files to align, but only partially (mainly numeric information). It would be great if you could have a look and complete this synchronization work in a PR using hardening/debian12-migration (this PR's branch) as base branch, please.

With regards to the "Ubuntu 20.04 TLS" sections, it would be great if you could check if the procedure described there is still working after the changes of this PR but that could be of course accomplished in a separate PR (related with what discussed in this other PR.

Thanks in advance!

[fisuda]

I sent the PR #4419.

In the current version (before merging this PR), it seems that the Orion can be built with mongo driver 1.24.3 on Ubuntu 20.04.

As of now, Ubuntu LTS (Long Term Support) version is 22.04. So I'm trying to build Orion on Ubuntu 22.04. But I enounted same error as building on debian 12. Because It seems that the base distribution of Ubuntu 22.04.3 LTS is debian 12. Atfer this PR is merged, I'd like to try to build Orion again. And I will update the "Ubuntu 20.04 LTS" sections to the "Ubuntu 22.04 LTS" sections.

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ cat /etc/debian_version
bookworm/sid

Thanks

[fgalan]

As of now, Ubuntu LTS (Long Term Support) version is 22.04. So I'm trying to build Orion on Ubuntu 22.04. But I enounted same error as building on debian 12. Because It seems that the base distribution of Ubuntu 22.04.3 LTS is debian 12. Atfer this PR is merged, I'd like to try to build Orion again. And I will update the "Ubuntu 20.04 LTS" sections to the "Ubuntu 22.04 LTS" sections.

Sounds nice! :)

[mapedraza]

LGTM