Support high-availability setups for the interactive tools proxy

[kysrpex]

This PR closes #16138. At the moment, information exchanges between the interactive tools proxy and Galaxy are done via a SQLite database. This makes it impossible to have the high-availability (redundant) setup for the proxy that we would like to have at useGalaxy.eu.

To solve the problem, I opted for a relatively simple solution. Since Galaxy already uses SQLAlchemy, I refactored the existing class InteractiveToolSqlite from lib/galaxy/managers/interactivetool.py to use SQLAlchemy instead of Python's built-in sqlite3 library. The changes are backwards compatible. Setting interactivetools_map to a path makes Galaxy use a SQLite database, just like it did before. Setting it to a SQLAlchemy database url allows to use any database supported by SQLAlchemy (e.g. PostgreSQL). Mappings are written to the table "gxitproxy", just as they did before.

Of course, this also needs support on the interactive tools proxy's side, which will be contributed separately.

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

This is a refactor of the "propagator" InteractiveToolSqlite in lib/galaxy/managers/interactivetool.py. Tests are already available and can be run using
./run_tests.sh -integration test/integration/test_interactivetools_api.py::TestInteractiveToolsIntegration. I used standard SQL so that the code is compatible with the widest possible variety of databases.

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[kysrpex]

I feel like I may be missing something regarding the FileLock. What role was it actually playing? Isn't SQLite already taking care of locks?

The pager module effectively controls access for separate threads, or separate processes, or both. Throughout this document whenever the word "process" is written you may substitute the word "thread" without changing the truth of the statement.

[bgruening]

Great stuff @kysrpex!

@sveinugu that might also be interesting for you.

[sveinugu]

Sounds good!

Just make sure that the info field keeps supporting json, as I opted for storing extra attributes to the proxy that are used for path-based interactive tools as json in the "info" field instead of adding new fields to the schema (the info field is not used for SQL queries, so I don't think there is a need to explicitly type this as JSON, plain text should be fine).

Currently the path-based ITs were reverted in the EU instance due to an error. I made a PR to revert this for the 24.1 release, now merged: usegalaxy-eu#242

So just make sure that the path-based ITs still run (first commit of that PR)!

[kysrpex]

@sveinugu Regarding the JSON, it's stored as plain text. Seems to work fine (see the psql output below from a Galaxy instance).

galaxy=> select * from gxitproxy;
      key      |         key_type          |     token     |      host      | port  |                                  info                                  
---------------+---------------------------+---------------+----------------+-------+------------------------------------------------------------------------
 326unmiv5sanl | interactivetoolentrypoint | 2jlcp7mzyjsbm | 192.168.122.64 | 32780 | {"requires_path_in_url": false, "requires_path_in_header_named": null}
(1 row)

I'll test the path-based ITs and let you know if they're working.

[kysrpex]

@sveinugu path-based ITs cause a redirection loop (I am using gx-it-proxy from galaxyproject/gx-it-proxy#21).

galaxy=> select * from gxitproxy;
      key      |         key_type          |     token     |      host      | port  |                                 info                                  
---------------+---------------------------+---------------+----------------+-------+-----------------------------------------------------------------------
 1540u2qksgef3 | interactivetoolentrypoint | 2ypyg6q4fsp20 | 192.168.122.64 | 32782 | {"requires_path_in_url": true, "requires_path_in_header_named": null}
(1 row)

Should they be working? Have you encountered this before? This may be a misconfiguration of my Galaxy instance, as I did not change what the proxy does with the sessions, just where it stores them.