Different python versions when compared with the 10.4 cluster

[kfot]

Hi,

The python dependencies` versions listed in the python dockerfile mismatch the versions listed in the release's documentation page. To be exact:

  ipython==7.19.0 \
  numpy==1.19.2 \

is not

ipython	7.22.0
numpy	1.20.1
--	--

Is such difference acceptable? If so, how one can guarantee a custom python package compatibility and which - cluster's or docker image's - dependencies' versions should be locked?

[jameslamb]

I have the same question, and can provide a bit more detail.

https://docs.databricks.com/release-notes/runtime/9.1.html claims that DBR 9.1 LTS uses Python 3.8.10.

The latest databricksruntime/python:9.1-LTS image has Python 3.8.0.

docker run \
    --rm \
    databricksruntime/python:9.1-LTS \
    /databricks/python3/bin/python3 --version

# Python 3.8.0

Can you please make the images match the documented contents of Databricks Runtime versions?