How to access package data files with Databricks integration

[sim-uw]

Is it possible to access package data files when running code on Databricks using the step launcher?

I've set up data files in our package like this: https://setuptools.pypa.io/en/latest/userguide/datafiles.html#subdirectory-for-data-files - using the "Subdirectory for Data Files" approach.

It's then possible to access them like this:

from importlib.resources import files
data_text = files('mypkg.data').joinpath('data1.txt').read_text()

I guess this works because I have our package installed locally for development using pip install -e .. However I'm not sure the package gets installed like this by Dagster when it gets uploaded to Databricks? (using the local_pipeline_package_path option).

Is this approach to data files possible with the Dagster Databricks integration? It's a little tricky to debug remotely. I just want to make sure it's not a dead end before investing more time trying to get it to work.

Update with code:

The following asset code works when run without any IO manager: (spectacle is the module we pass to Dagster as module_name)

    package_path = files('spectacle.data')
    print(package_path)
    path = package_path / 'config_test.csv'
    print(path)

When I add our Databricks step launcher IO manager the code errors on the files(...) line. This is the error:

dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "config_test":

  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dagster/_core/execution/plan/execute_plan.py", line 282, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dagster/_core/execution/plan/execute_step.py", line 521, in core_dagster_event_sequence_for_step
    for user_event in _step_output_error_checked_user_event_sequence(
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dagster/_core/execution/plan/execute_step.py", line 202, in _step_output_error_checked_user_event_sequence
    for user_event in user_event_sequence:
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dagster/_core/execution/plan/execute_step.py", line 100, in _process_asset_results_to_events
    for user_event in user_event_sequence:
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dagster/_core/execution/plan/compute.py", line 208, in execute_core_compute
    for step_output in _yield_compute_results(step_context, inputs, compute_fn, compute_context):
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dagster/_core/execution/plan/compute.py", line 177, in _yield_compute_results
    for event in iterate_with_context(
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dagster/_utils/__init__.py", line 466, in iterate_with_context
    return
  File "/usr/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dagster/_core/execution/plan/utils.py", line 84, in op_execution_error_boundary
    raise error_cls(

The above exception was caused by the following exception:
TypeError: expected str, bytes or os.PathLike object, not NoneType

  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dagster/_utils/__init__.py", line 464, in iterate_with_context
    next_output = next(iterator)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dagster/_core/execution/plan/compute_generator.py", line 141, in _coerce_op_compute_fn_to_iterator
    result = invoke_compute_fn(
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dagster/_core/execution/plan/compute_generator.py", line 129, in invoke_compute_fn
    return fn(context, **args_to_pass) if context_arg_provided else fn(**args_to_pass)
  File "/tmp/tmpiu6bhgj0/spectacle/assets/archetype_identifiers.py", line 29, in config_test
    package_path = files('spectacle.data')
  File "/usr/lib/python3.9/importlib/resources.py", line 147, in files
    return _common.from_package(_get_package(package))
  File "/usr/lib/python3.9/importlib/_common.py", line 14, in from_package
    return fallback_resources(package.__spec__)
  File "/usr/lib/python3.9/importlib/_common.py", line 18, in fallback_resources
    package_directory = pathlib.Path(spec.origin).parent
  File "/usr/lib/python3.9/pathlib.py", line 1072, in __new__
    self = cls._from_parts(args, init=False)
  File "/usr/lib/python3.9/pathlib.py", line 697, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/usr/lib/python3.9/pathlib.py", line 681, in _parse_args
    a = os.fspath(a)

[zyd14]

So the package that gets pushed to Databricks doesn't get installed in the pip install sense of the word. It just gets copied into the site packages. So if you data directory is under the path you specified with local_pipeline_package_path then it should be available through normal os manipulation methods. You could get the base directory of where your package is located on the databricks cluster with something like

from mypackage import __file__
import os
package_dir = os.path.dirname(__file__)
data_dir = os.path.join(package_dir, <relative_path_to_data_dir>)

[sim-uw]

Ah, that's a nice trick to get the path to the data directory.

I think this can work, even with Dagster Serverless and its Github deploy action.

Will take it for a spin.

Thanks!