My Dagster code is hanging or taking longer than I expect to execute. How can I troubleshoot?

[gibsondan]

• An op is taking longer than I would expect to run, how can I find the bottleneck?
• A dagster run is hanging and I want to know where it's stuck
• A code server has become unresponsive but doesn't seem to be memory or CPU-bound, why is it stuck?

[gibsondan]

We recommend that you use py-spy to profile your code.

For hanging code, py-spy dump can give you a dump of each thread, which usually makes it immediately clear where the hang is happening.

For slow code, py-spy record can produce an SVG file that gives you a flame graph of where the process is spending the most time. (We recommend py-spy record -f speedscope --idle to produce speedscope profiles, and to include idle CPU time in the results)

Note that py-spy usually requires elevated permissions in order to run.

--

A typical workflow for generating a py-spy dump for a hanging run in Kubernetes:

a) Set up your Dagster deployment so that each run pod is using a security context that can run py-spy https://github.com/benfred/py-spy#how-do-i-run-py-spy-in-kubernetes

For example, you can set this in your dagster_cloud.yaml file for your code location if you're running a Kubernetes agent to make both your code servers and run pods able to work with py-spy:

locations:
  - location_name: cloud-examples
    image: dagster/dagster-cloud-examples:latest
    code_source:
      package_name: dagster_cloud_examples
    container_context:
      k8s:
        server_k8s_config: # Raw kubernetes config for code servers launched by the agent
          container_config:
            securityContext:
              capabilities:
                add:
                  - SYS_PTRACE
        run_k8s_config: # Raw kubernetes config for runs launched by the agent
          container_config:
            securityContext:
              capabilities:
                add:
                  - SYS_PTRACE

Or if you're using the Dagster open-source helm chart, you can configure the run launcher to launch each run with

runLauncher:
  type: K8sRunLauncher
  config:
    k8sRunLauncher:
      runK8sConfig:
        containerConfig:
          securityContext:
            capabilities:
              add:
                - SYS_PTRACE

(Note that this gives the pod elevated permissions - check with your cluster admins to make sure this is an acceptable change to make temporarily)

See https://docs.dagster.io/dagster-cloud/deployment/agents/kubernetes/configuration-reference#per-location-configuration (Cloud) and https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#per-job-or-per-op-kubernetes-configuration (OSS) for more information on how to apply these types of customizations to your Kubernetes pods.

b) Launch a run and wait until it hangs

c) Check the event logs for the run to find the run pod, then kubectl exec into the pod to run py-spy:

kubectl exec -it <pod name here> /bin/bash

d) install py-spy, then run it:

pip install py-spy
py-spy dump --pid 1

This should output a dump of what each thread in the process is doing.

[abi-mutinex]

@gibsondan if I want to see what's taking my code location ~600s to load, can I use this? Where would I add it?

[geoHeil]

How would this work for the docker based deployment?
I I need to roll out py-spy by default?
Or would I manually have to connect to the spun up container and install it on the fly?

Would it be possible to reuse the https://dagster.slack.com/archives/C02LJ7G0LAZ/p1686306297612339 special tag to launch the run in a persistently running container (where I can prepare the install of pyspy)?

[gibsondan]

py-spy has instructions for running in docker as well: https://github.com/benfred/py-spy#how-do-i-run-py-spy-in-docker

this would work similarly as the k8s instructions above but with something like:

locations:
  - location_name: cloud-examples
    image: dagster/dagster-cloud-examples:latest
    code_source:
      package_name: dagster_cloud_examples
    container_context:
      docker:
        container_kwargs:
          cap_add:
            - SYS_PTRACE