How to reduce initial process start time in various scenarios? (Pipes, in_process_executor)

[erenz14]

To preface this, my team has been using Dagster for a few months now and it has allowed us to build something truly awesome. We appreciate all of the innovations Dagster has added to the orchestrator community!

Now to my question...
We're experiencing overhead between the RUN_STARTING and RUN_START events when running jobs.

These are the following scenarios where we're seeing the process creation time overhead:

• Locally using in_process_executor
• In kubernetes using K8sRunLauncher
• In kubernetes with dagster pipes every time we call PipesK8sClient.run (Surprisingly, the PipesSubprocessClient actually doesn't create a new process)

The amount of overhead varies depending on the environment, but for example, when running a job locally, that contains 3 ops that each just print a string, and using in_process_executor, it had 12 seconds of overhead. By removing basically all imports the overhead is reduced to 5 seconds.

I have the following questions:

• Is ~5 seconds minimum of time between RUN_STARTING and RUN_START expected?
• Is there anything I can do aside from just removing imports (which is actually unrealistic in most scenarios I'm assuming) to reduce the overhead?
• I noticed the time between RUN_STARTING and RUN_START isn't included in the overall run completion time. Just want to make sure that is intended?
• When using PipesK8sClient.run, it launches a new pod, and even that new pod has the same process creation overhead. For us this becomes an issue when a job has multiple PipesK8sClient.run calls, because each call adds to the overhead. As far as i can tell this is unavoidable at the moment since, w/ pipes, in_process_executor and multiprocess_executor with forkserver don't help. Is there any way to avoid this process creation overhead each time PipesK8sClient creates a new pod?

Run logs with example overhead:

[newfrontier0007]

It will be useful to get a solution.