Controlling cpu and memory resources for auto-materializing assets without jobs

[Daniel-Vetter-Coverwhale]

EDIT:
It looks like you can set an executor at the user code location level - https://docs.dagster.io/deployment/executors#for-a-code-location - and that it should apply to each job and asset. So I just need to do that, with some conditional logic for if the asset or job is being executed locally. It might be nice for this to be a deployment level setting like the run launcher, so that you don't need to worry about it at the code location level, but for now I can just add this to all of my code locations.

With everything else being so nicely wrapped around assets at this point, it kind of stinks to have to write a separate job to get control of resources, especially for jobs that would only include that single asset.

It would be great if we could use the same op_tags structure, or even better some class or typed_dictionary specific to resources, directly in the asset decorator without needing to wrap the asset in a job.

I found an old slack thread about this, and I also found a new more recent slack thread with no activity at the moment in ask-community, but figured I would start a discussion here to increase visibility and help better keep track of the issue/feature request.

Current op_tags

op_tags={
        "dagster-k8s/config": {
            "container_config": {
                "resources": {
                    "requests": {"cpu": "4", "memory": "8G"},
                    "limits": {"cpu": "4", "memory": "16G"},
                }
            },
        }
    }

Link to slack discussion from last year-ish - https://discuss.dagster.io/t/13163713/hey-there-i-am-trying-to-apply-a-tag-to-control-the-resource

Link to the more recent slack thread - https://dagster.slack.com/archives/C01U954MEER/p1709716088379659.

[psarka]

Appologies if I'm misunderstanding, but @asset decorator does take op_tags argument (CTRL+F for op_tags here: https://docs.dagster.io/_apidocs/assets#assets)

The structure is the same as the one you pasted, I'm using it in k8s and everything works.

[Daniel-Vetter-Coverwhale]

Your auto-materializing assets, not wrapped in a job with k8s_job_executor, are correctly requesting the resources you specify in op_tags on the asset?

Reading over this I don't think I was clear enough. I did mention in the first sentence about having to wrap the job, but then I later did incorrectly state that I would like to use the op_tags directly in the decorator, which is already possible, though they have no effect without the wrapping job, at least from what I can see.

Also, the response in this issue further validates the claim that you need a job for it to work -

"Note that setting the dagster-k8s/config tag on an asset will only work if you're using the k8s_job_executor that materializes each asset in its own Kubernetes pod. Setting dagster-k8s/config on a job will only work if you're using the K8sRunLauncher that runs each job in its own pod." - #14544

[Daniel-Vetter-Coverwhale]

Although now that I'm looking this up again, it seems like there is a way to specify an executor for every job and asset in a code location - https://docs.dagster.io/deployment/executors#for-a-code-location. So I guess that's what I need to do, and then my auto materializing assets should also use that. Just wrap that in a check for which run launcher is being used I suppose, to avoid issues when running locally

[psarka]

That's exactly what I'm doing 👍