Dagster and GCP Authentication

[slopp]

Using GCP services with Dagster requires authentication. How do you handle authenticating to GCP services in Dagster?

One approach:

1. Create a service account in GCP and assign it the proper roles / permissions. Download the service account json file.
2. Encode this JSON file as a string that can pointed to by a single environment variable.
3. In Dagster's entrypoint (eg the file where you define Definitions), decode the environment variable back into a json file on disk. Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to point at this file. Once set, most Google clients will just work without further configuration.

Step 2:

export GCP_CREDS_JSON_CREDS_BASE64="$(cat $GOOGLE_AUTH_CREDENTIALS | base64)"

Step 3:

import os
import json 
import base64

AUTH_FILE = "/tmp/gcp_creds.json"
with open(AUTH_FILE, "w") as f:
    json.dump(json.loads(base64.b64decode(os.getenv("GCP_CREDS_JSON_CREDS_BASE64"))), f)

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = AUTH_FILE

A benefit to this approach is that it allows you to add the encoded GCP credentials as an environment variable either locally or in Dagster Cloud.

[slopp]

Answer in first post! (See the "Google Workload Identity" suggestion below as well)

[shalabhc]

Does setting the GCP_CREDS_JSON_CREDS_BASE64 environment in dagster cloud require first adding it to the "Environment Variables" in the cloud UI? Maybe this should be added as Step 2.1

[hec10r]

This worked for me:

docker run \
  --network=dagster_cloud_agent \
  --volume $PWD/dagster.yaml:/opt/dagster/app/dagster.yaml:ro \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  -e GCP_CREDS_JSON_CREDS_BASE64=<variable value> \
  -it [docker.io/dagster/dagster-cloud-agent:latest](http://docker.io/dagster/dagster-cloud-agent:latest) \
  dagster-cloud agent run /opt/dagster/app

[hypr-platform]

thanks @slopp, it worked here

you know why we just can't use os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "service_account.json" on defs file? when i do that, dagster ignore it.

[slopp]

@hypr-platform where are you including that file, is it in the image you are building, or your Git repo alongside your application code. My guess is that the file is not where you expect and you might need to supply a full path instead of just the file name

[AndreaGiardini]

I would instead use Google workload identity. It's much easier to implement and maintain.

With Google workload identity you can assign GCP permissions to Kubernetes service accounts. It's not necessary to deal with keys and JSON files. As long as dagster is using the right Kubernetes service account, it will have the right permissions.

[slopp]

Thanks @AndreaGiardini I agree that is a great solution for Dagster Cloud Hybrid or OSS where you control the execution plane. For Dagster Cloud Serverless we do not currently provide an option for WI

[davejroth]

@slopp thanks so much for this guide. in step 3, how do you view the b64 encoded string to paste it into a dagster cloud environment variable? I tried using cat and printf to write the b64 string to a file but the code above throws a decoding error when I try to use it. It works if I don't change the env variable set by export GCP_CREDS_JSON_CREDS_BASE64="$(cat $GOOGLE_AUTH_CREDENTIALS | base64)" but I'm not sure how to get this into dagster cloud

[davejroth]

nvm I figured out that base64 encoded strings have new lines embedded within them and we can wrap them in double quotes to get them into dagster cloud

[noahtf13]

@slopp How do you test with this? This works fine locally and on serverless but we don't want to expose our usual environment to our test suite and this runs eagerly.

[slopp]

you can usually parameterize resources based on the runtime environment, eg https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/hooli_data_eng/resources/__init__.py#L67