Native Python types

[Kilo59]

I'm brand new to Dagster so apologies if this has been answered and I didn't see it.
But are there plans to support native python types and compound types in the type system, in the model of FastAPI/pydantic?

https://fastapi.tiangolo.com/features/#just-modern-python
https://pydantic-docs.helpmanual.io/

I actually initially ran into this project a few months ago and believe it or not the thing that prevented me from evaluating it more seriously at the time was seeing things like this...

from dagster import String, Bool, Any

or

def is_list_of_dicts(_, value):
    return isinstance(value, list) and all(
        isinstance(element, dict) for element in value
    )


SimpleDataFrame = DagsterType(
    name="SimpleDataFrame",
    type_check_fn=is_list_of_dicts,
    description="A naive representation of a data frame, e.g., as returned by csv.DictReader.",
)

Whereas in pydantic it's as simple as adding a compound type hint.

cereals: typing.List[dict]

I realize there's a lot of metadata being included there as well.
Similar to a pydantic Field

I'm not asking for anyone to sell me on dagster (I've come around).
But If this is something that seems feasible, I'd like to help.

Perhaps we could even shift some of the typing logic to pydantic?
https://pydantic-docs.helpmanual.io/usage/validation_decorator/

Maybe a first step would just be to make the UI understand and display field data from a pydantic model.

[sryza]

Hi @Kilo59,

As of Dagster 0.11.0, arguments to solids (and return values) can be annotated using any valid type annotation. E.g. you can do:

from typing import List

@solid
def my_solid(_, my_arg: List[dict]):
    pass

Is that what you're looking for?

[Kilo59]

I'm aware of this.

However, I still don't understand the point of a dagster.String or a dagster.Bool.
Why do you have your own types for builtin python types? What do they add?

Why doesn't the documentation tell me to just use a native typing.List? etc.

[schrockn]

There are a bunch of features attached to the type system, including interaction with our configuration system, the ability to emit structured metadata, and so forth. The representations of built-ins in the type system are largely for our own internals at this point and the python built-in equivalents can be used in their stead.

[slamer59]

Hello,
I am also intersted. So I tried directly with:

import requests
import csv
from dagster import pipeline, solid
from pydantic import BaseModel, FilePath
import csv


class Cereal(BaseModel):
    file: FilePath

@solid
def contruct_cereal() -> Cereal:
    return Cereal(file='cereal.csv')

@solid
def hello_cereal(context, cereal: Cereal):
    context.log.info(f"Processing {cereal.file}")
    with open(cereal.file, newline='') as csvfile:
        reader = csv.reader(csvfile)
        cereals = [row for row in reader]
    context.log.info(f"Found {len(cereals)} cereals")

    return cereals

@pipeline
def hello_cereal_pipeline():
    hello_cereal(contruct_cereal())

And it works ? No ?

Dagster fails if returned type is not Pydantic type.
Pydantic fails if file attributes is not a FilePath.

What would interest me is to propagate Pydantic failure to log without.

Thanks

[sryza]

Hi @slamer59 - I would expect the code you listed to work. Would you mind explaining what you mean by "What would interest me is to propagate Pydantic failure to log without." in a little more detail?

[slamer59]

Hello, the code is working when you have cereal.csv in the root folder.
It should fail if Pydantic doesnot find it.
file or directory at path "cereal.csv" does not exist (type=value_error.path.not_exists; path=cereal.csv)
Now this usefull tip is hidden. Make it appears in dagster log would help.

Propagate this error nicely to the dagster logs would be nice. I will have a look here I think:

https://docs.dagster.io/concepts/logging/loggers