Supporting Spark

[xoolive]

ping @rmonstein

Start from scripts/spark_sample.py in the spark branch:

• maybe we should start with a SparkMixin (see core.mixins.DataFrameMixin) for building the TrafficSpark structure
• then try to send the thing and bring it back
• then apply basic spark functions
• then try to make the rest work... simple functions in traffic.py/flight.py
• then look at fancy stuff in navigation.py et al

[xoolive]

@rmonstein extra questions crossing my mind:

• how to handle the .cache() method on TrafficSpark.
  I am sure it's not that hard, but we should think about it;
• should we differentiate when the result we want to get is not a TrafficSpark but a simple Spark dataframe? Should we deal with it based on annotations (possible, but I don't like it...) or somehow differently? (but how?)

[rmonstein]

@xoolive
I'm back on the topic and tried to setup a simple container to play around in and I found out that github has its own container registry. Is that feature available on this repository?

If yes, it would make sense to generate the docker images from the docker files after a commit and upload the registry. Idealy, we would then have an image for:

• traffic with juypter
• traffic with poetry
• traffic with spark

Does that make sense to you?

[xoolive]

Yes indeed. We should rebase this branch on master because I started this process already.
Look here: https://github.com/xoolive/traffic/tree/master/docker

The plan was to add as many Dockerfile as we want here.

[xoolive]

Re: container registry, I don't know much about it, but I am open to any cool addition :)