Testing

Data test setup.

Data tests piggy-back on top of MyPy internal utilities.

To make them work properly we need:

• MYPY_TEST_PREFIX variable pointing to pyspark-stubs root.
• MYPYPATH variable point to third_party/3 subdirectory.

Combined we have to run pytest in a setup equivalent to

MYPY_TEST_PREFIX=$PWD MYPYPATH=$PWD/third_party/3 pytest

See Travis configuration for details.

Speeding-up data tests.

Unfortunately data test are very slow. However, they're not computationally intensive are parallelize well.

pytest-xdist does a pretty good job here. For example the following

MYPY_TEST_PREFIX=$PWD MYPYPATH=$PWD/third_party/3 pytest -n 8

will run tests with eight concurrent jobs.

Testing against Spark examples

CI pipeline tests the code against Spark examples.

To do that we first clone Spark and then patch in case of unsupported (won't fix) features.