partition by TopologicalIsland

[derrickoswald]

A possibility exists that partitioning according to TopologicalIsland may be beneficial for many use-cases.

Most electric distribution consists of radial networks, where the network downstream of a supply transformer is an isolated network that can be modeled independently, albeit with a single attachment to the grid at the transformer. The network topology processor identifies these "islands of connectivity" and generates one TopologicalIsland object, which all contained Terminal and ConnectivityNode objects reference.

In some cases ganged/parallel transformers are used to supply higher power requirements, so the phrase "a supply transformer" in the above description should be phrased as "a group of supply transformers that share a common bus".

The term Trafokreis (english: transformer circuit) is used to refer to these islands serviced by a transformer group.

Not all islands correspond to a Trafokreis. Notable exceptions are:

• middle and high voltage islands (voltage ranges higher than the 400V distribution network)
• fragments where a normally open switch isolates a set of equipment from the network
• public lighting isolated by a 400:230 volt transformer - which in reality is it's own trafokreis

For some analysis use-cases, these topological islands can be processed in parallel independent of each other. It may be possible to partition the CIM RDD classes, based on their topological island, so as to bring all relevant objects to the same machine for processing a priori.

So, for example, the CIM classes related to the island for transformer(s) X would be assigned to the Spark worker Y. The number of topological islands far exceeds the number of workers, so each worker would have several islands.

The RDD.coalesce() method takes an optional PartiionCoalescer which generates an array of PartitionGroups, each of which has an array of Partitions and their preferred location (machine name). But there doesn't seem to be a way to use anything except a HashPartitioner or RangePartitioner with the default RDD, so this will probably involve creating a subclass of RDD with the desired partitioning infrastructure.

The RDD.groupBy method takes an optional Partitioner that would allow the creation of individual RDD for each partition, but it's unclear how this would work.

In any case, partitioning like this would require a mapping table between CIM rdf:ID and partition (Trafokreis) for all elements. This would include items not in the topology, such as assets, locations, etc. It also probably requires a partition0 to contain elements shared between islands such as voltages and power system resource types, etc.