From efec15c4ad665e11d4a499289eca723a3b172f0a Mon Sep 17 00:00:00 2001 From: nab880 Date: Fri, 14 Oct 2022 12:33:34 -0700 Subject: [PATCH 1/8] Adding Ember documentation Adding ember documentation and examples --- src/sst/elements/ember/Makefile.am | 2 + .../elements/ember/example/CreatingMotifs.md | 116 ++++++++++ src/sst/elements/ember/example/QuickStart.md | 217 ++++++++++++++++++ .../elements/ember/example/RunningMotifs.md | 190 +++++++++++++++ src/sst/elements/ember/example/example.py | 67 ++++++ .../ember/example/motifs/emberexample.cc | 30 +++ .../ember/example/motifs/emberexample.h | 70 ++++++ 7 files changed, 692 insertions(+) create mode 100644 src/sst/elements/ember/example/CreatingMotifs.md create mode 100644 src/sst/elements/ember/example/QuickStart.md create mode 100644 src/sst/elements/ember/example/RunningMotifs.md create mode 100644 src/sst/elements/ember/example/example.py create mode 100644 src/sst/elements/ember/example/motifs/emberexample.cc create mode 100644 src/sst/elements/ember/example/motifs/emberexample.h diff --git a/src/sst/elements/ember/Makefile.am b/src/sst/elements/ember/Makefile.am index ab907fd870..37f3d62c29 100644 --- a/src/sst/elements/ember/Makefile.am +++ b/src/sst/elements/ember/Makefile.am @@ -12,6 +12,8 @@ compdir = $(pkglibdir) comp_LTLIBRARIES = libember.la libember_la_SOURCES = \ + example/emberexample.h + example/emberexample.cc embergen.h \ embergen.cc \ embermap.h \ diff --git a/src/sst/elements/ember/example/CreatingMotifs.md b/src/sst/elements/ember/example/CreatingMotifs.md new file mode 100644 index 0000000000..803b7629c2 --- /dev/null +++ b/src/sst/elements/ember/example/CreatingMotifs.md @@ -0,0 +1,116 @@ +# C++ Ember Motifs +## Ember + +[Ember](http://sst-simulator.org/SSTPages/SSTElementEmber/) is a library representing various network communications. It is often used in conjunction with motifs. + +## Motif + +Motifs are condensed, efficient generators for simulating different kinds of communication/computation patterns. +The motif presented here does no real work, but more detailed motifs can be found in sst-elements/src/sst/elements/ember/mpi/motifs/. + +Motifs are executed as follows: + +1) The motif generator is initialized (The constructor) +2) The generate function is invoked, places events on the event queue, and returns either true or false +3) The events on the event queue are processed. +4) If the generate function in step 2 returns false, go to step 2. Otherwise, the motif is complete. + +Motifs are intended to be run as a 'job submission.' +The generate function models an entire iteration of an application, using the event queue to mix compute and MPI operations in every iteration. + +### Register Subcomponent + +Ember motifs need to be registered as an SST subcomponent. This typically happens in the header file. +For example: +``` +SST_ELI_REGISTER_SUBCOMPONENT_DERIVED( + Example, + "ember", + "ExampleMotif", + SST_ELI_ELEMENT_VERSION(1,0,0), + "Performs an idle on the node. No traffic can be generated.", + SST::Ember::EmberGenerator + ); +``` + +The parameters of this function are: + +1) class associated with this Motif +3) Aspect of elements being used, in the case of Motifs, "ember." +3) Identifier of the Motif. Note that the end of this parameter must always be Motif +4) SST elements version, +5) Comment describing what the motif does +6) EmberGenerator + +### Writing a constructor +``` +EmberExample::EmberExample(SST::ComponentId_t id, Params& params) : + EmberMessagePassingGenerator(id, params, "Example"), + +``` + + +The constructor also reads parameters from the python script. +The params are passed through the python file in the form: `ep.addMotif("Example firstParam=100 secondParam=200")`. Note no space is allowed before or after the = operator. Parameters read from the python file will be prepended with "arg." before being passed to the C++ file. i.e. "firstParam" becomes "arg.firstParam". +The constructor can be used to perform the setup operations necessary for the generating function. + +The params can be parsed in the C++ file with `firstParam = params.find("arg.firstParam", 100);` where the name of the parameter is "firstParam". + +### Writing a generate() function +``` +bool Example::generate( std::queue& evQ) +``` +The generate function is the 'main' function of a Motif. +If the python file invokes an `addMotif`, the generating function will be invoked until the generate function returns true. +Once the generate function returns, the events queued in the evQ variable will be performed. + +The generate function takes an event queue as a parameter. The event queue allows the user to include computation operations and MPI events in the simulation. Motifs are intended to be designed so that every call to generate() mimics an entire iteration of the application. The events that can be added to the event queue are listed in subsequent sections. + +. + +# User-defined events + +These functions allow the user to control how the simulation estimates computation time. + +* `enQ_compute(Queue&, uint64_t nanoSecondDelay)` -- The delay by the expected cost of the compute operation in nanoSeconds (without actually computing it) +* `enQ_compute( Queue& q, std::function func)`; -- A function is passed as a parameter and invoked. It returns a 64-bit unsigned integer that indicates the simulation delay associated with invoking the function (in nanoseconds). +* `enQ_getTime(evQ, &m_startTime)` -- sets m_startTime to the current time + +There are two types of compute functions that can be enqueued. + +The first computation operation takes a 64-bit integer as a parameter. This parameter is the amount of time needed in nanoseconds to perform the computation. The simulator then delays by that number of nanoseconds as if the actual computation had taken place. +The second compute takes a function as a parameter. When the event is processed, the function is invoked. The function returns the amount of time needed to perform the compute operation. The simulation is delayed by the return value of the function. The time delay could be estimated through a heuristic or measured through representative computation. + + +# MPI Events + + +We list the supported MPI events below. For more detail, see MPI API documentation. + + +* `enQ_init(evQ)` -- MPI initialize +* `enQ_fini(evQ)` -- MPI finalize +* `enQ_rank(evQ, m_newComm[0], &m_new_rank)` -- MPI rank +* `enQ_size(evQ, m_newComm[0], &m_new_size)` -- MPI size +* `enQ_send(evQ, x_neg, x_xferSize, 0, GroupWorld)` -- MPI send +* `enQ_recv(evQ, x_neg, x_xferSize, 0, GroupWorld)` -- MPI recv +* `enQ_isend(evQ, next_comm_rank, itemsThisPhase, 0, GroupWorld, &requests[next_request])` -- MPI isend +* `enQ_irecv(evQ, xface_down, items_per_cell * sizeof_cell * ny * nz, 0, GroupWorld, &requests[nextRequest])` -- MPI irecv +* `enQ_cancel(evQ, m_req[i])` -- MPI cancel +* `enQ_sendrecv(evQ, m_sendBuf, m_messageSize, DATA_TYPE, destRank(), 0xdeadbeef, m_recvBuf, m_messageSize, DATA_TYPE, srcRank(), 0xdeadbeef, GroupWorld, &m_resp)` -- MPI send or recv +* `enQ_wait(evQ, req)` -- MPI wait +* `enQ_waitany(evQ, m_req.size(), &m_req[0], &m_indx, &m_resp)` -- MPI waitany +* `enQ_waitall(evQ, 1, &m_req, (MessageResponse**)&m_resp)` -- MPI waitall +* `enQ_barrier(evQ, GroupWorld)` -- MPI barrier +* `enQ_bcast(evQ, m_sendBuf, m_count, DOUBLE, m_root, GroupWorld)` -- MPI broadcast +* `enQ_scatter(evQ, m_sendBuf, m_sendCnt, m_sendDsp.data(), LONG, m_recvBuf, m_count, LONG, m_root, GroupWorld)` +* `enQ_scatterv(evQ, m_sendBuf, &m_sendCnts[0], m_sendDsp.data(), LONG, m_recvBuf, m_count, LONG, m_root, GroupWorld)` -- MPI Scatter variable message size +* `enQ_reduce( evQ, m_sendBuf, m_recvBuf, m_count, DOUBLE, MP::SUM, m_redRoot, GroupWorld)` +* `enQ_allreduce(evQ, m_sendBuf, m_recvBuf, m_count, DOUBLE, m_op, GroupWorld)` -- MPI allreduce function +* `enQ_alltoall(evQ, m_sendBuf, m_colSendCnt, &m_colSendDsp_f[0], DOUBLE, m_colComm)` -- MPI alltoall with constant message size +* `enQ_alltoallv(evQ, m_sendBuf, &m_colSendCnts_f[0], &m_colSendDsp_f[0], DOUBLE, m_colComm)` -- MPI alltoall with varying message sizes +* `enQ_allgather(evQ, m_sendBuf, m_count, INT, m_recvBuf, m_count, INT, GroupWorld)` -- MPI allgather with messages of constant size +* `enQ_allgatherv(evQ, m_sendBuf, m_sendCnt, INT, m_recvBuf, &m_recvCnts[0], &m_recvDsp[0], INT, GroupWorld)` -- MPI allgather with messages of varying sizes +* `enQ_commSplit(evQ, *comm, color, key, newComm)` -- Splits the MPI comm +* `enQ_commCreate(evQ, GroupWorld, m_rowGrpRanks, &m_rowComm)` -- Creates MPI Com +* `enQ_commDestroy(evQ, m_rowComm)` --Destroys MPI comm diff --git a/src/sst/elements/ember/example/QuickStart.md b/src/sst/elements/ember/example/QuickStart.md new file mode 100644 index 0000000000..6a3bcba64c --- /dev/null +++ b/src/sst/elements/ember/example/QuickStart.md @@ -0,0 +1,217 @@ +# What is an Ember Motif? + +Ember Motifs provide representative workloads and then estimate performance on hardware using SST simulations. +This guide provides a directions on how to: + +* Implement a empty motif. +* Create and run motif in a SST simulation. + +Implementing and running Ember Motifs requires three different files: + +1. Python File + Provides information on the hardware to be simulated when running an Ember Motif + This directs SST on the available hardware components and how they are set up. +2. C++ file + Decides how the computation is structured and computed. +3. C++ header file + The header file contains primarily naming information + +# Quick Start Guide + +This guide gives a simple example of how to create and run a simple Motif. + +First ensure SST-core and SST-elements are properly installed on your machine. For download and installation see [SST-Downloads](http://sst-simulator.org/SSTPages/SSTMainDownloads/). + +Navigate to `sst-elements/src/elements/ember` inside the sst-elements library. +First, create a new directory to hold our example Motif. + +`mkdir ExampleMotif` + +Next, create a `ExampleMotif/exampleMotif.h` file +``` + #ifndef _H_EMBER_EXAMPLE + #define _H_EMBER_EXAMPLE + + #include "../../mpi/embermpigen.h" + + namespace SST { + namespace Ember { + + class ExampleGenerator : public EmberMessagePassingGenerator { + public: + + SST_ELI_REGISTER_SUBCOMPONENT_DERIVED( + Example, + "ember", + "ExampleMotif", + SST_ELI_ELEMENT_VERSION(1,0,0), + "Performs an idle on the node, no traffic can be generated.", + SST::Ember::EmberGenerator + ); + + SST_ELI_DOCUMENT_PARAMS( + ); + + SST_ELI_DOCUMENT_STATISTICS( + { "time-Init", "Time spent in Init event", "ns", 0}, + { "time-Finalize", "Time spent in Finalize event", "ns", 0}, + { "time-Rank", "Time spent in Rank event", "ns", 0}, + { "time-Size", "Time spent in Size event", "ns", 0}, + { "time-Send", "Time spent in Recv event", "ns", 0}, + { "time-Recv", "Time spent in Recv event", "ns", 0}, + { "time-Irecv", "Time spent in Irecv event", "ns", 0}, + { "time-Isend", "Time spent in Isend event", "ns", 0}, + { "time-Wait", "Time spent in Wait event", "ns", 0}, + { "time-Waitall", "Time spent in Waitall event", "ns", 0}, + { "time-Waitany", "Time spent in Waitany event", "ns", 0}, + { "time-Compute", "Time spent in Compute event", "ns", 0}, + { "time-Barrier", "Time spent in Barrier event", "ns", 0}, + { "time-Alltoallv", "Time spent in Alltoallv event", "ns", 0}, + { "time-Alltoall", "Time spent in Alltoall event", "ns", 0}, + { "time-Allreduce", "Time spent in Allreduce event", "ns", 0}, + { "time-Reduce", "Time spent in Reduce event", "ns", 0}, + { "time-Bcast", "Time spent in Bcast event", "ns", 0}, + { "time-Gettime", "Time spent in Gettime event", "ns", 0}, + { "time-Commsplit", "Time spent in Commsplit event", "ns", 0}, + { "time-Commcreate", "Time spent in Commcreate event", "ns", 0}, + ); + + + }; + } + } + + #endif /* _H_EMBER_EXAMPLE */ +``` +This creates a Motif generator class called ExampleGenerator that inherits from EmberMessagePassingGenerator. The Python code we provide later creates and invokes the generate function. +The SST Document Statistics provides tracking for initialization and various MPI function calls. + +Motifs are executed as follows: + + + +1) The motif generator is initialized (The contructor) +2) The generate function is invoked and returns either true or false +3) The events on the eventQueue are processed. +4) If the generate function in step 2 returned false, return to step 2, otherwise the motif is complete. + +Here is a motif that returns without doing any additional work. + +Then we create a file `ExampleMotif/Example.cc` with the contents: + +``` + #include + #include "Example.h" + + using namespace SST:Ember; + + ExampleGenerator::ExampleGenerator(SST::ComponentId_t id, Params& params) : + EmberMessagePassingGenerator(id, params, "Null" ) + { + } + + bool ExampleGenerator::generate( std::queue& evQ) + { + // Code here is repeated until true is returned. + return true; + } +``` + + +Motifs are intended to be ran as a 'job submission'. +The generate function models an entire iteration of an application. The event queue allows the motif to mix compute and MPI operations in every iteration. + + + +The C++ file loads the SST:Ember namespace giving it access to other Motifs and EmberEvents. +Each Motif generator has an constructor and generate function. + + +Then we add the `.cc` and `.h` file to `Makefile.am` + +i.e. +``` + embermemoryev.h \ + ExampleMotif/example.h \ + ExampleMotif/example.cc \ + libs/emberLib.h \ +``` + +Finally, a python file `example.py` needs to be created: + +``` + from email.mime import base + import sst + from sst.merlin.base import * + from sst.merlin.endpoint import * + from sst.merlin.interface import * + from sst.merlin.topology import * + + from sst.ember import * + + def example(): + PlatformDefinition.setCurrentPlatform("firefly-defaults") + + ### Setup the topology + topo = topoDragonFly() + topo.hosts_per_router = 2 + topo.routers_per_group = 4 + topo.intergroup_links = 2 + topo.num_groups = 4 + topo.algorithm = ["minimal","adaptive-local"] + + # Set up the routers + router = hr_router() + router.link_bw = "4GB/s" + router.flit_size = "8B" + router.xbar_bw = "6GB/s" + router.input_latency = "20ns" + router.output_latency = "20ns" + router.input_buf_size = "4kB" + router.output_buf_size = "4kB" + router.num_vns = 2 + router.xbar_arb = "merlin.xbar_arb_lru" + + topo.router = router + topo.link_latency = "20ns" + + ### set up the endpoint + networkif = ReorderLinkControl() + networkif.link_bw = "4GB/s" + networkif.input_buf_size = "1kB" + networkif.output_buf_size = "1kB" + + ep = EmberMPIJob(0,topo.getNumNodes()) + ep.network_interface = networkif + ep.addMotif("Example") + ep.nic.nic2host_lat= "100ns" + + system = System() + system.setTopology(topo) + system.allocateNodes(ep,"linear") + + system.build() + + if __name__ == "__main__": + example() +``` + +This follows common python syntax. +The hardware variables (topology, router and network interface) are created through assignment of a constructor +The fields of the variables are accessed using a dot operator. +Topology, router, and network interface variables need to be created for a simulation. More detailed descriptions of configuration options is in RunningMotifs.md. +Next an endpoint is created and given the motif to be used using a ep.addMotif("MotifName")call +Note multiple motifs can be added the same endpoint. The intended use for each motif to simulate an entire application. The event queue is used to simulate MPI events and computational workloads. Adding multiple motifs is useful for simulating workflows of a series of applications. + +Finally, a system variable is created and 'built' + +Remake SST-elements. + +To run the python script + +``` +sst example.py +``` + + + diff --git a/src/sst/elements/ember/example/RunningMotifs.md b/src/sst/elements/ember/example/RunningMotifs.md new file mode 100644 index 0000000000..3244247705 --- /dev/null +++ b/src/sst/elements/ember/example/RunningMotifs.md @@ -0,0 +1,190 @@ +# Ember Python Configurations + +The Python file in Ember specifies: +- How to set up the SST simulation +- Allows multiple configurations to be specified and run +- Loads Motifs into the simulation to be performed. + +# Running Ember + +Ember uses a python interpreter built into the SST compiler. +`sst python-file.py` executes the python script. + +# Key Components + +The python file specifies the format of the SST simulation. The python file specifies components of our simulation such as network Topology, Routers, NetworkInterface and the computation to be performed in the form of Motifs. + +First, we import the necessary components of merlin and set the Platform to firefly defaults. + + import sst + from sst.merlin.base import * + from sst.merlin.endpoint import * + from sst.merlin.interface import * + from sst.merlin.topology import * + + from sst.ember import * + + if __name__ == "__main__": + + PlatformDefinition.setCurrentPlatform("firefly-defaults") + +Next, we create a topology that our experiment will run on. + +`topo = topoDragonFly()` +Creates a dragonfly topology. +We can then specify parameters about the topology using a dot-operator. e.g., +`topo.num_groups = 4` + +# Topologies + +The python file allows the user to specify comparable hardware configurations. +Four different topologies can be specified with different routing algorithms. +Here we give a comprehensive list of the topologies and how they can be initialized. + +1. HyperX `topoHyperX` + * `shape` + Specifies the shape of the mesh. i.e. topo.shape=4x4 or topo.shape=4x4x4. Any number of dimensions is supported. + * `width` + The number of links between routers in each dimension is specified in the same manner as shape. + * `local_ports` + Number of endpoints attached to each router + * `algorithm` + Routing algorithm to use. + * `DOAL` + * `DOR` + * `DOR-ND` + * `MIN-A` + * `VDAL` + * `valiant` +2. Torus `topoTorus` + * Shape + Specifies the mesh grid size, i.e., topo.shape=4x4 or topo.shape=4x4x4. Any number of dimensions is supported. + * Width + Number of links between routers in each dimension, specified in the same way as shape. + * local\_ports + Number of endpoints attached to each router +3. Fat Tree `topoFatTree` + * `shape` + The shape of the fat tree. Specified as pair of downlinks and uplinks per level separated by a colon. i.e., topo.shape="4,2:3,5" specifies the first level has 4 down and 2 up, and the following level has 3 down and 5 up. + If only one number is given, it is interpreted as the number of downlinks and up is set to 0. + * `routing_alg` Routing algorithm to use + * `deterministic` (default) + * `adaptive` + * `adaptive threshold.` + The threshold is used to determine if a packet will adaptively route. +4. Dragonfly `topoDragonFly` + + * `hosts_per_router` + Number of hosts connected to each router + * `routers_per_group` + The number of links used to connect to routers in the same group. + * `intergroup_per_router` + The number of links per router connected to other groups. + * `intergroup_links` + The number of links between each pair of groups. + * `num_groups` + Number of groups in a network + * `intergroup_links` + * `algorithm` + Specifies how each virtual network routes messages. + Specified for each router.vn. i.e., if router.vn=2, than topo.algorithm=\["minimal", "adaptive-local"], the first router uses minimal, and the second will use adaptive-local. + * `minimal` (default)\ + * `adaptive_local` + * `ugal` + * `min-a` + * `valiant` + Valiant only operates if there are more than two num\_groups. Otherwise, there is no point in using valiant routing. + * `adaptive_threshold` + Threshold when making adaptive routing decisions. + * `global_link_map` + An array specifying connectivity of global links in each dragonfly group + * `global_route_mode` + Mode for interpreting global link map + * `absolute` (default) + * `relative` + +# Creating a Router + +Once the router has been created, the topology needs to be linked to the router. Additionally, the link\_latency can be set in the topology at this point. + + topo.router = router + topo.link_latency = "20ns" + +Parameters for a high radix router or `hr_routers`: +* `id` +The ID of the router +* `num_ports` +Number of ports the router has +* `topology` +Name of the topology subcomponent loaded to control routing +* `xbar_arb` +Arbitration unit used for the crossbar. Default is `merlin.xbar_arb_lru`. +* `merlin.xbar_arb_lru` +Uses Least recently used arbitration for `hr_router.` +* `merlin.xbar_arb_age` +Age-based arbitration unit for `hr_router` +* `merlin.xbar_arb_lru` +Least recently used arbitration unit with infinite crossbar for `hr_router` +* `merlin.xbar_arb_rand` +Random arbitration unit for `hr_router` +* `merlin.xbar_arb_rr` +Round-robin arbitration unit for `hr_router` +* `link_bw` +Bandwidth of the links specified in either b/s or B/s (can include SI prefix) +* `flit_size` +Flit size specified in either b or B +* `xbar_bw` +Bandwidth of the crossbar specified in either b/s or B/s +* `input_latency` +Latency of packets entering switch into input buffers. Specified in s. +* `output_latency` +Latency of packets exiting switch from output buffers. Specified in s. +* `network_inspectors` +Comma-separated list of network inspectors to put on output ports +* `oql_track_port` +Set to true track output queue length for an entire port. False tracks per VC (default). + +# Creating Network Interface + +Create the network interface: + + networkif = ReorderLinkControl() + +The ReorderLinkControl() creates a network interface that can handle out-of-order packet arrival. Events are sequenced, and order is reconstructed on receive. + +# Specifying Computation + +Initialize the MPIJob to use all the nodes in our topology: + +`ep = EmberMPIJob(0, topo.getNumNodes())` +The network interface then needs to be linked to the `ep` variable +`ep.network_interface = networkif()` +Then a series of Motifs can be queued for computation. +``` + ep.addMotif("Init") + ep.addMotif("Allreduce") + ep.addMotif("Fini") +``` +The `addMotif` function adds the specified Motif to a queue. The Motif is named through an SST\_ELI\_REGISTER\_SUBCOMPONENT\_DERIVED command in the C++ Motif definition (usually in the include file). The SST\_ELI\_REGISTER\_SUBCOMPONENT\_DERIVED parameter follows the naming convention "ExampleMotif", and to add the Motif to `ep` using `ep.addMotif("Example")` The "Motif" portion is implied in the naming. +Parameters can be passed to motifs through the string. The parameters are read as a list of assignments, separated by whitespace. For example, a motif 'Sum' that takes three integers as a parameter named x, y, and z +The Motif would be invoked `ep.addMotif("Sum x=4 y=5 z=6")` would pass the arguments as args.x, args.y, and args.z with assigned values 4, 5, and 6, respectively. The arguments are passed in a Param object to the motif generator to be parsed. + +Some additional functions that can be called on an endpoint or the `ep` variable: +* getName() +Returns the name of the ep. i.e., it will return "EmberMPIJob" +* enableMotifLog() +Starts logging the Motifs as they are executed. + +Some common MPI-function call motifs that exist in Ember: + +# Running the simulation + +Finally, we create the 'system', which then runs the simulation. The topology is set with the setTopology function, and the endpoint is specified by system.allocateNodes. Build executes the motifs in the endpoint. +``` + system = System() + system.setTopology(topo) + system.allocateNodes(ep,"linear") + system.build() +``` +While most of the System is self-explanatory, the system allows the user to specify how threads/processes are assigned to nodes in the allocateNodes function. +The allocateNodes function allows the user to specify how the nodes are allocated. In the example, "linear" is chosen. In linear, the nodes are sorted, and threads are placed in linear order onto the simulated nodes. There are additional ways to allocate nodes, such as "random", "interval", and "indexed". diff --git a/src/sst/elements/ember/example/example.py b/src/sst/elements/ember/example/example.py new file mode 100644 index 0000000000..98a673522e --- /dev/null +++ b/src/sst/elements/ember/example/example.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python +# +# Copyright 2009-2022 NTESS. Under the terms +# of Contract DE-NA0003525 with NTESS, the U.S. +# Government retains certain rights in this software. +# +# Copyright (c) 2009-2022, NTESS +# All rights reserved. +# +# This file is part of the SST software package. For license +# information, see the LICENSE file in the top level directory of the +# distribution. + +import sst +from sst.merlin.base import * +from sst.merlin.endpoint import * +from sst.merlin.interface import * +from sst.merlin.topology import * +from sst.ember import * + + +def example(): + PlatformDefinition.setCurrentPlatform("firefly-defaults") + + # Setup the topology + topo = topoDragonFly() + topo.hosts_per_router = 2 + topo.routers_per_group = 4 + topo.intergroup_links = 2 + topo.num_groups = 4 + topo.algorithm = ["minimal", "adaptive-local"] + + # Set up the routers + router = hr_router() + router.link_bw = "4GB/s" + router.flit_size = "8B" + router.xbar_bw = "6GB/s" + router.input_latency = "20ns" + router.output_latency = "20ns" + router.input_buf_size = "4kB" + router.output_buf_size = "4kB" + router.num_vns = 2 + router.xbar_arb = "merlin.xbar_arb_lru" + + topo.router = router + topo.link_latency = "20ns" + + # set up the endpoint + networkif = ReorderLinkControl() + networkif.link_bw = "4GB/s" + networkif.input_buf_size = "1kB" + networkif.output_buf_size = "1kB" + + ep = EmberMPIJob(0, topo.getNumNodes()) + ep.network_interface = networkif + ep.addMotif("Example") + ep.nic.nic2host_lat = "100ns" + + system = System() + system.setTopology(topo) + system.allocateNodes(ep, "linear") + + system.build() + + +if __name__ == "__main__": + example() diff --git a/src/sst/elements/ember/example/motifs/emberexample.cc b/src/sst/elements/ember/example/motifs/emberexample.cc new file mode 100644 index 0000000000..82a5f40078 --- /dev/null +++ b/src/sst/elements/ember/example/motifs/emberexample.cc @@ -0,0 +1,30 @@ +// Copyright 2009-2022 NTESS. Under the terms +// of Contract DE-NA0003525 with NTESS, the U.S. +// Government retains certain rights in this software. +// +// Copyright (c) 2009-2022, NTESS +// All rights reserved. +// +// Portions are copyright of other developers: +// See the file CONTRIBUTORS.TXT in the top level directory +// of the distribution for more information. +// +// This file is part of the SST software package. For license +// information, see the LICENSE file in the top level directory of the +// distribution. + + +#include +#include "emberexample.h" + +using namespace SST::Ember; + +Example::Example(SST::ComponentId_t id, Params& params) : + EmberMessagePassingGenerator(id, params, "Null" ) +{ +} + +bool Example::generate( std::queue& evQ) +{ + return true; +} diff --git a/src/sst/elements/ember/example/motifs/emberexample.h b/src/sst/elements/ember/example/motifs/emberexample.h new file mode 100644 index 0000000000..089184b502 --- /dev/null +++ b/src/sst/elements/ember/example/motifs/emberexample.h @@ -0,0 +1,70 @@ +// Copyright 2009-2022 NTESS. Under the terms +// of Contract DE-NA0003525 with NTESS, the U.S. +// Government retains certain rights in this software. +// +// Copyright (c) 2009-2022, NTESS +// All rights reserved. +// +// Portions are copyright of other developers: +// See the file CONTRIBUTORS.TXT in the top level directory +// of the distribution for more information. +// +// This file is part of the SST software package. For license +// information, see the LICENSE file in the top level directory of the +// distribution. + +#ifndef _H_EMBER_FOO +#define _H_EMBER_FOO + +#include "../../mpi/embermpigen.h" + +namespace SST { +namespace Ember { + +class Foo : public EmberMessagePassingGenerator { + public: + + SST_ELI_REGISTER_SUBCOMPONENT_DERIVED( + Foo, + "ember", + "ExampleMotif", + SST_ELI_ELEMENT_VERSION(1,0,0), + "Performs an idle on the node, no traffic can be generated.", + SST::Ember::EmberGenerator + ); + + SST_ELI_DOCUMENT_PARAMS( + ); + + SST_ELI_DOCUMENT_STATISTICS( + { "time-Init", "Time spent in Init event", "ns", 0}, + { "time-Finalize", "Time spent in Finalize event", "ns", 0}, + { "time-Rank", "Time spent in Rank event", "ns", 0}, + { "time-Size", "Time spent in Size event", "ns", 0}, + { "time-Send", "Time spent in Recv event", "ns", 0}, + { "time-Recv", "Time spent in Recv event", "ns", 0}, + { "time-Irecv", "Time spent in Irecv event", "ns", 0}, + { "time-Isend", "Time spent in Isend event", "ns", 0}, + { "time-Wait", "Time spent in Wait event", "ns", 0}, + { "time-Waitall", "Time spent in Waitall event", "ns", 0}, + { "time-Waitany", "Time spent in Waitany event", "ns", 0}, + { "time-Compute", "Time spent in Compute event", "ns", 0}, + { "time-Barrier", "Time spent in Barrier event", "ns", 0}, + { "time-Alltoallv", "Time spent in Alltoallv event", "ns", 0}, + { "time-Alltoall", "Time spent in Alltoall event", "ns", 0}, + { "time-Allreduce", "Time spent in Allreduce event", "ns", 0}, + { "time-Reduce", "Time spent in Reduce event", "ns", 0}, + { "time-Bcast", "Time spent in Bcast event", "ns", 0}, + { "time-Gettime", "Time spent in Gettime event", "ns", 0}, + { "time-Commsplit", "Time spent in Commsplit event", "ns", 0}, + { "time-Commcreate", "Time spent in Commcreate event", "ns", 0}, + ); + + Example(SST::ComponentId_t id, Params& params); + bool generate( std::queue& evQ); +}; + +} +} + +#endif /* _H_EMBER_FOO */ From a604b957d742e9fd2147160cc3b855dfec99d130 Mon Sep 17 00:00:00 2001 From: Neil Butcher <77634936+nab880@users.noreply.github.com> Date: Fri, 14 Oct 2022 13:23:43 -0700 Subject: [PATCH 2/8] Update Makefile.am --- src/sst/elements/ember/Makefile.am | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/sst/elements/ember/Makefile.am b/src/sst/elements/ember/Makefile.am index 37f3d62c29..1122f9eee1 100644 --- a/src/sst/elements/ember/Makefile.am +++ b/src/sst/elements/ember/Makefile.am @@ -12,8 +12,8 @@ compdir = $(pkglibdir) comp_LTLIBRARIES = libember.la libember_la_SOURCES = \ - example/emberexample.h - example/emberexample.cc + example/motifs/emberexample.h + example/motifs/emberexample.cc embergen.h \ embergen.cc \ embermap.h \ From 6ae5d8e9626d782d029689178d6907f57bc2e306 Mon Sep 17 00:00:00 2001 From: nab880 Date: Fri, 14 Oct 2022 12:33:34 -0700 Subject: [PATCH 3/8] Adding Ember documentation Adding ember documentation and examples Update Makefile.am --- src/sst/elements/ember/Makefile.am | 2 + .../elements/ember/example/CreatingMotifs.md | 116 ++++++++++ src/sst/elements/ember/example/QuickStart.md | 217 ++++++++++++++++++ .../elements/ember/example/RunningMotifs.md | 190 +++++++++++++++ src/sst/elements/ember/example/example.py | 67 ++++++ .../ember/example/motifs/emberexample.cc | 30 +++ .../ember/example/motifs/emberexample.h | 70 ++++++ 7 files changed, 692 insertions(+) create mode 100644 src/sst/elements/ember/example/CreatingMotifs.md create mode 100644 src/sst/elements/ember/example/QuickStart.md create mode 100644 src/sst/elements/ember/example/RunningMotifs.md create mode 100644 src/sst/elements/ember/example/example.py create mode 100644 src/sst/elements/ember/example/motifs/emberexample.cc create mode 100644 src/sst/elements/ember/example/motifs/emberexample.h diff --git a/src/sst/elements/ember/Makefile.am b/src/sst/elements/ember/Makefile.am index ab907fd870..bf75971bd6 100644 --- a/src/sst/elements/ember/Makefile.am +++ b/src/sst/elements/ember/Makefile.am @@ -12,6 +12,8 @@ compdir = $(pkglibdir) comp_LTLIBRARIES = libember.la libember_la_SOURCES = \ + example/motifs/emberexample.h \ + example/motifs/emberexample.cc \ embergen.h \ embergen.cc \ embermap.h \ diff --git a/src/sst/elements/ember/example/CreatingMotifs.md b/src/sst/elements/ember/example/CreatingMotifs.md new file mode 100644 index 0000000000..803b7629c2 --- /dev/null +++ b/src/sst/elements/ember/example/CreatingMotifs.md @@ -0,0 +1,116 @@ +# C++ Ember Motifs +## Ember + +[Ember](http://sst-simulator.org/SSTPages/SSTElementEmber/) is a library representing various network communications. It is often used in conjunction with motifs. + +## Motif + +Motifs are condensed, efficient generators for simulating different kinds of communication/computation patterns. +The motif presented here does no real work, but more detailed motifs can be found in sst-elements/src/sst/elements/ember/mpi/motifs/. + +Motifs are executed as follows: + +1) The motif generator is initialized (The constructor) +2) The generate function is invoked, places events on the event queue, and returns either true or false +3) The events on the event queue are processed. +4) If the generate function in step 2 returns false, go to step 2. Otherwise, the motif is complete. + +Motifs are intended to be run as a 'job submission.' +The generate function models an entire iteration of an application, using the event queue to mix compute and MPI operations in every iteration. + +### Register Subcomponent + +Ember motifs need to be registered as an SST subcomponent. This typically happens in the header file. +For example: +``` +SST_ELI_REGISTER_SUBCOMPONENT_DERIVED( + Example, + "ember", + "ExampleMotif", + SST_ELI_ELEMENT_VERSION(1,0,0), + "Performs an idle on the node. No traffic can be generated.", + SST::Ember::EmberGenerator + ); +``` + +The parameters of this function are: + +1) class associated with this Motif +3) Aspect of elements being used, in the case of Motifs, "ember." +3) Identifier of the Motif. Note that the end of this parameter must always be Motif +4) SST elements version, +5) Comment describing what the motif does +6) EmberGenerator + +### Writing a constructor +``` +EmberExample::EmberExample(SST::ComponentId_t id, Params& params) : + EmberMessagePassingGenerator(id, params, "Example"), + +``` + + +The constructor also reads parameters from the python script. +The params are passed through the python file in the form: `ep.addMotif("Example firstParam=100 secondParam=200")`. Note no space is allowed before or after the = operator. Parameters read from the python file will be prepended with "arg." before being passed to the C++ file. i.e. "firstParam" becomes "arg.firstParam". +The constructor can be used to perform the setup operations necessary for the generating function. + +The params can be parsed in the C++ file with `firstParam = params.find("arg.firstParam", 100);` where the name of the parameter is "firstParam". + +### Writing a generate() function +``` +bool Example::generate( std::queue& evQ) +``` +The generate function is the 'main' function of a Motif. +If the python file invokes an `addMotif`, the generating function will be invoked until the generate function returns true. +Once the generate function returns, the events queued in the evQ variable will be performed. + +The generate function takes an event queue as a parameter. The event queue allows the user to include computation operations and MPI events in the simulation. Motifs are intended to be designed so that every call to generate() mimics an entire iteration of the application. The events that can be added to the event queue are listed in subsequent sections. + +. + +# User-defined events + +These functions allow the user to control how the simulation estimates computation time. + +* `enQ_compute(Queue&, uint64_t nanoSecondDelay)` -- The delay by the expected cost of the compute operation in nanoSeconds (without actually computing it) +* `enQ_compute( Queue& q, std::function func)`; -- A function is passed as a parameter and invoked. It returns a 64-bit unsigned integer that indicates the simulation delay associated with invoking the function (in nanoseconds). +* `enQ_getTime(evQ, &m_startTime)` -- sets m_startTime to the current time + +There are two types of compute functions that can be enqueued. + +The first computation operation takes a 64-bit integer as a parameter. This parameter is the amount of time needed in nanoseconds to perform the computation. The simulator then delays by that number of nanoseconds as if the actual computation had taken place. +The second compute takes a function as a parameter. When the event is processed, the function is invoked. The function returns the amount of time needed to perform the compute operation. The simulation is delayed by the return value of the function. The time delay could be estimated through a heuristic or measured through representative computation. + + +# MPI Events + + +We list the supported MPI events below. For more detail, see MPI API documentation. + + +* `enQ_init(evQ)` -- MPI initialize +* `enQ_fini(evQ)` -- MPI finalize +* `enQ_rank(evQ, m_newComm[0], &m_new_rank)` -- MPI rank +* `enQ_size(evQ, m_newComm[0], &m_new_size)` -- MPI size +* `enQ_send(evQ, x_neg, x_xferSize, 0, GroupWorld)` -- MPI send +* `enQ_recv(evQ, x_neg, x_xferSize, 0, GroupWorld)` -- MPI recv +* `enQ_isend(evQ, next_comm_rank, itemsThisPhase, 0, GroupWorld, &requests[next_request])` -- MPI isend +* `enQ_irecv(evQ, xface_down, items_per_cell * sizeof_cell * ny * nz, 0, GroupWorld, &requests[nextRequest])` -- MPI irecv +* `enQ_cancel(evQ, m_req[i])` -- MPI cancel +* `enQ_sendrecv(evQ, m_sendBuf, m_messageSize, DATA_TYPE, destRank(), 0xdeadbeef, m_recvBuf, m_messageSize, DATA_TYPE, srcRank(), 0xdeadbeef, GroupWorld, &m_resp)` -- MPI send or recv +* `enQ_wait(evQ, req)` -- MPI wait +* `enQ_waitany(evQ, m_req.size(), &m_req[0], &m_indx, &m_resp)` -- MPI waitany +* `enQ_waitall(evQ, 1, &m_req, (MessageResponse**)&m_resp)` -- MPI waitall +* `enQ_barrier(evQ, GroupWorld)` -- MPI barrier +* `enQ_bcast(evQ, m_sendBuf, m_count, DOUBLE, m_root, GroupWorld)` -- MPI broadcast +* `enQ_scatter(evQ, m_sendBuf, m_sendCnt, m_sendDsp.data(), LONG, m_recvBuf, m_count, LONG, m_root, GroupWorld)` +* `enQ_scatterv(evQ, m_sendBuf, &m_sendCnts[0], m_sendDsp.data(), LONG, m_recvBuf, m_count, LONG, m_root, GroupWorld)` -- MPI Scatter variable message size +* `enQ_reduce( evQ, m_sendBuf, m_recvBuf, m_count, DOUBLE, MP::SUM, m_redRoot, GroupWorld)` +* `enQ_allreduce(evQ, m_sendBuf, m_recvBuf, m_count, DOUBLE, m_op, GroupWorld)` -- MPI allreduce function +* `enQ_alltoall(evQ, m_sendBuf, m_colSendCnt, &m_colSendDsp_f[0], DOUBLE, m_colComm)` -- MPI alltoall with constant message size +* `enQ_alltoallv(evQ, m_sendBuf, &m_colSendCnts_f[0], &m_colSendDsp_f[0], DOUBLE, m_colComm)` -- MPI alltoall with varying message sizes +* `enQ_allgather(evQ, m_sendBuf, m_count, INT, m_recvBuf, m_count, INT, GroupWorld)` -- MPI allgather with messages of constant size +* `enQ_allgatherv(evQ, m_sendBuf, m_sendCnt, INT, m_recvBuf, &m_recvCnts[0], &m_recvDsp[0], INT, GroupWorld)` -- MPI allgather with messages of varying sizes +* `enQ_commSplit(evQ, *comm, color, key, newComm)` -- Splits the MPI comm +* `enQ_commCreate(evQ, GroupWorld, m_rowGrpRanks, &m_rowComm)` -- Creates MPI Com +* `enQ_commDestroy(evQ, m_rowComm)` --Destroys MPI comm diff --git a/src/sst/elements/ember/example/QuickStart.md b/src/sst/elements/ember/example/QuickStart.md new file mode 100644 index 0000000000..6a3bcba64c --- /dev/null +++ b/src/sst/elements/ember/example/QuickStart.md @@ -0,0 +1,217 @@ +# What is an Ember Motif? + +Ember Motifs provide representative workloads and then estimate performance on hardware using SST simulations. +This guide provides a directions on how to: + +* Implement a empty motif. +* Create and run motif in a SST simulation. + +Implementing and running Ember Motifs requires three different files: + +1. Python File + Provides information on the hardware to be simulated when running an Ember Motif + This directs SST on the available hardware components and how they are set up. +2. C++ file + Decides how the computation is structured and computed. +3. C++ header file + The header file contains primarily naming information + +# Quick Start Guide + +This guide gives a simple example of how to create and run a simple Motif. + +First ensure SST-core and SST-elements are properly installed on your machine. For download and installation see [SST-Downloads](http://sst-simulator.org/SSTPages/SSTMainDownloads/). + +Navigate to `sst-elements/src/elements/ember` inside the sst-elements library. +First, create a new directory to hold our example Motif. + +`mkdir ExampleMotif` + +Next, create a `ExampleMotif/exampleMotif.h` file +``` + #ifndef _H_EMBER_EXAMPLE + #define _H_EMBER_EXAMPLE + + #include "../../mpi/embermpigen.h" + + namespace SST { + namespace Ember { + + class ExampleGenerator : public EmberMessagePassingGenerator { + public: + + SST_ELI_REGISTER_SUBCOMPONENT_DERIVED( + Example, + "ember", + "ExampleMotif", + SST_ELI_ELEMENT_VERSION(1,0,0), + "Performs an idle on the node, no traffic can be generated.", + SST::Ember::EmberGenerator + ); + + SST_ELI_DOCUMENT_PARAMS( + ); + + SST_ELI_DOCUMENT_STATISTICS( + { "time-Init", "Time spent in Init event", "ns", 0}, + { "time-Finalize", "Time spent in Finalize event", "ns", 0}, + { "time-Rank", "Time spent in Rank event", "ns", 0}, + { "time-Size", "Time spent in Size event", "ns", 0}, + { "time-Send", "Time spent in Recv event", "ns", 0}, + { "time-Recv", "Time spent in Recv event", "ns", 0}, + { "time-Irecv", "Time spent in Irecv event", "ns", 0}, + { "time-Isend", "Time spent in Isend event", "ns", 0}, + { "time-Wait", "Time spent in Wait event", "ns", 0}, + { "time-Waitall", "Time spent in Waitall event", "ns", 0}, + { "time-Waitany", "Time spent in Waitany event", "ns", 0}, + { "time-Compute", "Time spent in Compute event", "ns", 0}, + { "time-Barrier", "Time spent in Barrier event", "ns", 0}, + { "time-Alltoallv", "Time spent in Alltoallv event", "ns", 0}, + { "time-Alltoall", "Time spent in Alltoall event", "ns", 0}, + { "time-Allreduce", "Time spent in Allreduce event", "ns", 0}, + { "time-Reduce", "Time spent in Reduce event", "ns", 0}, + { "time-Bcast", "Time spent in Bcast event", "ns", 0}, + { "time-Gettime", "Time spent in Gettime event", "ns", 0}, + { "time-Commsplit", "Time spent in Commsplit event", "ns", 0}, + { "time-Commcreate", "Time spent in Commcreate event", "ns", 0}, + ); + + + }; + } + } + + #endif /* _H_EMBER_EXAMPLE */ +``` +This creates a Motif generator class called ExampleGenerator that inherits from EmberMessagePassingGenerator. The Python code we provide later creates and invokes the generate function. +The SST Document Statistics provides tracking for initialization and various MPI function calls. + +Motifs are executed as follows: + + + +1) The motif generator is initialized (The contructor) +2) The generate function is invoked and returns either true or false +3) The events on the eventQueue are processed. +4) If the generate function in step 2 returned false, return to step 2, otherwise the motif is complete. + +Here is a motif that returns without doing any additional work. + +Then we create a file `ExampleMotif/Example.cc` with the contents: + +``` + #include + #include "Example.h" + + using namespace SST:Ember; + + ExampleGenerator::ExampleGenerator(SST::ComponentId_t id, Params& params) : + EmberMessagePassingGenerator(id, params, "Null" ) + { + } + + bool ExampleGenerator::generate( std::queue& evQ) + { + // Code here is repeated until true is returned. + return true; + } +``` + + +Motifs are intended to be ran as a 'job submission'. +The generate function models an entire iteration of an application. The event queue allows the motif to mix compute and MPI operations in every iteration. + + + +The C++ file loads the SST:Ember namespace giving it access to other Motifs and EmberEvents. +Each Motif generator has an constructor and generate function. + + +Then we add the `.cc` and `.h` file to `Makefile.am` + +i.e. +``` + embermemoryev.h \ + ExampleMotif/example.h \ + ExampleMotif/example.cc \ + libs/emberLib.h \ +``` + +Finally, a python file `example.py` needs to be created: + +``` + from email.mime import base + import sst + from sst.merlin.base import * + from sst.merlin.endpoint import * + from sst.merlin.interface import * + from sst.merlin.topology import * + + from sst.ember import * + + def example(): + PlatformDefinition.setCurrentPlatform("firefly-defaults") + + ### Setup the topology + topo = topoDragonFly() + topo.hosts_per_router = 2 + topo.routers_per_group = 4 + topo.intergroup_links = 2 + topo.num_groups = 4 + topo.algorithm = ["minimal","adaptive-local"] + + # Set up the routers + router = hr_router() + router.link_bw = "4GB/s" + router.flit_size = "8B" + router.xbar_bw = "6GB/s" + router.input_latency = "20ns" + router.output_latency = "20ns" + router.input_buf_size = "4kB" + router.output_buf_size = "4kB" + router.num_vns = 2 + router.xbar_arb = "merlin.xbar_arb_lru" + + topo.router = router + topo.link_latency = "20ns" + + ### set up the endpoint + networkif = ReorderLinkControl() + networkif.link_bw = "4GB/s" + networkif.input_buf_size = "1kB" + networkif.output_buf_size = "1kB" + + ep = EmberMPIJob(0,topo.getNumNodes()) + ep.network_interface = networkif + ep.addMotif("Example") + ep.nic.nic2host_lat= "100ns" + + system = System() + system.setTopology(topo) + system.allocateNodes(ep,"linear") + + system.build() + + if __name__ == "__main__": + example() +``` + +This follows common python syntax. +The hardware variables (topology, router and network interface) are created through assignment of a constructor +The fields of the variables are accessed using a dot operator. +Topology, router, and network interface variables need to be created for a simulation. More detailed descriptions of configuration options is in RunningMotifs.md. +Next an endpoint is created and given the motif to be used using a ep.addMotif("MotifName")call +Note multiple motifs can be added the same endpoint. The intended use for each motif to simulate an entire application. The event queue is used to simulate MPI events and computational workloads. Adding multiple motifs is useful for simulating workflows of a series of applications. + +Finally, a system variable is created and 'built' + +Remake SST-elements. + +To run the python script + +``` +sst example.py +``` + + + diff --git a/src/sst/elements/ember/example/RunningMotifs.md b/src/sst/elements/ember/example/RunningMotifs.md new file mode 100644 index 0000000000..3244247705 --- /dev/null +++ b/src/sst/elements/ember/example/RunningMotifs.md @@ -0,0 +1,190 @@ +# Ember Python Configurations + +The Python file in Ember specifies: +- How to set up the SST simulation +- Allows multiple configurations to be specified and run +- Loads Motifs into the simulation to be performed. + +# Running Ember + +Ember uses a python interpreter built into the SST compiler. +`sst python-file.py` executes the python script. + +# Key Components + +The python file specifies the format of the SST simulation. The python file specifies components of our simulation such as network Topology, Routers, NetworkInterface and the computation to be performed in the form of Motifs. + +First, we import the necessary components of merlin and set the Platform to firefly defaults. + + import sst + from sst.merlin.base import * + from sst.merlin.endpoint import * + from sst.merlin.interface import * + from sst.merlin.topology import * + + from sst.ember import * + + if __name__ == "__main__": + + PlatformDefinition.setCurrentPlatform("firefly-defaults") + +Next, we create a topology that our experiment will run on. + +`topo = topoDragonFly()` +Creates a dragonfly topology. +We can then specify parameters about the topology using a dot-operator. e.g., +`topo.num_groups = 4` + +# Topologies + +The python file allows the user to specify comparable hardware configurations. +Four different topologies can be specified with different routing algorithms. +Here we give a comprehensive list of the topologies and how they can be initialized. + +1. HyperX `topoHyperX` + * `shape` + Specifies the shape of the mesh. i.e. topo.shape=4x4 or topo.shape=4x4x4. Any number of dimensions is supported. + * `width` + The number of links between routers in each dimension is specified in the same manner as shape. + * `local_ports` + Number of endpoints attached to each router + * `algorithm` + Routing algorithm to use. + * `DOAL` + * `DOR` + * `DOR-ND` + * `MIN-A` + * `VDAL` + * `valiant` +2. Torus `topoTorus` + * Shape + Specifies the mesh grid size, i.e., topo.shape=4x4 or topo.shape=4x4x4. Any number of dimensions is supported. + * Width + Number of links between routers in each dimension, specified in the same way as shape. + * local\_ports + Number of endpoints attached to each router +3. Fat Tree `topoFatTree` + * `shape` + The shape of the fat tree. Specified as pair of downlinks and uplinks per level separated by a colon. i.e., topo.shape="4,2:3,5" specifies the first level has 4 down and 2 up, and the following level has 3 down and 5 up. + If only one number is given, it is interpreted as the number of downlinks and up is set to 0. + * `routing_alg` Routing algorithm to use + * `deterministic` (default) + * `adaptive` + * `adaptive threshold.` + The threshold is used to determine if a packet will adaptively route. +4. Dragonfly `topoDragonFly` + + * `hosts_per_router` + Number of hosts connected to each router + * `routers_per_group` + The number of links used to connect to routers in the same group. + * `intergroup_per_router` + The number of links per router connected to other groups. + * `intergroup_links` + The number of links between each pair of groups. + * `num_groups` + Number of groups in a network + * `intergroup_links` + * `algorithm` + Specifies how each virtual network routes messages. + Specified for each router.vn. i.e., if router.vn=2, than topo.algorithm=\["minimal", "adaptive-local"], the first router uses minimal, and the second will use adaptive-local. + * `minimal` (default)\ + * `adaptive_local` + * `ugal` + * `min-a` + * `valiant` + Valiant only operates if there are more than two num\_groups. Otherwise, there is no point in using valiant routing. + * `adaptive_threshold` + Threshold when making adaptive routing decisions. + * `global_link_map` + An array specifying connectivity of global links in each dragonfly group + * `global_route_mode` + Mode for interpreting global link map + * `absolute` (default) + * `relative` + +# Creating a Router + +Once the router has been created, the topology needs to be linked to the router. Additionally, the link\_latency can be set in the topology at this point. + + topo.router = router + topo.link_latency = "20ns" + +Parameters for a high radix router or `hr_routers`: +* `id` +The ID of the router +* `num_ports` +Number of ports the router has +* `topology` +Name of the topology subcomponent loaded to control routing +* `xbar_arb` +Arbitration unit used for the crossbar. Default is `merlin.xbar_arb_lru`. +* `merlin.xbar_arb_lru` +Uses Least recently used arbitration for `hr_router.` +* `merlin.xbar_arb_age` +Age-based arbitration unit for `hr_router` +* `merlin.xbar_arb_lru` +Least recently used arbitration unit with infinite crossbar for `hr_router` +* `merlin.xbar_arb_rand` +Random arbitration unit for `hr_router` +* `merlin.xbar_arb_rr` +Round-robin arbitration unit for `hr_router` +* `link_bw` +Bandwidth of the links specified in either b/s or B/s (can include SI prefix) +* `flit_size` +Flit size specified in either b or B +* `xbar_bw` +Bandwidth of the crossbar specified in either b/s or B/s +* `input_latency` +Latency of packets entering switch into input buffers. Specified in s. +* `output_latency` +Latency of packets exiting switch from output buffers. Specified in s. +* `network_inspectors` +Comma-separated list of network inspectors to put on output ports +* `oql_track_port` +Set to true track output queue length for an entire port. False tracks per VC (default). + +# Creating Network Interface + +Create the network interface: + + networkif = ReorderLinkControl() + +The ReorderLinkControl() creates a network interface that can handle out-of-order packet arrival. Events are sequenced, and order is reconstructed on receive. + +# Specifying Computation + +Initialize the MPIJob to use all the nodes in our topology: + +`ep = EmberMPIJob(0, topo.getNumNodes())` +The network interface then needs to be linked to the `ep` variable +`ep.network_interface = networkif()` +Then a series of Motifs can be queued for computation. +``` + ep.addMotif("Init") + ep.addMotif("Allreduce") + ep.addMotif("Fini") +``` +The `addMotif` function adds the specified Motif to a queue. The Motif is named through an SST\_ELI\_REGISTER\_SUBCOMPONENT\_DERIVED command in the C++ Motif definition (usually in the include file). The SST\_ELI\_REGISTER\_SUBCOMPONENT\_DERIVED parameter follows the naming convention "ExampleMotif", and to add the Motif to `ep` using `ep.addMotif("Example")` The "Motif" portion is implied in the naming. +Parameters can be passed to motifs through the string. The parameters are read as a list of assignments, separated by whitespace. For example, a motif 'Sum' that takes three integers as a parameter named x, y, and z +The Motif would be invoked `ep.addMotif("Sum x=4 y=5 z=6")` would pass the arguments as args.x, args.y, and args.z with assigned values 4, 5, and 6, respectively. The arguments are passed in a Param object to the motif generator to be parsed. + +Some additional functions that can be called on an endpoint or the `ep` variable: +* getName() +Returns the name of the ep. i.e., it will return "EmberMPIJob" +* enableMotifLog() +Starts logging the Motifs as they are executed. + +Some common MPI-function call motifs that exist in Ember: + +# Running the simulation + +Finally, we create the 'system', which then runs the simulation. The topology is set with the setTopology function, and the endpoint is specified by system.allocateNodes. Build executes the motifs in the endpoint. +``` + system = System() + system.setTopology(topo) + system.allocateNodes(ep,"linear") + system.build() +``` +While most of the System is self-explanatory, the system allows the user to specify how threads/processes are assigned to nodes in the allocateNodes function. +The allocateNodes function allows the user to specify how the nodes are allocated. In the example, "linear" is chosen. In linear, the nodes are sorted, and threads are placed in linear order onto the simulated nodes. There are additional ways to allocate nodes, such as "random", "interval", and "indexed". diff --git a/src/sst/elements/ember/example/example.py b/src/sst/elements/ember/example/example.py new file mode 100644 index 0000000000..98a673522e --- /dev/null +++ b/src/sst/elements/ember/example/example.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python +# +# Copyright 2009-2022 NTESS. Under the terms +# of Contract DE-NA0003525 with NTESS, the U.S. +# Government retains certain rights in this software. +# +# Copyright (c) 2009-2022, NTESS +# All rights reserved. +# +# This file is part of the SST software package. For license +# information, see the LICENSE file in the top level directory of the +# distribution. + +import sst +from sst.merlin.base import * +from sst.merlin.endpoint import * +from sst.merlin.interface import * +from sst.merlin.topology import * +from sst.ember import * + + +def example(): + PlatformDefinition.setCurrentPlatform("firefly-defaults") + + # Setup the topology + topo = topoDragonFly() + topo.hosts_per_router = 2 + topo.routers_per_group = 4 + topo.intergroup_links = 2 + topo.num_groups = 4 + topo.algorithm = ["minimal", "adaptive-local"] + + # Set up the routers + router = hr_router() + router.link_bw = "4GB/s" + router.flit_size = "8B" + router.xbar_bw = "6GB/s" + router.input_latency = "20ns" + router.output_latency = "20ns" + router.input_buf_size = "4kB" + router.output_buf_size = "4kB" + router.num_vns = 2 + router.xbar_arb = "merlin.xbar_arb_lru" + + topo.router = router + topo.link_latency = "20ns" + + # set up the endpoint + networkif = ReorderLinkControl() + networkif.link_bw = "4GB/s" + networkif.input_buf_size = "1kB" + networkif.output_buf_size = "1kB" + + ep = EmberMPIJob(0, topo.getNumNodes()) + ep.network_interface = networkif + ep.addMotif("Example") + ep.nic.nic2host_lat = "100ns" + + system = System() + system.setTopology(topo) + system.allocateNodes(ep, "linear") + + system.build() + + +if __name__ == "__main__": + example() diff --git a/src/sst/elements/ember/example/motifs/emberexample.cc b/src/sst/elements/ember/example/motifs/emberexample.cc new file mode 100644 index 0000000000..82a5f40078 --- /dev/null +++ b/src/sst/elements/ember/example/motifs/emberexample.cc @@ -0,0 +1,30 @@ +// Copyright 2009-2022 NTESS. Under the terms +// of Contract DE-NA0003525 with NTESS, the U.S. +// Government retains certain rights in this software. +// +// Copyright (c) 2009-2022, NTESS +// All rights reserved. +// +// Portions are copyright of other developers: +// See the file CONTRIBUTORS.TXT in the top level directory +// of the distribution for more information. +// +// This file is part of the SST software package. For license +// information, see the LICENSE file in the top level directory of the +// distribution. + + +#include +#include "emberexample.h" + +using namespace SST::Ember; + +Example::Example(SST::ComponentId_t id, Params& params) : + EmberMessagePassingGenerator(id, params, "Null" ) +{ +} + +bool Example::generate( std::queue& evQ) +{ + return true; +} diff --git a/src/sst/elements/ember/example/motifs/emberexample.h b/src/sst/elements/ember/example/motifs/emberexample.h new file mode 100644 index 0000000000..089184b502 --- /dev/null +++ b/src/sst/elements/ember/example/motifs/emberexample.h @@ -0,0 +1,70 @@ +// Copyright 2009-2022 NTESS. Under the terms +// of Contract DE-NA0003525 with NTESS, the U.S. +// Government retains certain rights in this software. +// +// Copyright (c) 2009-2022, NTESS +// All rights reserved. +// +// Portions are copyright of other developers: +// See the file CONTRIBUTORS.TXT in the top level directory +// of the distribution for more information. +// +// This file is part of the SST software package. For license +// information, see the LICENSE file in the top level directory of the +// distribution. + +#ifndef _H_EMBER_FOO +#define _H_EMBER_FOO + +#include "../../mpi/embermpigen.h" + +namespace SST { +namespace Ember { + +class Foo : public EmberMessagePassingGenerator { + public: + + SST_ELI_REGISTER_SUBCOMPONENT_DERIVED( + Foo, + "ember", + "ExampleMotif", + SST_ELI_ELEMENT_VERSION(1,0,0), + "Performs an idle on the node, no traffic can be generated.", + SST::Ember::EmberGenerator + ); + + SST_ELI_DOCUMENT_PARAMS( + ); + + SST_ELI_DOCUMENT_STATISTICS( + { "time-Init", "Time spent in Init event", "ns", 0}, + { "time-Finalize", "Time spent in Finalize event", "ns", 0}, + { "time-Rank", "Time spent in Rank event", "ns", 0}, + { "time-Size", "Time spent in Size event", "ns", 0}, + { "time-Send", "Time spent in Recv event", "ns", 0}, + { "time-Recv", "Time spent in Recv event", "ns", 0}, + { "time-Irecv", "Time spent in Irecv event", "ns", 0}, + { "time-Isend", "Time spent in Isend event", "ns", 0}, + { "time-Wait", "Time spent in Wait event", "ns", 0}, + { "time-Waitall", "Time spent in Waitall event", "ns", 0}, + { "time-Waitany", "Time spent in Waitany event", "ns", 0}, + { "time-Compute", "Time spent in Compute event", "ns", 0}, + { "time-Barrier", "Time spent in Barrier event", "ns", 0}, + { "time-Alltoallv", "Time spent in Alltoallv event", "ns", 0}, + { "time-Alltoall", "Time spent in Alltoall event", "ns", 0}, + { "time-Allreduce", "Time spent in Allreduce event", "ns", 0}, + { "time-Reduce", "Time spent in Reduce event", "ns", 0}, + { "time-Bcast", "Time spent in Bcast event", "ns", 0}, + { "time-Gettime", "Time spent in Gettime event", "ns", 0}, + { "time-Commsplit", "Time spent in Commsplit event", "ns", 0}, + { "time-Commcreate", "Time spent in Commcreate event", "ns", 0}, + ); + + Example(SST::ComponentId_t id, Params& params); + bool generate( std::queue& evQ); +}; + +} +} + +#endif /* _H_EMBER_FOO */ From f5a4ac14692fcad873abe6a944766bceee3d4565 Mon Sep 17 00:00:00 2001 From: Neil Butcher <77634936+nab880@users.noreply.github.com> Date: Fri, 14 Oct 2022 15:07:50 -0700 Subject: [PATCH 4/8] Update Makefile.am --- src/sst/elements/ember/Makefile.am | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/sst/elements/ember/Makefile.am b/src/sst/elements/ember/Makefile.am index 1122f9eee1..bf75971bd6 100644 --- a/src/sst/elements/ember/Makefile.am +++ b/src/sst/elements/ember/Makefile.am @@ -12,8 +12,8 @@ compdir = $(pkglibdir) comp_LTLIBRARIES = libember.la libember_la_SOURCES = \ - example/motifs/emberexample.h - example/motifs/emberexample.cc + example/motifs/emberexample.h \ + example/motifs/emberexample.cc \ embergen.h \ embergen.cc \ embermap.h \ From 366ce1963a2d86e2a5e445c8fdf7bb1f38e6030a Mon Sep 17 00:00:00 2001 From: nab880 Date: Mon, 17 Oct 2022 09:12:55 -0700 Subject: [PATCH 5/8] Update emberexample.h --- src/sst/elements/ember/example/motifs/emberexample.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/sst/elements/ember/example/motifs/emberexample.h b/src/sst/elements/ember/example/motifs/emberexample.h index 089184b502..7312a342ba 100644 --- a/src/sst/elements/ember/example/motifs/emberexample.h +++ b/src/sst/elements/ember/example/motifs/emberexample.h @@ -13,19 +13,19 @@ // information, see the LICENSE file in the top level directory of the // distribution. -#ifndef _H_EMBER_FOO -#define _H_EMBER_FOO +#ifndef _H_EMBER_EXAMPLE +#define _H_EMBER_EXAMPLE #include "../../mpi/embermpigen.h" namespace SST { namespace Ember { -class Foo : public EmberMessagePassingGenerator { +class Example : public EmberMessagePassingGenerator { public: SST_ELI_REGISTER_SUBCOMPONENT_DERIVED( - Foo, + Example, "ember", "ExampleMotif", SST_ELI_ELEMENT_VERSION(1,0,0), @@ -67,4 +67,4 @@ class Foo : public EmberMessagePassingGenerator { } } -#endif /* _H_EMBER_FOO */ +#endif /* _H_EMBER_EXAMPLE */ From 30fd01995f38eed305d9a3996caf77e8be7ef282 Mon Sep 17 00:00:00 2001 From: Neil Butcher <77634936+nab880@users.noreply.github.com> Date: Mon, 1 May 2023 11:51:47 -0700 Subject: [PATCH 6/8] Delete CreatingMotifs.md remove md files --- .../elements/ember/example/CreatingMotifs.md | 116 ------------------ 1 file changed, 116 deletions(-) delete mode 100644 src/sst/elements/ember/example/CreatingMotifs.md diff --git a/src/sst/elements/ember/example/CreatingMotifs.md b/src/sst/elements/ember/example/CreatingMotifs.md deleted file mode 100644 index 803b7629c2..0000000000 --- a/src/sst/elements/ember/example/CreatingMotifs.md +++ /dev/null @@ -1,116 +0,0 @@ -# C++ Ember Motifs -## Ember - -[Ember](http://sst-simulator.org/SSTPages/SSTElementEmber/) is a library representing various network communications. It is often used in conjunction with motifs. - -## Motif - -Motifs are condensed, efficient generators for simulating different kinds of communication/computation patterns. -The motif presented here does no real work, but more detailed motifs can be found in sst-elements/src/sst/elements/ember/mpi/motifs/. - -Motifs are executed as follows: - -1) The motif generator is initialized (The constructor) -2) The generate function is invoked, places events on the event queue, and returns either true or false -3) The events on the event queue are processed. -4) If the generate function in step 2 returns false, go to step 2. Otherwise, the motif is complete. - -Motifs are intended to be run as a 'job submission.' -The generate function models an entire iteration of an application, using the event queue to mix compute and MPI operations in every iteration. - -### Register Subcomponent - -Ember motifs need to be registered as an SST subcomponent. This typically happens in the header file. -For example: -``` -SST_ELI_REGISTER_SUBCOMPONENT_DERIVED( - Example, - "ember", - "ExampleMotif", - SST_ELI_ELEMENT_VERSION(1,0,0), - "Performs an idle on the node. No traffic can be generated.", - SST::Ember::EmberGenerator - ); -``` - -The parameters of this function are: - -1) class associated with this Motif -3) Aspect of elements being used, in the case of Motifs, "ember." -3) Identifier of the Motif. Note that the end of this parameter must always be Motif -4) SST elements version, -5) Comment describing what the motif does -6) EmberGenerator - -### Writing a constructor -``` -EmberExample::EmberExample(SST::ComponentId_t id, Params& params) : - EmberMessagePassingGenerator(id, params, "Example"), - -``` - - -The constructor also reads parameters from the python script. -The params are passed through the python file in the form: `ep.addMotif("Example firstParam=100 secondParam=200")`. Note no space is allowed before or after the = operator. Parameters read from the python file will be prepended with "arg." before being passed to the C++ file. i.e. "firstParam" becomes "arg.firstParam". -The constructor can be used to perform the setup operations necessary for the generating function. - -The params can be parsed in the C++ file with `firstParam = params.find("arg.firstParam", 100);` where the name of the parameter is "firstParam". - -### Writing a generate() function -``` -bool Example::generate( std::queue& evQ) -``` -The generate function is the 'main' function of a Motif. -If the python file invokes an `addMotif`, the generating function will be invoked until the generate function returns true. -Once the generate function returns, the events queued in the evQ variable will be performed. - -The generate function takes an event queue as a parameter. The event queue allows the user to include computation operations and MPI events in the simulation. Motifs are intended to be designed so that every call to generate() mimics an entire iteration of the application. The events that can be added to the event queue are listed in subsequent sections. - -. - -# User-defined events - -These functions allow the user to control how the simulation estimates computation time. - -* `enQ_compute(Queue&, uint64_t nanoSecondDelay)` -- The delay by the expected cost of the compute operation in nanoSeconds (without actually computing it) -* `enQ_compute( Queue& q, std::function func)`; -- A function is passed as a parameter and invoked. It returns a 64-bit unsigned integer that indicates the simulation delay associated with invoking the function (in nanoseconds). -* `enQ_getTime(evQ, &m_startTime)` -- sets m_startTime to the current time - -There are two types of compute functions that can be enqueued. - -The first computation operation takes a 64-bit integer as a parameter. This parameter is the amount of time needed in nanoseconds to perform the computation. The simulator then delays by that number of nanoseconds as if the actual computation had taken place. -The second compute takes a function as a parameter. When the event is processed, the function is invoked. The function returns the amount of time needed to perform the compute operation. The simulation is delayed by the return value of the function. The time delay could be estimated through a heuristic or measured through representative computation. - - -# MPI Events - - -We list the supported MPI events below. For more detail, see MPI API documentation. - - -* `enQ_init(evQ)` -- MPI initialize -* `enQ_fini(evQ)` -- MPI finalize -* `enQ_rank(evQ, m_newComm[0], &m_new_rank)` -- MPI rank -* `enQ_size(evQ, m_newComm[0], &m_new_size)` -- MPI size -* `enQ_send(evQ, x_neg, x_xferSize, 0, GroupWorld)` -- MPI send -* `enQ_recv(evQ, x_neg, x_xferSize, 0, GroupWorld)` -- MPI recv -* `enQ_isend(evQ, next_comm_rank, itemsThisPhase, 0, GroupWorld, &requests[next_request])` -- MPI isend -* `enQ_irecv(evQ, xface_down, items_per_cell * sizeof_cell * ny * nz, 0, GroupWorld, &requests[nextRequest])` -- MPI irecv -* `enQ_cancel(evQ, m_req[i])` -- MPI cancel -* `enQ_sendrecv(evQ, m_sendBuf, m_messageSize, DATA_TYPE, destRank(), 0xdeadbeef, m_recvBuf, m_messageSize, DATA_TYPE, srcRank(), 0xdeadbeef, GroupWorld, &m_resp)` -- MPI send or recv -* `enQ_wait(evQ, req)` -- MPI wait -* `enQ_waitany(evQ, m_req.size(), &m_req[0], &m_indx, &m_resp)` -- MPI waitany -* `enQ_waitall(evQ, 1, &m_req, (MessageResponse**)&m_resp)` -- MPI waitall -* `enQ_barrier(evQ, GroupWorld)` -- MPI barrier -* `enQ_bcast(evQ, m_sendBuf, m_count, DOUBLE, m_root, GroupWorld)` -- MPI broadcast -* `enQ_scatter(evQ, m_sendBuf, m_sendCnt, m_sendDsp.data(), LONG, m_recvBuf, m_count, LONG, m_root, GroupWorld)` -* `enQ_scatterv(evQ, m_sendBuf, &m_sendCnts[0], m_sendDsp.data(), LONG, m_recvBuf, m_count, LONG, m_root, GroupWorld)` -- MPI Scatter variable message size -* `enQ_reduce( evQ, m_sendBuf, m_recvBuf, m_count, DOUBLE, MP::SUM, m_redRoot, GroupWorld)` -* `enQ_allreduce(evQ, m_sendBuf, m_recvBuf, m_count, DOUBLE, m_op, GroupWorld)` -- MPI allreduce function -* `enQ_alltoall(evQ, m_sendBuf, m_colSendCnt, &m_colSendDsp_f[0], DOUBLE, m_colComm)` -- MPI alltoall with constant message size -* `enQ_alltoallv(evQ, m_sendBuf, &m_colSendCnts_f[0], &m_colSendDsp_f[0], DOUBLE, m_colComm)` -- MPI alltoall with varying message sizes -* `enQ_allgather(evQ, m_sendBuf, m_count, INT, m_recvBuf, m_count, INT, GroupWorld)` -- MPI allgather with messages of constant size -* `enQ_allgatherv(evQ, m_sendBuf, m_sendCnt, INT, m_recvBuf, &m_recvCnts[0], &m_recvDsp[0], INT, GroupWorld)` -- MPI allgather with messages of varying sizes -* `enQ_commSplit(evQ, *comm, color, key, newComm)` -- Splits the MPI comm -* `enQ_commCreate(evQ, GroupWorld, m_rowGrpRanks, &m_rowComm)` -- Creates MPI Com -* `enQ_commDestroy(evQ, m_rowComm)` --Destroys MPI comm From bc78d355ddebb7ec038e080312a27e28735db77d Mon Sep 17 00:00:00 2001 From: Neil Butcher <77634936+nab880@users.noreply.github.com> Date: Mon, 1 May 2023 11:52:14 -0700 Subject: [PATCH 7/8] Delete QuickStart.md delete md files --- src/sst/elements/ember/example/QuickStart.md | 217 ------------------- 1 file changed, 217 deletions(-) delete mode 100644 src/sst/elements/ember/example/QuickStart.md diff --git a/src/sst/elements/ember/example/QuickStart.md b/src/sst/elements/ember/example/QuickStart.md deleted file mode 100644 index 6a3bcba64c..0000000000 --- a/src/sst/elements/ember/example/QuickStart.md +++ /dev/null @@ -1,217 +0,0 @@ -# What is an Ember Motif? - -Ember Motifs provide representative workloads and then estimate performance on hardware using SST simulations. -This guide provides a directions on how to: - -* Implement a empty motif. -* Create and run motif in a SST simulation. - -Implementing and running Ember Motifs requires three different files: - -1. Python File - Provides information on the hardware to be simulated when running an Ember Motif - This directs SST on the available hardware components and how they are set up. -2. C++ file - Decides how the computation is structured and computed. -3. C++ header file - The header file contains primarily naming information - -# Quick Start Guide - -This guide gives a simple example of how to create and run a simple Motif. - -First ensure SST-core and SST-elements are properly installed on your machine. For download and installation see [SST-Downloads](http://sst-simulator.org/SSTPages/SSTMainDownloads/). - -Navigate to `sst-elements/src/elements/ember` inside the sst-elements library. -First, create a new directory to hold our example Motif. - -`mkdir ExampleMotif` - -Next, create a `ExampleMotif/exampleMotif.h` file -``` - #ifndef _H_EMBER_EXAMPLE - #define _H_EMBER_EXAMPLE - - #include "../../mpi/embermpigen.h" - - namespace SST { - namespace Ember { - - class ExampleGenerator : public EmberMessagePassingGenerator { - public: - - SST_ELI_REGISTER_SUBCOMPONENT_DERIVED( - Example, - "ember", - "ExampleMotif", - SST_ELI_ELEMENT_VERSION(1,0,0), - "Performs an idle on the node, no traffic can be generated.", - SST::Ember::EmberGenerator - ); - - SST_ELI_DOCUMENT_PARAMS( - ); - - SST_ELI_DOCUMENT_STATISTICS( - { "time-Init", "Time spent in Init event", "ns", 0}, - { "time-Finalize", "Time spent in Finalize event", "ns", 0}, - { "time-Rank", "Time spent in Rank event", "ns", 0}, - { "time-Size", "Time spent in Size event", "ns", 0}, - { "time-Send", "Time spent in Recv event", "ns", 0}, - { "time-Recv", "Time spent in Recv event", "ns", 0}, - { "time-Irecv", "Time spent in Irecv event", "ns", 0}, - { "time-Isend", "Time spent in Isend event", "ns", 0}, - { "time-Wait", "Time spent in Wait event", "ns", 0}, - { "time-Waitall", "Time spent in Waitall event", "ns", 0}, - { "time-Waitany", "Time spent in Waitany event", "ns", 0}, - { "time-Compute", "Time spent in Compute event", "ns", 0}, - { "time-Barrier", "Time spent in Barrier event", "ns", 0}, - { "time-Alltoallv", "Time spent in Alltoallv event", "ns", 0}, - { "time-Alltoall", "Time spent in Alltoall event", "ns", 0}, - { "time-Allreduce", "Time spent in Allreduce event", "ns", 0}, - { "time-Reduce", "Time spent in Reduce event", "ns", 0}, - { "time-Bcast", "Time spent in Bcast event", "ns", 0}, - { "time-Gettime", "Time spent in Gettime event", "ns", 0}, - { "time-Commsplit", "Time spent in Commsplit event", "ns", 0}, - { "time-Commcreate", "Time spent in Commcreate event", "ns", 0}, - ); - - - }; - } - } - - #endif /* _H_EMBER_EXAMPLE */ -``` -This creates a Motif generator class called ExampleGenerator that inherits from EmberMessagePassingGenerator. The Python code we provide later creates and invokes the generate function. -The SST Document Statistics provides tracking for initialization and various MPI function calls. - -Motifs are executed as follows: - - - -1) The motif generator is initialized (The contructor) -2) The generate function is invoked and returns either true or false -3) The events on the eventQueue are processed. -4) If the generate function in step 2 returned false, return to step 2, otherwise the motif is complete. - -Here is a motif that returns without doing any additional work. - -Then we create a file `ExampleMotif/Example.cc` with the contents: - -``` - #include - #include "Example.h" - - using namespace SST:Ember; - - ExampleGenerator::ExampleGenerator(SST::ComponentId_t id, Params& params) : - EmberMessagePassingGenerator(id, params, "Null" ) - { - } - - bool ExampleGenerator::generate( std::queue& evQ) - { - // Code here is repeated until true is returned. - return true; - } -``` - - -Motifs are intended to be ran as a 'job submission'. -The generate function models an entire iteration of an application. The event queue allows the motif to mix compute and MPI operations in every iteration. - - - -The C++ file loads the SST:Ember namespace giving it access to other Motifs and EmberEvents. -Each Motif generator has an constructor and generate function. - - -Then we add the `.cc` and `.h` file to `Makefile.am` - -i.e. -``` - embermemoryev.h \ - ExampleMotif/example.h \ - ExampleMotif/example.cc \ - libs/emberLib.h \ -``` - -Finally, a python file `example.py` needs to be created: - -``` - from email.mime import base - import sst - from sst.merlin.base import * - from sst.merlin.endpoint import * - from sst.merlin.interface import * - from sst.merlin.topology import * - - from sst.ember import * - - def example(): - PlatformDefinition.setCurrentPlatform("firefly-defaults") - - ### Setup the topology - topo = topoDragonFly() - topo.hosts_per_router = 2 - topo.routers_per_group = 4 - topo.intergroup_links = 2 - topo.num_groups = 4 - topo.algorithm = ["minimal","adaptive-local"] - - # Set up the routers - router = hr_router() - router.link_bw = "4GB/s" - router.flit_size = "8B" - router.xbar_bw = "6GB/s" - router.input_latency = "20ns" - router.output_latency = "20ns" - router.input_buf_size = "4kB" - router.output_buf_size = "4kB" - router.num_vns = 2 - router.xbar_arb = "merlin.xbar_arb_lru" - - topo.router = router - topo.link_latency = "20ns" - - ### set up the endpoint - networkif = ReorderLinkControl() - networkif.link_bw = "4GB/s" - networkif.input_buf_size = "1kB" - networkif.output_buf_size = "1kB" - - ep = EmberMPIJob(0,topo.getNumNodes()) - ep.network_interface = networkif - ep.addMotif("Example") - ep.nic.nic2host_lat= "100ns" - - system = System() - system.setTopology(topo) - system.allocateNodes(ep,"linear") - - system.build() - - if __name__ == "__main__": - example() -``` - -This follows common python syntax. -The hardware variables (topology, router and network interface) are created through assignment of a constructor -The fields of the variables are accessed using a dot operator. -Topology, router, and network interface variables need to be created for a simulation. More detailed descriptions of configuration options is in RunningMotifs.md. -Next an endpoint is created and given the motif to be used using a ep.addMotif("MotifName")call -Note multiple motifs can be added the same endpoint. The intended use for each motif to simulate an entire application. The event queue is used to simulate MPI events and computational workloads. Adding multiple motifs is useful for simulating workflows of a series of applications. - -Finally, a system variable is created and 'built' - -Remake SST-elements. - -To run the python script - -``` -sst example.py -``` - - - From b17ed6f429f015dbb394cc97cfe1d7e0d1ffdf5a Mon Sep 17 00:00:00 2001 From: Neil Butcher <77634936+nab880@users.noreply.github.com> Date: Mon, 1 May 2023 11:53:13 -0700 Subject: [PATCH 8/8] Delete RunningMotifs.md --- .../elements/ember/example/RunningMotifs.md | 190 ------------------ 1 file changed, 190 deletions(-) delete mode 100644 src/sst/elements/ember/example/RunningMotifs.md diff --git a/src/sst/elements/ember/example/RunningMotifs.md b/src/sst/elements/ember/example/RunningMotifs.md deleted file mode 100644 index 3244247705..0000000000 --- a/src/sst/elements/ember/example/RunningMotifs.md +++ /dev/null @@ -1,190 +0,0 @@ -# Ember Python Configurations - -The Python file in Ember specifies: -- How to set up the SST simulation -- Allows multiple configurations to be specified and run -- Loads Motifs into the simulation to be performed. - -# Running Ember - -Ember uses a python interpreter built into the SST compiler. -`sst python-file.py` executes the python script. - -# Key Components - -The python file specifies the format of the SST simulation. The python file specifies components of our simulation such as network Topology, Routers, NetworkInterface and the computation to be performed in the form of Motifs. - -First, we import the necessary components of merlin and set the Platform to firefly defaults. - - import sst - from sst.merlin.base import * - from sst.merlin.endpoint import * - from sst.merlin.interface import * - from sst.merlin.topology import * - - from sst.ember import * - - if __name__ == "__main__": - - PlatformDefinition.setCurrentPlatform("firefly-defaults") - -Next, we create a topology that our experiment will run on. - -`topo = topoDragonFly()` -Creates a dragonfly topology. -We can then specify parameters about the topology using a dot-operator. e.g., -`topo.num_groups = 4` - -# Topologies - -The python file allows the user to specify comparable hardware configurations. -Four different topologies can be specified with different routing algorithms. -Here we give a comprehensive list of the topologies and how they can be initialized. - -1. HyperX `topoHyperX` - * `shape` - Specifies the shape of the mesh. i.e. topo.shape=4x4 or topo.shape=4x4x4. Any number of dimensions is supported. - * `width` - The number of links between routers in each dimension is specified in the same manner as shape. - * `local_ports` - Number of endpoints attached to each router - * `algorithm` - Routing algorithm to use. - * `DOAL` - * `DOR` - * `DOR-ND` - * `MIN-A` - * `VDAL` - * `valiant` -2. Torus `topoTorus` - * Shape - Specifies the mesh grid size, i.e., topo.shape=4x4 or topo.shape=4x4x4. Any number of dimensions is supported. - * Width - Number of links between routers in each dimension, specified in the same way as shape. - * local\_ports - Number of endpoints attached to each router -3. Fat Tree `topoFatTree` - * `shape` - The shape of the fat tree. Specified as pair of downlinks and uplinks per level separated by a colon. i.e., topo.shape="4,2:3,5" specifies the first level has 4 down and 2 up, and the following level has 3 down and 5 up. - If only one number is given, it is interpreted as the number of downlinks and up is set to 0. - * `routing_alg` Routing algorithm to use - * `deterministic` (default) - * `adaptive` - * `adaptive threshold.` - The threshold is used to determine if a packet will adaptively route. -4. Dragonfly `topoDragonFly` - - * `hosts_per_router` - Number of hosts connected to each router - * `routers_per_group` - The number of links used to connect to routers in the same group. - * `intergroup_per_router` - The number of links per router connected to other groups. - * `intergroup_links` - The number of links between each pair of groups. - * `num_groups` - Number of groups in a network - * `intergroup_links` - * `algorithm` - Specifies how each virtual network routes messages. - Specified for each router.vn. i.e., if router.vn=2, than topo.algorithm=\["minimal", "adaptive-local"], the first router uses minimal, and the second will use adaptive-local. - * `minimal` (default)\ - * `adaptive_local` - * `ugal` - * `min-a` - * `valiant` - Valiant only operates if there are more than two num\_groups. Otherwise, there is no point in using valiant routing. - * `adaptive_threshold` - Threshold when making adaptive routing decisions. - * `global_link_map` - An array specifying connectivity of global links in each dragonfly group - * `global_route_mode` - Mode for interpreting global link map - * `absolute` (default) - * `relative` - -# Creating a Router - -Once the router has been created, the topology needs to be linked to the router. Additionally, the link\_latency can be set in the topology at this point. - - topo.router = router - topo.link_latency = "20ns" - -Parameters for a high radix router or `hr_routers`: -* `id` -The ID of the router -* `num_ports` -Number of ports the router has -* `topology` -Name of the topology subcomponent loaded to control routing -* `xbar_arb` -Arbitration unit used for the crossbar. Default is `merlin.xbar_arb_lru`. -* `merlin.xbar_arb_lru` -Uses Least recently used arbitration for `hr_router.` -* `merlin.xbar_arb_age` -Age-based arbitration unit for `hr_router` -* `merlin.xbar_arb_lru` -Least recently used arbitration unit with infinite crossbar for `hr_router` -* `merlin.xbar_arb_rand` -Random arbitration unit for `hr_router` -* `merlin.xbar_arb_rr` -Round-robin arbitration unit for `hr_router` -* `link_bw` -Bandwidth of the links specified in either b/s or B/s (can include SI prefix) -* `flit_size` -Flit size specified in either b or B -* `xbar_bw` -Bandwidth of the crossbar specified in either b/s or B/s -* `input_latency` -Latency of packets entering switch into input buffers. Specified in s. -* `output_latency` -Latency of packets exiting switch from output buffers. Specified in s. -* `network_inspectors` -Comma-separated list of network inspectors to put on output ports -* `oql_track_port` -Set to true track output queue length for an entire port. False tracks per VC (default). - -# Creating Network Interface - -Create the network interface: - - networkif = ReorderLinkControl() - -The ReorderLinkControl() creates a network interface that can handle out-of-order packet arrival. Events are sequenced, and order is reconstructed on receive. - -# Specifying Computation - -Initialize the MPIJob to use all the nodes in our topology: - -`ep = EmberMPIJob(0, topo.getNumNodes())` -The network interface then needs to be linked to the `ep` variable -`ep.network_interface = networkif()` -Then a series of Motifs can be queued for computation. -``` - ep.addMotif("Init") - ep.addMotif("Allreduce") - ep.addMotif("Fini") -``` -The `addMotif` function adds the specified Motif to a queue. The Motif is named through an SST\_ELI\_REGISTER\_SUBCOMPONENT\_DERIVED command in the C++ Motif definition (usually in the include file). The SST\_ELI\_REGISTER\_SUBCOMPONENT\_DERIVED parameter follows the naming convention "ExampleMotif", and to add the Motif to `ep` using `ep.addMotif("Example")` The "Motif" portion is implied in the naming. -Parameters can be passed to motifs through the string. The parameters are read as a list of assignments, separated by whitespace. For example, a motif 'Sum' that takes three integers as a parameter named x, y, and z -The Motif would be invoked `ep.addMotif("Sum x=4 y=5 z=6")` would pass the arguments as args.x, args.y, and args.z with assigned values 4, 5, and 6, respectively. The arguments are passed in a Param object to the motif generator to be parsed. - -Some additional functions that can be called on an endpoint or the `ep` variable: -* getName() -Returns the name of the ep. i.e., it will return "EmberMPIJob" -* enableMotifLog() -Starts logging the Motifs as they are executed. - -Some common MPI-function call motifs that exist in Ember: - -# Running the simulation - -Finally, we create the 'system', which then runs the simulation. The topology is set with the setTopology function, and the endpoint is specified by system.allocateNodes. Build executes the motifs in the endpoint. -``` - system = System() - system.setTopology(topo) - system.allocateNodes(ep,"linear") - system.build() -``` -While most of the System is self-explanatory, the system allows the user to specify how threads/processes are assigned to nodes in the allocateNodes function. -The allocateNodes function allows the user to specify how the nodes are allocated. In the example, "linear" is chosen. In linear, the nodes are sorted, and threads are placed in linear order onto the simulated nodes. There are additional ways to allocate nodes, such as "random", "interval", and "indexed".