implement nonblocking vector collectives using generalized requests

[jeffhammond]

It's possible to this using a thread, perhaps just using the blocking collective implementation hanging off of a thread.

[gentryx]

Question: can't you use MPI_Ineighbor_alltoallw() instead of MPI_Neighbor_alltoallw() in src/vcollectives_x.c? Opening a new thread seems expensive and might be difficult to port.

[jeffhammond]

You are a co-author of a paper that contains the following:

Unfortunately, nonblocking v-collectives cannot be implemented by using the aforementioned approaches. In the case of the neighborhood collective implementation, we cannot free the vector temporaries holding the counts, displacements, and datatypes until the operation has completed. If callback functions associated with request completion were present in the MPI standard (see [17] for a proposal of this), then it would be possible to free the temporary buffers using this callback. Since one cannot associate a single request with multiple nonblocking operations, the point-to-point implementation is not viable for the nonblocking v-collectives.

;-)

[gentryx]

D'oh!
/hangs head low

[gentryx]

After reading this part of the standard, I assume it's possible to implement this via the non-blocking variant and generalized requests, but without threads. We can free the temporary arrays in the callback functions, right?

[jeffhammond]

Unfortunately, generalized requests are useless without threads. This is a well-known flaw. I think it is covered in http://www.mcs.anl.gov/~thakur/papers/grequest-redesign.pdf. If not there, then I look up the details, but the basic idea is that generalized requests cannot make progress by any means other than threads. Querying the status does not cause them to make progress.

We can use MPICH generalized requests instead of threads, but that is not a portable implementation. However, since OpenMPI does not support MPI_THREAD_MULTIPLE, it is not really less portable to assume MPICH extended generalized requests versus MPI generalized requests with threads.

Another solution, which works just fine for users but I prefer not to implement in BigMPI itself, is to create a new request object and associated query functions that can store and manage the temporaries. If we added a BigMPI_Request type, it could hold temporary buffers (for the ALLTOALLW implementation) or a vector of MPI_Requests (for the point-to-point implementation). Then we would need BigMPI_{Test,Wait}, at which point BigMPI is no longer a simple drop-in replacement.

[gentryx]

Hmm, maybe I'm (again) missing something obvious, but does BigMPI actally need to manually drive progress? I thought it was sufficient to block for the requrest returned by MPI_I${COLLECTIVE}v in the callback functions query_fn, free_fn, cancel_fn -- these are invoked from MPI on user request.

[jeffhammond]

No, that's the problem with generalized requests. If you create one then
wait on it, it will hang forever unless there is a thread driving progress
out-of-band.

And you probably wonder how such a useless feature got into the standard in
the first place. Welcome to design-by-committee :-)

On Thu, Oct 29, 2015 at 10:54 AM, Andreas Schäfer [email protected]
wrote:

Hmm, maybe I'm (again) missing something, but does BigMPI actally need to
manually drive progress? I thought it was sufficient to block for the
requrest returned by MPI_I${COLLECTIVE}v in the callback functions
query_fn, free_fn, cancel_fn -- these are invoked from MPI on user request.

—
Reply to this email directly or view it on GitHub
#27 (comment).

Jeff Hammond
[email protected]
http://jeffhammond.github.io/

[gentryx]

Wow... :-(