Topic/cuda aware communications

[bosilca]

Add support for sending and receiving the data directly from and to devices. There are few caveats (noted on the commit log).

1. The first question is how is such a device selected ?

The allocation of such a copy happen way before the scheduler is invoked
for a task, in fact before the task is even ready. Thus, we need to
decide on the location of this copy only based on some static
information, such as the task affinity. Therefore, this approach only
works for owner-compute type of tasks, where the task will be executed
on the device that owns the data used for the task affinity.

2. Pass the correct data copy across the entire system, instead of
   falling back to data copy of the device 0 (CPU memory)

TODOs

• rebase on c11 atomic fix
• Add a configure option to enable GPU-aware communications.
• Add a runtime configuration to turn on/off the gpu-aware comms?
• Pass -g 2 tests
• Failure with ctest get_best_device scheduling.c:157: int __parsec_execute(parsec_execution_stream_t *, parsec_task_t *): Assertion NULL != copy->original && NULL != copy->original->device_copies[0]'
• Failure with ctest nvlink, stress (segfault)
• Failure with ctest stage (presumably identical to intermittent failure in gemm/potrf) device_gpu.c:2470: int parsec_device_kernel_epilog(parsec_device_gpu_module_t *, parsec_gpu_task_t *): Assertion PARSEC_DATA_STATUS_UNDER _TRANSFER == cpu_copy->data_transfer_status' failed.
• RO data between tasks may reach an assert when doing D2D between devices that do not have peer_access between them
• readers values are miscounted when 2 or more GPUs are used per rank Topic/cuda aware communications #671 (comment)

[abouteiller]

Now passing 1-gpu/node, 8 ranks PTG POTRF
Sorry I had to force-push there were issues with rebasing on master