Enum for memory modes

[timmoon10]

We currently specify memory modes for the Memory class with an unsigned int:

https://github.com/LLNL/Elemental/blob/ae00b5b727b0847f46f264eb5fc2b39c02c83bca/include/El/core/Memory/decl.hpp#L62

Things are messy since the same number means different things on different devices (mode 1 means pinned memory on CPU and CUB memory pool on GPU). This was not a good design (mea culpa). Imagine an idyllic future with something like:

Matrix<float,Device::CPU> A;
A.MemoryMode(); // Returns memory_mode_type::default
A.SetMemoryMode(memory_mode_type::pinned_memory);

This would require making simultaneous changes in LBANN.

[ndryden]

I have no substantial comment, other than to say that this should be done. We may want to be even more general, but this seems like a good step.