Implement Tensor.type

[tfogal]

🚀 Feature

Implement Tensor.type

Motivation

NeMo text-to-image model.

cc @tfogal

[k223kim]

Hi Team! I have made a rough draft of the implementation of Tensor.type. I was not sure how to implement non_blocking at the moment. As per this description, it states the following:

If True, and the source is in pinned memory and destination is on the GPU or vice versa, the copy is performed asynchronously with respect to the host. Otherwise, the argument has no effect.

How do I know the source and destination from the input arguments of Tensor.type? Also, I am not sure where to put the test case for this implementation. Any suggestions and feedback would be appreciated!

[t-vi]

@tfogal It's cool that we want to support it, but in terms of good style, NeMo should really be using .to, .is_cuda and .dtype instead, which is the much more standard way to do what is does (and much more efficient).

[tfogal]

triage team: let's figure out how to guide Kaeun here.

@t-vi ack. need to figure out where these are coming from (could be megatron core, for example) before I can track on GitHub, but I added it to my personal list.