[OV]: load and convert llms in original precision

[eaidova]

What does this PR do?

allow loading bfloat16 and float16 models in original precision for conversion. It significantly reduces memory consumption and loading time during model conversion for large models

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[echarlaix]

why only for text-generation tasks ?

[eaidova]

in future we will propagate for other model types. text-generation models (especially LLMs) in most cases suffer for ineffective weights conversion when we load weights saved in float16/bfloat16 and implicit convert them to float32 when call from_pretrained method. The idea behind these changes is allowing load model as is for avoid weights conversion (it reduces memory footprint, e.g. instead of ~27GB RAM for 7b models you need only 13.5GB for that) and also speedup conversion of some operations with weights like linear or matmul converting them directly to ov without passing tracing.

[echarlaix]

could we instead update and use ov_config.dtype instead of adding patch_16bit_model ?

[eaidova]

layers patching is time consuming process we would like to avoid doing that for cases where it is not really neseccary. We need it only for pytorch model wegiths represented in fp16 and bf16 format for avoid issues tracing on cpu (torchscript does not support some ops running on cpu in this format). ov_config.dtype currently used for output model type and signal to compress weights to fp16 after conversion, mixing them in 1 parameter may lead to some issues:

• inability to convert fp32 pytorch models to fp16 openvino model (now it is possible)
• compressing model weights twice (having model in fp16 on torch side and then repeat compression for openvino)

[echarlaix]

Looks great, thanks a lot @eaidova

[eaidova]

Looks great, thanks a lot @eaidova

@echarlaix thanks, we still investigating impact on models accuracy and quantization on our side. Could you please do not merge these changes, until we do not have whole picture?

[eaidova]

@IlyasMoutawwakil could you please merge?