Will int8 PTQ reduce VRAM for VGG-19?

[jonahclarsen]

Hi all,

I primarily want to use Torch-TensorRT to make VGG-19 take up 3-4x less space in VRAM than in plain Libtorch full-precision, with int8 PTQ. I am working on testing it myself but haven't yet been able to get PTQ working (#1091).

Does anyone have experience using PTQ with VGG who can comment on if VGG-19 will use significantly less VRAM after int8 PTQ?

Thanks!

[peri044]

a pytorch nn.module (with FP32) weights vs INT8 TRT engine embedded in a torchscript module - The latter would consume less memory. However, I don't know if the memory savings would be 3-4x. You can try running a python example and check. (For reference: https://github.com/pytorch/TensorRT/blob/master/tests/py/test_ptq_dataloader_calibrator.py)

[jonahclarsen]

Thanks for your reply. I've yet to get Linux set up, or the Windows build working with PTQ, but once I do, I'll definitely check that out. I was just hoping someone might have insight in the meantime.