Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte #931

Open
Anas0x45 opened this issue Sep 12, 2024 · 0 comments

Comments

@Anas0x45
Copy link

Description
I encountered a UnicodeDecodeError when running the Triton Model Analyzer in a Docker container with GPU support. The issue seems to occur when retrieving the GPU device name. Here’s the traceback:
Traceback (most recent call last):
File "/usr/local/bin/model-analyzer", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/model_analyzer/entrypoint.py", line 263, in main
gpus = GPUDeviceFactory().verify_requested_gpus(config.gpus)
File "/usr/local/lib/python3.10/dist-packages/model_analyzer/device/gpu_device_factory.py", line 39, in init
self.init_all_devices()
File "/usr/local/lib/python3.10/dist-packages/model_analyzer/device/gpu_device_factory.py", line 71, in init_all_devices
device_name = device_atrributes.deviceName
File "/usr/local/lib/python3.10/dist-packages/model_analyzer/monitor/dcgm/dcgm_structs.py", line 464, in getattribute
return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte

Triton Information
Docker image: nvcr.io/nvidia/tritonserver:24.07-py3-sdk

To Reproduce

  • Running Triton Model Analyzer in Docker with GPU support (docker run --gpus all).
  • The error occurs when retrieving the GPU device name using dcgm_structs.py.

Solution that works for me:
Adding errors="ignore" or errors="replace" to the decoding logic temporarily fixes the crash but may cause data loss.

class _DcgmStructure(Structure):

    def __getattribute__(self, key):
        value = super().__getattribute__(key)
        if isinstance(value, bytes):
            try:
                return value.decode("utf-8")
            except UnicodeDecodeError as e:
                print(f"Failed to decode bytes: {e}")
                return value.decode("utf-8", errors="replace")
        if isclass(value):
            return _WrappedStructure(value)
        return value
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant