unexpected explicit tensor data for input tensor 'attention_mask' for model 'pipeline-poc-inference__isvc-211152d1e7' of type 'INT32', expected datatype 'INT64'

[MLHafizur]

Hi deployed an ONNX model on ModelMesh serving on Triton Runtime. Now trying to create python grpc client. But getting the following issues:

Got the model metadata:
platform: "onnxruntime_onnx"

inputs {
  name: "attention_mask"
  datatype: "INT64"
  shape: 32
  shape: 64
}
inputs {
  name: "input.1"
  datatype: "INT64"
  shape: 32
  shape: 64
}
outputs {
  name: "1643"
  datatype: "FP32"
  shape: 32
  shape: 16
}

But now struggling to figure out the input data type issue, although tried in many different ways:

Traceback (most recent call last):
  File "predict.py", line 97, in <module>
    response = grpc_stub.ModelInfer(request)
  File "/home/hafizur/miniconda3/envs/onnx/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/hafizur/miniconda3/envs/onnx/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "inference.GRPCInferenceService/ModelInfer: INVALID_ARGUMENT: unexpected explicit tensor data for input tensor 'attention_mask' for model 'pipeline-poc-inference__isvc-211152d1e7' of type 'INT32', expected datatype 'INT64'"
        debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:8033 {created_time:"2023-02-22T15:50:59.6900669-05:00", grpc_status:3, grpc_message:"inference.GRPCInferenceService/ModelInfer: INVALID_ARGUMENT: unexpected explicit tensor data for input tensor \'attention_mask\' for model \'pipeline-poc-inference__isvc-211152d1e7\' of type \'INT32\', expected datatype \'INT64\'"}"

The input datas are like:
[[  101  2424  2041 ...  1997  1037   102]
 [  101  1996  2343 ...  9228  2003   102]
 [  101  9710 22002 ...  1010  2256   102]
 ...
 [  101 26624  2139 ...  2055  4825   102]
 [  101  2508 22889 ...     0     0     0]
 [  101 10352 10958 ...  2053  2386   102]]
[[1 1 1 ... 1 1 1]
 [1 1 1 ... 1 1 1]
 [1 1 1 ... 1 1 1]
 ...
 [1 1 1 ... 1 1 1]
 [1 1 1 ... 0 0 0]
 [1 1 1 ... 1 1 1]]
b_input_ids data type: int64
b_input_ids shape: (32, 64)
b_input_mask data type: int64
b_input_mask shape: (32, 64)

Here are the final codes I tried to run:

def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()

input_path = "."

# load the needed input (as needed by ONNX)
dataloader = joblib.load(os.path.join(input_path, "eval_dataloader.pkl"))

# pick a sample batch as sample input
batches = [batch for batch in dataloader]
_batch = batches[0]
# inputs needed for the model
#b_input_ids = _batch[0].to('cpu').long().to(torch.int64)
b_input_ids = _batch[0].to('cpu')
b_input_ids = to_numpy(b_input_ids)
#b_input_mask = _batch[1].to('cpu').long().to(torch.int64)
b_input_mask = _batch[1].to('cpu')
b_input_mask = to_numpy(b_input_mask)
print(b_input_ids)
print(b_input_mask)

print("b_input_ids data type:", b_input_ids.dtype)
print("b_input_ids shape:", b_input_ids.shape)
print("b_input_mask data type:", b_input_mask.dtype)
print("b_input_mask shape:", b_input_mask.shape)

# Send request to the server
grpc_channel = grpc.insecure_channel("localhost:8033")
grpc_stub = service_pb2_grpc.GRPCInferenceServiceStub(grpc_channel)


model_name = "pipeline-poc-inference"
model_version = ""


request = service_pb2.ModelMetadataRequest(name=model_name,
                                               version=model_version)
response = grpc_stub.ModelMetadata(request)
print("model metadata:\n{}".format(response))

#b_input_mask = b_input_mask.astype('int64')


#b_input_mask = b_input_mask.astype(np.int64)

b_input_mask = np.array(b_input_mask, dtype=np.int64)




# Infer
request = service_pb2.ModelInferRequest()
request.model_name = model_name
request.model_version = model_version
request.id = "my request id"

input0 = service_pb2.ModelInferRequest().InferInputTensor()
input0.name = "attention_mask"
# input0.datatype = "INT64"
# input0.shape.extend([32, 64])
# #input0.contents.int_contents[:] = b_input_mask
# #input0.contents.int_contents[:] = b_input_mask.tolist()
# #input0.contents.int_contents[:] = list(map(int, b_input_mask.tolist()))
# input0.contents.int_contents[:] = list(map(int, b_input_mask.ravel().tolist()))


input1 = service_pb2.ModelInferRequest().InferInputTensor()
input1.name = "input.1"
# input1.datatype = "INT64"
# input1.shape.extend([32, 64])
# #input0.contents.int_contents[: : ] = b_input_ids
# #input1.contents.int_contents[:] = b_input_ids.tolist()
# #input1.contents.int_contents[:] = list(map(int, b_input_ids.tolist()))
# input1.contents.int_contents[:] = list(map(int, b_input_ids.ravel().tolist()))

input0.datatype = "INT64"
input0.shape.extend([b_input_mask.shape[0], b_input_mask.shape[1]])
input0.contents.int_contents.extend(b_input_mask.ravel().tolist())

input1.datatype = "INT64"
input1.shape.extend([b_input_ids.shape[0], b_input_ids.shape[1]])
input1.contents.int_contents.extend(b_input_ids.ravel().tolist())


request.inputs.extend([input0, input1])

output0 = service_pb2.ModelInferRequest().InferRequestedOutputTensor()
output0.name = "1643"
request.outputs.extend([output0])

response = grpc_stub.ModelInfer(request)

print("response:\n{}".format(response))

What could be the issue?

[dyastremsky]

You should be able to resolve this issue by changing list(map(int, b_input_mask.ravel().tolist())) to list(map(int64, b_input_mask.ravel().tolist())). In the future, if you run into an issue like this, checking the type of your input at each step via type() can be helpful. Triton is expecting an int64 due to the model config and input datatype, but the data is int32.