The ai.djl.Model.getBlock() returns null OR - NullPointerException ai.djl.inference.Predictor.predictInternal()

[JeeDevUser]

Hi all,

I am trying to do text summarization by using Pegasus-xsum
Scenario:

• downloaded all files under 'Files and versions' section, to the local drive, from location:
  https://huggingface.co/google/pegasus-xsum/tree/main
• the relevant part of code:

String modelPath = "d:\\...\\pegasus-xsum\\";

    String inputText = "The input text that should be summarized";

     // 1. Tokenizer initialization:
     HuggingFaceTokenizer tokenizer = HuggingFaceTokenizer.newInstance(Paths.get(modelPath));

     // 2. Tokenization of input text
     Encoding encoding = tokenizer.encode(inputText);
     long[] tokenizedInput = encoding.getIds(); 

     System.out.println("tokenizedInput: " + Arrays.toString(tokenizedInput));

     ai.djl.Model model = Model.newInstance(modelPath);
     System.out.println("Model loaded successfully: " + model);
     System.out.println("model.getBlock(): " + model.getBlock());

THE PROBLEM: the last sysout prints null.
-Because of that, I am getting the

java.lang.NullPointerException
 at ai.djl.inference.Predictor.predictInternal(Predictor.java:147)

What I am missed?

[frankfliu]

@JeeDevUser

DJL can only load torchscript (or onnx, tensorflow saved-model bundle) format, you need to use jit.trace to convert to torchscript (or onnx) format.

This is a text generation model, it's quite complicated. We cannot use djl-convert to automatically convert to DJL format. You can take a look this example to run it in java: https://github.com/deepjavalibrary/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/nlp/TextGeneration.java

You might want to take a look:

1. https://github.com/deepjavalibrary/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/nlp/TextGeneration.java#L66
2. https://github.com/deepjavalibrary/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/nlp/TextGeneration.java#L171
3. the model contains keras weight, you might want to try to convert to tensorflow saved-model bundle format, and load with DJL

[JeeDevUser]

Thanks for reply!
I am just converter pytorch_model.bin to the TorchScript format (via Python script), and got the
pegasus_xsum_torchscript.pt file. Now I am building the model as follows:

 Model model = Model.newInstance("/path/toThe/pegasus_xsum_torchscript.pt", "PyTorch");
 System.out.println("Model loaded successfully: " + model);
 System.out.println("model.getBlock(): " + model.getBlock());

But the problem did not go away :-(, the last sysout prints null again:

Model loaded successfully: Model (
	Name: d:\IDEJE_ZA_AI_APP\Hugging Face Modeli\pegasus-xsum\pegasus_xsum_torchscript.pt
	Data Type: float32
)
model.getBlock(): null

Can i even use Pegasus-xsum this way? The goal is to summarize the text.
Is there any DJL example which does Text sumarization, in any way?

[frankfliu]

You need load the model:

Model model = Model.newInstane("pegausus", "PyTorch");
model.load(Paths.get("/path/toThe/pegasus_xsum_torchscript.pt");

[JeeDevUser]

@frankfliu , Well done! I don't know how I missed it!
now i get the following block:
model.getBlock(): PtSymbolBlock

Thanks...now i have a new problem but i will open a new topic