Heap Memory error while generating EvaluatedModel Report for large projects

[losmateusz1]

Description:
When attempting to generate an EvaluatedModel report for larger analyzer result files (~55MB), ORT encounters a heap memory error, causing the process to fail.
Details:
The error appears to occur in the valueToTree method, located in the EvaluatedModel reporter module:
EvaluatedModel.kt#L174.
This issue arises while reading the object into a tree using Jackson's ObjectMapper at the following line:
ObjectMapper.java#L3151.
The current implementation struggles to handle large data sizes, leading to excessive memory consumption.

Questions and Suggestions:

1. Are there potential optimizations in this part of the code to address the memory limitations?
2. Would it be feasible to introduce a mechanism to process large analyzer results incrementally or in smaller chunks?
3. Could alternative serialization methods or configurations within Jackson help mitigate this issue?

[mnonnenmacher]

You could try to enable the deduplicateDependencyTree option of the EvaluatedModelReporter, it significantly reduces the memory consumption. You can enable it by passing -O EvaluatedModel=deduplicateDependencyTree=true to the report command.

[sschuberth]

FYI @Etsija, this might also be an option for us.

[Etsija]

It was indeed the answer - now the Reporter runs happily on my Analyzer+Scanner+Evaluator result file on ORT core.

[losmateusz1]

Thank you very much for the answer, we tested with this option and it works now.

[heliocastro]

One information on top, we used to push this in one single run, - f WebApp,EvaluateModel but then we couldn't pass the options at same time for both.
We explicitly separate that now in two runs.

[mnonnenmacher]

It should be possible to pass -O EvaluatedModel=deduplicateDependencyTree=true and -O WebApp=deduplicateDependencyTree=true at the same time.

[Etsija]

@mnonnenmacher is deduplicateDependencyTree option reporter specific? WebApp runs happily now with it, but CycloneDX doesn't, so is the option missing from it? What about SPDXDocument?

Trying with this, using 16GB for my Java:

ort report -i "$EVALUATE_RESULT" -o "$OUTPUT_DIR" -f CycloneDX -O CycloneDX=deduplicateDependencyTree=true

[Etsija]

Answering to myself: from looking at CycloneDXReporter.kt and SPDXDocumentReporter.kt, they don't support the said option yet.

[mnonnenmacher]

Correct, the option only exists for the EvaluatedModel and WebApp reporters (note that the WebApp reporter internally uses the EvaluatedModel reporter). The reason is that when the option was implemented only those two reporters suffered from the memory issues, the other reporters worked fine.

[Etsija]

OK would a ticket be OK to create for those two reporters as well?

[mnonnenmacher]

Sure, if it is required for more reports maybe it should become a global option, but let's discuss that in the ticket you create.

[Etsija]

I created #9889.