Performance

[paulobreim]

I would like to know who is using IPED effectively, that is, to work on an expert investigation.
I'm indexing 9 hard drives of 1TB each, using a computer with:
Xeon E5-2658A
64GB Ram
SSD 1.5TB
To index the 1TB HD it takes from 16 hours to 30 hours.
I'm using a forensic profile.

I don't know how to analyze why it is taking so long. Would anyone know how to say?
Thank you,
Paulo

[wladimirleite]

Can you share a log file?
It doesn't need to be the full file, I would like to take a look at its beginning/end. Something like the first and the last 1000 lines should be enough.

[lfcnassif]

Check if you are following the updated performance tips: https://github.com/sepinf-inc/IPED/wiki/Performance-Tips

[lfcnassif]

Just reviewed Tika's OfficeParserConfig options and they have an "includeShapeBasedContent" option defaulted to true, maybe disabling that could help. Go to conf/ParserConfig.xml and try adding the following parameter to OOXMLParser config block:

<param name="includeShapeBasedContent" type="bool">false</param>

[paulobreim]

Good! Since I started with your suggestion using java -Xms30G -Xmx30G and disable enabledExternalParsing , look the diference compared with picture that I sent before.

Practically finished, but now it is just 1 task parsing a trashfile .xlsx with 224MB.
Apparently stopped because the end time no longer updates.
The log is giant.

I will wait by 1 hour ans if not finished I will cancel and retar with --continue.

tks

[lfcnassif]

Great! Stopping and using --continue probably will not work, the xlsx file processing will be retried, unless you make an explicit configuration to ignore the file as I pointed. Possibly it is better to wait the timeout to be triggered.

Providing this and other xlsx files would help us trying to improve Tika/POI xlsx parsing...

[paulobreim]

uauuu logfile has 5GB.
I will get it.

And I will run again with false

And get some .xlsx files.

[paulobreim]

@tc-elite really the problem is when we have large xls files. I sent the log and 10 xlsx files do @lfcnassif and now we have examples to analise the problem.

[paulobreim]

New Tests.
I finished out 9 complete tests, all with HDs of real cases, 8 which 1TB and 1 witch 512GB, using the IPED version 4 with forensic profile.
I used 2 computers 1 with 64GB and the other with 16GB. It was only possible to finish all of them on the 64GB computer with the suggested java parameters.
I will do new tests with the next version.

[lfcnassif]

You were possibly affected by #539