Nitrite/Java: Question about Processor in 4.x branch

[sea-steve]

Hi,

I've been experimenting with Nitrite 4.x and am currently battling an issue with Processor causing a long delay at startup of an app that uses an MVStore db. A database of about 200K records used to open very quickly under 3.x, now takes about 15 minutes for the app to start up. I've confirmed that this is due to adding a Processor to my ObjectRepository's after opening them. My code looks something like this:

ObjectRepository myModelRepo = nitriteDb.getRepository(MyModel.class);
myModelRepo.addProcessor(myProcessor);

The Processor is for writes -- it adds/updates dateCreated, dateUpdated, and a UUID values to my model objects. I've confirmed that the Processor is the culprit by commenting out the "addProcessor()" line above: after doing so, the app starts up in a few seconds.

The root cause seems to be the "doProcess(processor)" line in the addProcessor() method here: https://github.com/nitrite/nitrite-java/blob/4.x/nitrite/src/main/java/org/dizitart/no2/collection/operation/CollectionOperations.java#L80

The doProcess() method appears to iterate over all records in the collection and perform an update. It's unclear to me why adding a Processor to an ObjectRepository would have this behavior? But, maybe I don't understand the purpose of Processor: I had assumed that it was something like an Oracle Trigger.

Is there something I'm misunderstanding about how a Processor works or its intended use?

BTW, thanks so much for Nitrite -- it's one of the easiest Java persistence solutions I've ever used.

Thanks

Steve

[anidotnet]

Good catch! The processor api is still work in progress. I have made some changes in the processor api in 4.x branch. Let me know your feedback.

The processor is not something like a db trigger, instead it is more like a data processor. One use case for processor is - automatically encrypt some sensitive data before saving, and decrypting the data before reading the data from the database.

[sea-steve]

Thanks for the response. In your change, it looks like you removed the line I mentioned. That should solve my issue. I will try it out.

Unfortunately, a bigger issue I'm facing is the db file size with 4.x. My dataset was less than 100MB under 3.x, but after exporting it and then importing into 4.x, the file is now 7GB! Even after using the compressHigh() option, which btw seemed to have no effect (still 7GB). That's using MVStore. I was hoping to use RocksDb, but that faired even worse: data import was too slow to be usable and often crashed the JVM... Any thoughts on these issues? (Sorry, I know this is short on specifics.)

[anidotnet]

I need to see some reproducible code for this behavior before I comment on anything.

[sea-steve]

Yep, I get it, just thought that since 4.x has been baking for some time already, you might have some ideas about the issues I've mentioned. For now, I'm going to abandon 4.x and switch back to using 3.x. As I have time/interest, I'll experiment further with 4.x and post repro cases if possible. Thanks for your help on the Processor issue.