Index breaks after upgrade migration

[ishvo]

OrientDB Version: 2.2.37 -> 3.2.36

Java Version: openjdk version "1.8.0_412"

OS: Macbook pro M3 using docker: linux/arm64/v8

Expected behavior

Rebuilding an index does not break the database

Actual behavior

When causing a rebuild after the import it break the database, making it impossible to log in over console or over the java library.

Steps to reproduce

Export a database which have a class with an index created using: "CREATE INDEX Idx ON Example (id) NOTUNIQUE METADATA {ignoreNullValues: true}" on orientdb 2.2.37 (ran directly on the host) and export using the console.
Import into orientdb3.2.36 (dockerized).
Log into the web interface and rebuild the index.
I could not fix the broken database.

Edit: I did not encounter this issue on "UNIQUE" indices.

Workaround

After importing the database, drop the index and create it again.
The old index had "SBTREE" engine and the new one has "CELL_BTREE" to which I can't seem to find documentation.
This is only possible if the database is not broken yet.

[tglman]

Hi,

Thanks to report this, we will double check, do you have by any chances logs/stack traces related to this problem, it may speed up the resolution.

Regards

[ishvo]

Yes.

This is a breaking NOTUNIQUE index rebuild (it loops somewhere so I cut it at some point):

2024-12-13 10:08:22 2024-12-13 09:08:22:059 SEVER Exception `15B56D77` in storage `plocal:/orientdb/databases/db`: 3.2.36 (build ${buildNumber}, branch UNKNOWN) [OLocalPaginatedStorage]
2024-12-13 10:08:22 com.orientechnologies.orient.core.storage.index.sbtree.local.v1.OSBTreeException: Null keys are not supported.      DB name="db"     Component Name="IdxContractClaimed"
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.index.sbtree.local.v1.OSBTreeV1.checkNullSupport(OSBTreeV1.java:810)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.index.sbtree.local.v1.OSBTreeV1.get(OSBTreeV1.java:189)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.index.engine.OSBTreeIndexEngine.get(OSBTreeIndexEngine.java:202)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doGetIndexValue(OAbstractPaginatedStorage.java:3079)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.getIndexValue(OAbstractPaginatedStorage.java:3054)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.index.OIndexMultiValues.getRidsIgnoreTx(OIndexMultiValues.java:91)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.index.OIndexMultiValues.getRids(OIndexMultiValues.java:120)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.index.OIndexAbstract.doDelete(OIndexAbstract.java:661)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:473)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:338)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.sql.parser.ORebuildIndexStatement.executeSimple(ORebuildIndexStatement.java:51)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.sql.executor.OSingleOpExecutionPlan.executeInternal(OSingleOpExecutionPlan.java:63)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.sql.parser.OSimpleExecStatement.execute(OSimpleExecStatement.java:45)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.sql.parser.OStatement.execute(OStatement.java:76)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.command(ODatabaseDocumentEmbedded.java:634)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommand.executeStatement(OServerCommandPostCommand.java:221)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommand.execute(OServerCommandPostCommand.java:104)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommandGraph.execute(OServerCommandPostCommandGraph.java:34)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:242)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:800)
2024-12-13 10:08:22     at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:68)
2024-12-13 10:08:22 
2024-12-13 10:08:22 2024-12-13 09:08:22:059 SEVER {db=db} Error Dropping Index IdxContractClaimed [OIndexNotUnique]
2024-12-13 10:08:22 com.orientechnologies.orient.core.storage.index.sbtree.local.v1.OSBTreeException: Null keys are not supported.      DB name="db"     Component Name="IdxContractClaimed"
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.index.sbtree.local.v1.OSBTreeV1.checkNullSupport(OSBTreeV1.java:810)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.index.sbtree.local.v1.OSBTreeV1.get(OSBTreeV1.java:189)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.index.engine.OSBTreeIndexEngine.get(OSBTreeIndexEngine.java:202)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doGetIndexValue(OAbstractPaginatedStorage.java:3079)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.getIndexValue(OAbstractPaginatedStorage.java:3054)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.index.OIndexMultiValues.getRidsIgnoreTx(OIndexMultiValues.java:91)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.index.OIndexMultiValues.getRids(OIndexMultiValues.java:120)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.index.OIndexAbstract.doDelete(OIndexAbstract.java:661)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:473)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:338)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.sql.parser.ORebuildIndexStatement.executeSimple(ORebuildIndexStatement.java:51)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.sql.executor.OSingleOpExecutionPlan.executeInternal(OSingleOpExecutionPlan.java:63)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.sql.parser.OSimpleExecStatement.execute(OSimpleExecStatement.java:45)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.sql.parser.OStatement.execute(OStatement.java:76)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.command(ODatabaseDocumentEmbedded.java:634)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommand.executeStatement(OServerCommandPostCommand.java:221)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommand.execute(OServerCommandPostCommand.java:104)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommandGraph.execute(OServerCommandPostCommandGraph.java:34)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:242)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:800)
2024-12-13 10:08:22     at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:68)
2024-12-13 10:08:22 
2024-12-13 10:08:22 2024-12-13 09:08:22:068 SEVER Exception `6336CD9A` in storage `plocal:/orientdb/databases/db`: 3.2.36 (build ${buildNumber}, branch UNKNOWN) [OLocalPaginatedStorage]
2024-12-13 10:08:22 com.orientechnologies.orient.core.exception.OStorageException: Exception during execution of atomic operation inside of storage db   DB name="db"
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.calculateInsideAtomicOperation(OAtomicOperationsManager.java:128)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.addIndexEngine(OAbstractPaginatedStorage.java:2724)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:487)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:338)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.sql.parser.ORebuildIndexStatement.executeSimple(ORebuildIndexStatement.java:51)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.sql.executor.OSingleOpExecutionPlan.executeInternal(OSingleOpExecutionPlan.java:63)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.sql.parser.OSimpleExecStatement.execute(OSimpleExecStatement.java:45)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.sql.parser.OStatement.execute(OStatement.java:76)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.command(ODatabaseDocumentEmbedded.java:634)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommand.executeStatement(OServerCommandPostCommand.java:221)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommand.execute(OServerCommandPostCommand.java:104)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommandGraph.execute(OServerCommandPostCommandGraph.java:34)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:242)
2024-12-13 10:08:22     at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:800)
2024-12-13 10:08:22     at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:68)
2024-12-13 10:08:22 Caused by: com.orientechnologies.orient.core.exception.OStorageException: Exception during execution of component operation inside of storage db     DB name="db"
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.calculateInsideComponentOperation(OAtomicOperationsManager.java:226)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.calculateInsideComponentOperation(OAtomicOperationsManager.java:213)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.calculateInsideComponentOperation(ODurableComponent.java:96)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.index.sbtreebonsai.local.OSBTreeBonsaiLocal.createComponent(OSBTreeBonsaiLocal.java:88)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.lambda$addIndexEngine$13(OAbstractPaginatedStorage.java:2781)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.calculateInsideAtomicOperation(OAtomicOperationsManager.java:122)
2024-12-13 10:08:22     ... 14 more
2024-12-13 10:08:22 Caused by: com.orientechnologies.orient.core.exception.OStorageException: Ridbag component with name IdxContractClaimed.irs already exists    DB name="db"
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.index.sbtreebonsai.local.OSBTreeBonsaiLocal.lambda$createComponent$0(OSBTreeBonsaiLocal.java:93)
2024-12-13 10:08:22     at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.calculateInsideComponentOperation(OAtomicOperationsManager.java:221)
2024-12-13 10:08:22     ... 19 more

[tglman]

Hi,

I think this is the case because of changed behavior, so now if you set "ignoreNullValues=true" if you save a record with a null value it does give an error, this was not the case before, I'm double checking this logic and when it was changed, because ignoring should not mean failing, but I guess the old storage maintainer didn't think so, will update soon on this.

[tglman]

Hi,

I actually found that there where two checks one high level to avoid to send null to low level, and one low level that create this issue, in the case of rebuild index the first high level check was missing, so I did add it, this should be fixed from the next hotfix 3.2.38.

Regards