Evaluate / List all places we use nativeDb or platform and any other 'deprecated' APIs.

[nick4598]

This will have implications when it comes time to consume itwinjs 5.0. NativeDb will no longer be accessible at all due to the private symbol approach. We need public API replacements for everything we currently MUST have from NativeDb in the transformer.

We can consider getting rid of some if we deem them no longer necessary, but we definitely can't get rid of them all. Another alternate approach is to somehow give this repo special privileges, but it seems impossible with the private symbol approach. I think this repo is one of the few that was considered acceptable to use internal APIs? But I may be misremembering that. I think its a moot point since itwinjs-core is moving in the direction of private symbols.

Failing PR run that lists all the ones currently in imodel-transformer
This doesn't mention all the other packages. ctrl + f for IModelHost.platform and nativeDb to find the rest.

relatedPR that disabled all them for now

LIST GOES HERE:

Internal Usage

Preview Give feedback

1. nativeDb.extractChangedInstanceIdsFromChangeSets
2. nativeDb.schemaToXmlString
3. IModelHost.platform.RevisionUtility.recompressRevision
4. IModelHost.platform.RevisionUtility.disassembleRevision
5. IModelHost.platform.RevisionUtility.assembleRevision
6. IModelHost.platform.RevisionUtility.normalizeLzmaParams
7. IModelHost.platform.RevisionUtility.computeStatistics
8. IModelHost.platform.RevisionUtility.getUncompressSize

Need of @preapproved

Preview Give feedback

1. nativeDb.setITwinId
2. nativeDb.deleteAllTxns
3. nativeDb.openIModel
4. nativeDb.resetBriefcaseId
5. nativeDb.saveLocalValue
6. nativeDb.saveChanges
7. nativeDb.closeFile
8. nativeDb.getChangeTrackingMemoryUsed
9. nativeDb.completeCreateChangeset
10. nativeDb.startCreateChangeset
11. nativeDb.applyChangeset
12. nativeDb.startProfiler
13. nativeDb.stopProfiler
14. IModelHost.platform.DgnDb

Public

Preview Give feedback

1. nativeDb.exportSchema
2. nativeDb.inlineGeometryPartReferences
3. nativeDb.simplifyElementGeometry
4. nativeDb.insertElement
5. nativeDb.getLastError

• Changes
  • 10/10/2024
    • Removed nativeDb.extractChangedInstanceIdsFromChangeSets and replaced with ChangedInstanceIds.initialize - replace nativedb extractChangedInstanceIdsFromChangeSets #213
  • 10/14/2024
    • Removed IModelHost.platform.RevisionUtility because they are not being used by any of our tests - Remove revision utility #215
  • 10/15/2024
    • Removed nativeDb.schemaToXmlString and replaced with SchemaTestUtils.schemaToXmlString - Replace nativedb schemaToXmlString  #217
  • 10/16/2024
    • Moved nativeDb.getChangeTrackingMemoryUsed to preapproved tasklist - still useful in test-app
    • Moved nativeDb.completeCreateChangeset to preapproved tasklist - still useful in performance-test
    • Moved nativeDb.startCreateChangeset to preapproved tasklist - still useful in performance-test
    • Moved nativeDb.applyChangeset to preapproved tasklist - still useful in performance-test
    • Moved nativeDb.startProfiler to preapproved tasklist - still useful in performance-script
    • Moved nativeDb.stopProfiler to preapproved tasklist - still useful in performance-script
    • Moved IModelHost.platform.DgnDb to preapproved tasklist - still useful in performance-test, test-app, other tests

[derbynn]

MUST HAVE REPLACEMENT:

• exportSchema
  • Links to usage:
    • https://github.com/iTwin/imodel-transformer/blob/main/packages/transformer/src/IModelTransformer.ts#L2749

    • imodel-transformer/packages/performance-tests/test/cases/prepare-fork.ts

      Line 58 in 7131a96

      sourceDb.nativeDb.exportSchemas(schemaDumpDir);

    • imodel-transformer/packages/performance-tests/test/cases/identity-transformer.ts

      Line 53 in 7131a96

      sourceDb.nativeDb.exportSchemas(schemaDumpDir);

• inlineGeometryPartReferences *
  • Links to usage:

    • imodel-transformer/packages/transformer/src/IModelImporter.ts

      Line 764 in 7131a96

      const result = this.targetDb.nativeDb.inlineGeometryPartReferences();

• simplifyElementGeometry *
  • Not found in itwinjs-core
  • Links to usage:

    • imodel-transformer/packages/transformer/src/IModelImporter.ts

      Line 308 in 7131a96

      this.targetDb.nativeDb.simplifyElementGeometry({

    • imodel-transformer/packages/transformer/src/IModelImporter.ts

      Line 346 in 7131a96

      this.targetDb.nativeDb.simplifyElementGeometry({

• insertElement * (not sure if we use nativeDb's insertElement on purpose. forceUseId: this.options.preserveELementIdsForFiltering seems important and only available on nativeDb.)
  • Links to usage:
    • https://github.com/iTwin/imodel-transformer/blob/main/packages/transformer/src/IModelImporter.ts#L297
• getLastError * (branchprovenanceinitializer which uses attach database, so maybe it needs getLastError)
  • Links to usage:

    • imodel-transformer/packages/transformer/src/BranchProvenanceInitializer.ts

      Line 77 in 7131a96

      args.branch.nativeDb.getLastError()

    • imodel-transformer/packages/transformer/src/BranchProvenanceInitializer.ts

      Line 89 in 7131a96

      args.branch.nativeDb.getLastError()

    • imodel-transformer/packages/transformer/src/BranchProvenanceInitializer.ts

      Line 115 in 7131a96

      `Error detaching db (we will close anyway): ${args.branch.nativeDb.getLastError()}`

    • imodel-transformer/packages/transformer/src/BranchProvenanceInitializer.ts

      Line 121 in 7131a96

      args.branch.nativeDb.getLastError()

• setToStandalone seems important for tests. Standalone iModels are read/write files that are not associated with an iTwin or managed by iModelHub
• setITwinId (setToStandalone)
  • Found in IModelNative.platform.DgnDb() which is internal and will soon be inaccessible due to private symbol approach
  • Links to usage:

    • imodel-transformer/packages/performance-tests/test/iModelUtils.ts

      Line 37 in 7131a96

      nativeDb.setITwinId(Guid.empty); // empty iTwinId means "standalone"

    • imodel-transformer/packages/test-app/src/Main.ts

      Line 503 in 7131a96

      nativeDb.setITwinId(Guid.empty); // empty iTwinId means "standalone"

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformerHub.test.ts

      Line 900 in 7131a96

      masterSeedDb.nativeDb.setITwinId(iTwinId); // workaround for "ContextId was not properly setup in the checkpoint" issue

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformerHub.test.ts

      Line 996 in 7131a96

      masterSeedDb.nativeDb.setITwinId(iTwinId); // workaround for "ContextId was not properly setup in the checkpoint" issue

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformerHub.test.ts

      Line 1113 in 7131a96

      masterSeedDb.nativeDb.setITwinId(iTwinId); // workaround for "ContextId was not properly setup in the checkpoint" issue

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformerHub.test.ts

      Line 1265 in 7131a96

      masterSeedDb.nativeDb.setITwinId(iTwinId); // workaround for "ContextId was not properly setup in the checkpoint" issue

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformerHub.test.ts

      Line 2112 in 7131a96

      masterSeedDb.nativeDb.setITwinId(iTwinId); // workaround for "ContextId was not properly setup in the checkpoint" issue

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformerHub.test.ts

      Line 3382 in 7131a96

      masterSeedDb.nativeDb.setITwinId(iTwinId); // workaround for "ContextId was not properly setup in the checkpoint" issue

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformer.test.ts

      Line 2169 in 7131a96

      nativeDb.setITwinId(Guid.empty); // empty iTwinId means "standalone"

    • imodel-transformer/packages/transformer/src/test/standalone/BranchProvenanceInitializer.test.ts

      Line 428 in 7131a96

      nativeDb.setITwinId(Guid.empty); // empty iTwinId means "standalone"

• deleteAllTxns (setToStandalone)
  • Found in IModelNative.platform.DgnDb() which is internal and will soon be inaccessible due to private symbol approach
  • Links to usage:

    • imodel-transformer/packages/performance-tests/test/iModelUtils.ts

      Line 39 in 7131a96

      nativeDb.deleteAllTxns(); // necessary before resetting briefcaseId

    • imodel-transformer/packages/test-app/src/Main.ts

      Line 505 in 7131a96

      nativeDb.deleteAllTxns(); // necessary before resetting briefcaseId

    • imodel-transformer/packages/transformer/src/test/TestUtils/IModelTestUtils.ts

      Line 898 in 7131a96

      iModelDb.nativeDb.deleteAllTxns();

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformer.test.ts

      Line 2171 in 7131a96

      nativeDb.deleteAllTxns(); // necessary before resetting briefcaseId

    • imodel-transformer/packages/transformer/src/test/standalone/BranchProvenanceInitializer.test.ts

      Line 430 in 7131a96

      nativeDb.deleteAllTxns(); // necessary before resetting briefcaseId

• openIModel (setToStandalone)
  • Found in IModelNative.platform.DgnDb() which is internal and will soon be inaccessible due to private symbol approach
  • Links to usage:

    • imodel-transformer/packages/performance-tests/test/iModelUtils.ts

      Line 36 in 7131a96

      nativeDb.openIModel(iModelPath, OpenMode.ReadWrite);

    • imodel-transformer/packages/test-app/src/Main.ts

      Line 502 in 7131a96

      nativeDb.openIModel(iModelPath, OpenMode.ReadWrite);

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformer.test.ts

      Line 2168 in 7131a96

      nativeDb.openIModel(iModelName, OpenMode.ReadWrite);

    • imodel-transformer/packages/transformer/src/test/standalone/BranchProvenanceInitializer.test.ts

      Line 427 in 7131a96

      nativeDb.openIModel(iModelName, OpenMode.ReadWrite);

• resetBriefcaseId (setToStandalone)
  • Found in IModelNative.platform.DgnDb() which is internal and will soon be inaccessible due to private symbol
  • Links to usage:

    • imodel-transformer/packages/performance-tests/test/iModelUtils.ts

      Line 40 in 7131a96

      nativeDb.resetBriefcaseId(BriefcaseIdValue.Unassigned); // standalone iModels should always have BriefcaseId unassigned

    • imodel-transformer/packages/test-app/src/Main.ts

      Line 506 in 7131a96

      nativeDb.resetBriefcaseId(BriefcaseIdValue.Unassigned); // standalone iModels should always have BriefcaseId unassigned

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformer.test.ts

      Line 2172 in 7131a96

      nativeDb.resetBriefcaseId(BriefcaseIdValue.Unassigned); // standalone iModels should always have BriefcaseId unassigned

    • imodel-transformer/packages/transformer/src/test/standalone/BranchProvenanceInitializer.test.ts

      Line 431 in 7131a96

      nativeDb.resetBriefcaseId(BriefcaseIdValue.Unassigned); // standalone iModels should always have BriefcaseId unassigned

• saveLocalValue (setToStandalone)
  • Found in IModelNative.platform.DgnDb() which is internal and will soon be inaccessible due to private symbol
  • Links to usage:

    • imodel-transformer/packages/performance-tests/test/iModelUtils.ts

      Line 41 in 7131a96

      nativeDb.saveLocalValue("StandaloneEdit", JSON.stringify({ txns: true }));

    • imodel-transformer/packages/test-app/src/Main.ts

      Line 507 in 7131a96

      nativeDb.saveLocalValue(

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformer.test.ts

      Line 2173 in 7131a96

      nativeDb.saveLocalValue("StandaloneEdit", JSON.stringify({ txns: true }));

    • imodel-transformer/packages/transformer/src/test/standalone/BranchProvenanceInitializer.test.ts

      Line 432 in 7131a96

      nativeDb.saveLocalValue("StandaloneEdit", JSON.stringify({ txns: true }));

• saveChanges (setToStandalone)
• closeFile (setToStandalone)
  • Found in IModelNative.platform.DgnDb() which is internal and will soon be inaccessible due to private symbol
  • Links to usage:

    • imodel-transformer/packages/performance-tests/test/iModelUtils.ts

      Line 43 in 7131a96

      nativeDb.closeFile();

    • imodel-transformer/packages/test-app/src/Main.ts

      Line 512 in 7131a96

      nativeDb.closeFile();

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformer.test.ts

      Line 2175 in 7131a96

      nativeDb.closeFile();

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformer.test.ts

      Line 2207 in 7131a96

      nativeDb.closeFile();

    • imodel-transformer/packages/transformer/src/test/standalone/BranchProvenanceInitializer.test.ts

      Line 434 in 7131a96

      nativeDb.closeFile();

MIGHT NEED REPLACEMENT:

• getChangeTrackingMemoryUsed (test-app)
  • Not found in itwinjs-core
  • Link to usage:

    • imodel-transformer/packages/test-app/src/Transformer.ts

      Line 428 in 7131a96

      const bytesUsed = this.targetDb.nativeDb.getChangeTrackingMemoryUsed(); // can't call this internal method unless targetDb has change tracking enabled

• completeCreateChangeset (performance-tests)
  • Found in IModelNative.platform.DgnDb() which is internal and will soon be inaccessible due to private symbol
  • Link to usage:

    • imodel-transformer/packages/performance-tests/test/rawInserts.ts

      Line 199 in 7131a96

      imodel.nativeDb.completeCreateChangeset({ index: 0 });

• startCreateChangeset (performance-tests)
  • Found in IModelNative.platform.DgnDb() which is internal and will soon be inaccessible due to private symbol
  • Link to usage:

    • imodel-transformer/packages/performance-tests/test/rawInserts.ts

      Line 190 in 7131a96

      const changeset = imodel.nativeDb.startCreateChangeset();

• applyChangeset (performance-tests)
  • Not found in itwinjs-core
  • Link to usage:

    • imodel-transformer/packages/performance-tests/test/rawInserts.ts

      Line 87 in 7131a96

      changesetDb.nativeDb.applyChangeset(changeset1);

• startProfiler * (performance-scripts, runWithSqliteProfiler)
  • Not found in itwinjs-core
  • Link to usage:

    • imodel-transformer/packages/performance-scripts/runWithSqliteProfiler.ts

      Line 25 in 7131a96

      const db = argOffset ? args[argOffset] : args.find((a): a is IModelDb => typeof a?.nativeDb?.startProfiler === "function");

    • imodel-transformer/packages/performance-scripts/runWithSqliteProfiler.ts

      Line 28 in 7131a96

      db.nativeDb.startProfiler("transformer", "processChanges", true, true);

• stopProfiler * (performance-scripts, runWithSqliteProfiler)
  • Not found in itwinjs-core
  • Link to usage:

    • imodel-transformer/packages/performance-scripts/runWithSqliteProfiler.ts

      Line 31 in 7131a96

      const profileResult = db.nativeDb.stopProfiler();

MIGHT BE ABLE TO REMOVE:

• extractChangedInstanceIdsFromChangeSets (We could probably remove usage on this, use our own ChangedInstanceIds class)
  • Not found in itwinjs-core
  • Links to usage:

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformerHub.test.ts

      Line 1559 in 7131a96

      master.db.nativeDb.extractChangedInstanceIdsFromChangeSets([

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformerHub.test.ts

      Line 1638 in 7131a96

      replayedDb.nativeDb.extractChangedInstanceIdsFromChangeSets([

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformerHub.test.ts

      Line 1772 in 7131a96

      sourceDb.nativeDb.extractChangedInstanceIdsFromChangeSets([

    • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformerHub.test.ts

      Line 2573 in 7131a96

      branchDb.nativeDb.extractChangedInstanceIdsFromChangeSets([

Yet to be sorted:

• nativeDb Functions

  • hasPendingTxns

    • Public

  • hasUnsavedChanges

    • Public

  • schemaToXmlString (there could be a workaround here... not sure. the one in core uses DOMParser)

    • Not found. Only found example function in itwinjs-core/example-code/snippets/src/pcf/DynamicSchema.ts
    • Links to usage:

      • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformer.test.ts

        Line 3584 in 7131a96

        const biscoreText = sourceDb.nativeDb.schemaToXmlString("BisCore");

• IModelHost.platform

  • DgnDb

    • moved to iModelNative which is internal and will soon be inaccessible due to private symbol approach
    • Links to usage:

      • imodel-transformer/packages/performance-tests/test/iModelUtils.ts

        Line 35 in 7131a96

        const nativeDb = new IModelHost.platform.DgnDb();

      • imodel-transformer/packages/test-app/src/Main.ts

        Line 501 in 7131a96

        const nativeDb = new IModelHost.platform.DgnDb();

      • imodel-transformer/packages/transformer/src/test/standalone/IModelTransformer.test.ts

        Line 2202 in 7131a96

        const nativeDb = StandaloneDb.openDgnDb(

      • imodel-transformer/packages/transformer/src/test/standalone/BranchProvenanceInitializer.test.ts

        Line 426 in 7131a96

        const nativeDb = new IModelHost.platform.DgnDb();

  • RevisionUtility.recompressRevision

    • moved to iModelNative which is internal and will soon be inaccessible due to private symbol approach
    • Link to usage:

      • imodel-transformer/packages/transformer/src/test/TestUtils/RevisionUtility.ts

        Line 60 in 7131a96

        return IModelHost.platform.RevisionUtility.recompressRevision(

  • RevisionUtility.disassembleRevision

    • moved to iModelNative which is internal and will soon be inaccessible due to private symbol approach
    • Link to usage:

      • imodel-transformer/packages/transformer/src/test/TestUtils/RevisionUtility.ts

        Line 72 in 7131a96

        return IModelHost.platform.RevisionUtility.disassembleRevision(

  • RevisionUtility.assembleRevision

    • moved to iModelNative which is internal and will soon be inaccessible due to private symbol approach
    • Link to usage:

      • imodel-transformer/packages/transformer/src/test/TestUtils/RevisionUtility.ts

        Line 87 in 7131a96

        return IModelHost.platform.RevisionUtility.assembleRevision(

  • RevisionUtility.normalizeLzmaParams

    • moved to iModelNative which is internal and will soon be inaccessible due to private symbol approach
    • Link to usage:

      • imodel-transformer/packages/transformer/src/test/TestUtils/RevisionUtility.ts

        Line 96 in 7131a96

        IModelHost.platform.RevisionUtility.normalizeLzmaParams(

  • RevisionUtility.computeStatistics

    • moved to iModelNative which is internal and will soon be inaccessible due to private symbol approach
    • Link to usage:

      • imodel-transformer/packages/transformer/src/test/TestUtils/RevisionUtility.ts

        Line 108 in 7131a96

        IModelHost.platform.RevisionUtility.computeStatistics(

  • RevisionUtility.getUncompressSize

    • moved to iModelNative which is internal and will soon be inaccessible due to private symbol approach
    • Link to usage:

      • imodel-transformer/packages/transformer/src/test/TestUtils/RevisionUtility.ts

        Line 118 in 7131a96

        IModelHost.platform.RevisionUtility.getUncompressSize(sourceFile)

*Denotes that the function is not being used exclusively by tests

[aruniverse]

fyi @pmconne @wgoehrig to keep you guys in the loop

[pmconne]

Would it be possible to provide a list of the APIs you are using that currently do not have an alternative public API, with link(s) to your usage of each?

[nick4598]

Thanks for the suggestion, we will work on gathering that information

[nick4598]

Daniel has updated the list with links to their usage. I also tried to give a rough categorization of each one, "must have replacement", "might need replacement".