Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No longer track partially commited element references #206

Merged
merged 10 commits into from
Oct 1, 2024

Conversation

ViliusRuskys
Copy link
Contributor

Tracking each unresolved reference to be able to resolve it as soon as possible has a big impact on memory consumption when there are large amounts of pending references.
One such case is multiple display styles with hundreds of thousands of excluded elements.

The transformer now only keeps track of element and aspect ids that were only partially completed. These elements and aspects will be reprocessed once all the elements have been processed by the transformer. The only unresolvable references at that point will be the elements that were intentionally filtered out by the transformer or the invalid ones.

@ViliusRuskys ViliusRuskys requested review from a team as code owners September 6, 2024 14:47
@nick4598
Copy link
Collaborator

/azp run iTwin.imodel-transformer docs

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

packages/transformer/src/IModelExporter.ts Show resolved Hide resolved
packages/transformer/src/IModelTransformer.ts Outdated Show resolved Hide resolved
packages/transformer/src/IModelTransformer.ts Outdated Show resolved Hide resolved
packages/transformer/src/IModelTransformer.ts Outdated Show resolved Hide resolved
packages/transformer/src/IModelTransformer.ts Outdated Show resolved Hide resolved
@nick4598
Copy link
Collaborator

What is the thought process behind removing the models and relationships?

@ViliusRuskys
Copy link
Contributor Author

What is the thought process behind removing the models and relationships?

The transformer did not handle those before as well. The collectUnmappedReferences was only called from 3 places: onExportElement, onExportElementUniqueAspect, onExportElementMultiAspects.

References to Models will be resolved the same way as references to Elements, because transformer will check for existence of a Model through its modeled element which will always have the same Id.
Not sure if Models themselves can have any references to Elements other than their modeled element that could be dangling, but the transformer did not do anything about those before this update as well.

As for Relationships, exportRelationships is called after all elements and aspects will already have been exported, so in theory they should never have dangling references. Also, a relationship would have to have a reference to a 3rd element other than its target and source elements for such case to occur.

nick4598
nick4598 previously approved these changes Sep 25, 2024
@ViliusRuskys ViliusRuskys merged commit e6ac567 into main Oct 1, 2024
14 of 16 checks passed
@ViliusRuskys ViliusRuskys deleted the partial-element-fix branch October 1, 2024 13:36
nick4598 added a commit that referenced this pull request Nov 20, 2024
Tracking each unresolved reference to be able to resolve it as soon as
possible has a big impact on memory consumption when there are large
amounts of pending references.
One such case is multiple display styles with hundreds of thousands of
excluded elements.

The transformer now only keeps track of element and aspect ids that were
only partially completed. These elements and aspects will be reprocessed
once all the elements have been processed by the transformer. The only
unresolvable references at that point will be the elements that were
intentionally filtered out by the transformer or the invalid ones.

---------

Co-authored-by: Nick Tessier <22119573+nick4598@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants