Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Port 11686 Reduce cases of mass deletion #1363

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

yaelibarg
Copy link
Contributor

@yaelibarg yaelibarg commented Jan 30, 2025

Description

What - Reduce cases of mass deletion from integrations (could be because of bad api's we use or just getting {} from port on the mapping)

Why - we dont want to loose data in port

How - intreduce flag entityDeletionThreshold

Type of change

Please leave one option from the following and delete the rest:

  • Bug fix (non-breaking change which fixes an issue)

All tests should be run against the port production environment(using a testing org).

Core testing checklist

  • Integration able to create all default resources from scratch
  • Resync finishes successfully
  • Resync able to create entities
  • Resync able to update entities
  • Resync able to detect and delete entities
  • Scheduled resync able to abort existing resync and start a new one
  • Tested with at least 2 integrations from scratch
  • Tested with Kafka and Polling event listeners
  • Tested deletion of entities that don't pass the selector

@yaelibarg yaelibarg requested a review from a team as a code owner January 30, 2025 15:01
@yaelibarg yaelibarg changed the title [Core] Port 11686 Reduce cases of mass deletion from integration transfer go [Core] Port 11686 Reduce cases of mass deletion Jan 30, 2025
…es-of-mass-deletion-from-integration-transfer-go
@@ -78,6 +78,7 @@ async def delete_diff(
self,
entities: EntityDiff,
user_agent_type: UserAgentType,
entity_deletion_threshold: float = 0.9,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think default here should be Null and the value will be passed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it will be clearer to communicate it through precentage e.g. 90 rather than 0.9, wdyt?

Comment on lines +93 to 94
keeping_entities=len(kept_entities),
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
keeping_entities=len(kept_entities),
)
keeping_entities=len(kept_entities),
entity_deletion_threshold=entity_deletion_threshold
)

)

await self._safe_delete(diff.deleted, kept_entities, user_agent_type)
delete_rate = len(diff.deleted) / len(entities["before"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets use another parameter name

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants