feat(triage signals): Fixability based stopping point for autofix #103076

Mihir-Mavalankar · 2025-11-10T18:30:48Z

PR Details

Changing the logic to set the default stopping point of Autofix based on fixability score. The UI stopping point will still override this until the Settings UI is changed.
Using this feature flag to just try out in the Seer org: https://github.com/getsentry/sentry-options-automator/blob/main/options/default/flagpole.yaml#L3207
More details in triage signals implementation document: https://www.notion.so/sentry/Triage-Signals-V0-Technical-Implementation-Details-2a18b10e4b5d8086a7ceddaf4194849a?source=copy_link#2a48b10e4b5d8064a52ac6b519630fc0

codecov · 2025-11-10T18:49:00Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
29563	2	29561	244

View the top 2 failed test(s) by shortest run time

tests.sentry.seer.autofix.test_issue_summary.TestRunAutomationStoppingPoint::test_with_feature_flag

Stack Traces | 4.31s run time

#x1B[1m#x1B[.../testutils/helpers/features.py#x1B[0m:215: in wrapped_method
    return method(*args, **kwargs)
#x1B[1m#x1B[.../hostedtoolcache/Python/3.13.1.../x64/lib/python3.13/unittest/mock.py#x1B[0m:1424: in patched
    return func(*newargs, **newkeywargs)
#x1B[1m#x1B[31mE   TypeError: TestRunAutomationStoppingPoint.test_with_feature_flag() missing 2 required positional arguments: 'score' and 'expected'#x1B[0m

tests.sentry.seer.autofix.test_issue_summary.TestRunAutomationStoppingPoint::test_without_feature_flag

Stack Traces | 4.59s run time

#x1B[1m#x1B[.../seer/autofix/test_issue_summary.py#x1B[0m:766: in test_without_feature_flag
    assert mock_trigger.call_args[1]["stopping_point"] is None
#x1B[1m#x1B[31mE   TypeError: 'NoneType' object is not subscriptable#x1B[0m

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

JoshFerge · 2025-11-10T18:53:18Z

src/sentry/seer/autofix/issue_summary.py

+    event_id: str,
+    user_id: int | None,
+    auto_run_source: str,
+    stopping_point: AutofixStoppingPoint | None = None,


we're changing the signature of the task which could cause issues as old/new versions exist at the same time, but since we're adding a default to the parameter, this is okay. (we may want to add **kwargs to the signature for future in case a parameter gets removed)

Yeah I thought a default should be okay. I think I will keep as is for now because this will become a required parameter once the the fallback stuff is removed.

JoshFerge

how does this change interact with what values people have configured?

tests/sentry/seer/autofix/test_issue_summary.py

src/sentry/seer/autofix/issue_summary.py

Mihir-Mavalankar · 2025-11-10T19:10:29Z

how does this change interact with what values people have configured?

Currently it does not. My next PR will have the UI change the completely removes that settings and turns automation into a on and off toggle.

src/sentry/seer/autofix/issue_summary.py

kddubey · 2025-11-10T19:28:12Z

src/sentry/seer/autofix/issue_summary.py

        return

+    stopping_point = None
+    if features.has("organizations:triage-signals-v0", group.organization):


hmm does this flag need to be a ProjectFeature, not an OrganizationFeature? then pass group.project instead

looks like this is tested, but i'm not familiar w/ how it's mocked under the hood

why does it need to be a project feature?

Hmm it not sure either. This said organization here so I went with organizationFeature.

also, is this feature registered? was that done in a separate PR?

Or since this only applies to the Seer project should I change the feature in flagpole to be a project feature?

the flag checks project_id

what do you mean by this?

the feature flag's filter is on project_id

https://github.com/getsentry/sentry-options-automator/blob/edc0dd6fe25c4365104ea7471b65a9c5e695430a/options/default/flagpole.yaml#L3213-L3217

oh... wait can an organization feature even filter on project_id?

https://develop.sentry.dev/backend/application-domains/feature-flags/flagpole/

doesn't seem like it. @Mihir-Mavalankar can you double check these docs and make sure our flag usage is correct?

Yeah that's right. I have changed it to a project feature now. Options PR is merged and manager is updated in this PR.

tests/sentry/seer/autofix/test_issue_summary.py

not sure about feature flag stuff

src/sentry/seer/autofix/issue_summary.py

tests/sentry/seer/autofix/test_issue_summary.py

sentry · 2025-11-10T23:11:49Z

Issues attributed to commits in this pull request

This pull request was merged and Sentry observed the following issues:

‼️ RateLimitExceeded: Query on could not be run due t... in prod

Mihir-Mavalankar added 2 commits November 10, 2025 10:25

feat(triage signals): Fixability based stopping point for autofix

6b6c285

feat(triage signals): Fixability based stopping point for autofix

c46691c

Mihir-Mavalankar self-assigned this Nov 10, 2025

Mihir-Mavalankar requested a review from a team as a code owner November 10, 2025 18:30

github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Nov 10, 2025

Mihir-Mavalankar requested a review from kddubey November 10, 2025 18:45

fix tests

f963b1b

JoshFerge reviewed Nov 10, 2025

View reviewed changes

vercel bot deployed to Preview November 10, 2025 18:56 View deployment

JoshFerge reviewed Nov 10, 2025

View reviewed changes

cursor bot reviewed Nov 10, 2025

View reviewed changes

tests/sentry/seer/autofix/test_issue_summary.py Outdated Show resolved Hide resolved

tests/sentry/seer/autofix/test_issue_summary.py Outdated Show resolved Hide resolved

JoshFerge reviewed Nov 10, 2025

View reviewed changes

src/sentry/seer/autofix/issue_summary.py Show resolved Hide resolved

fix tests

c0a7acc

vercel bot deployed to Preview November 10, 2025 19:03 View deployment

JoshFerge previously approved these changes Nov 10, 2025

View reviewed changes

kddubey reviewed Nov 10, 2025

View reviewed changes

JoshFerge reviewed Nov 10, 2025

View reviewed changes

tests/sentry/seer/autofix/test_issue_summary.py Outdated Show resolved Hide resolved

test import

8138a10

vercel bot deployed to Preview November 10, 2025 19:39 View deployment

Mihir-Mavalankar added 2 commits November 10, 2025 13:09

chnaged to project feature

1342ac2

stop at code changes instead of open PR

7a8247c

cursor bot reviewed Nov 10, 2025

View reviewed changes

src/sentry/seer/autofix/issue_summary.py Show resolved Hide resolved

tests/sentry/seer/autofix/test_issue_summary.py Outdated Show resolved Hide resolved

stop at code changes instead of open PR

a26a38d

vercel bot deployed to Preview November 10, 2025 21:13 View deployment

vercel bot deployed to Preview November 10, 2025 21:16 View deployment

Mihir-Mavalankar requested a review from kddubey November 10, 2025 21:16

kddubey approved these changes Nov 10, 2025

View reviewed changes

Mihir-Mavalankar enabled auto-merge (squash) November 10, 2025 21:25

Mihir-Mavalankar merged commit fa3d768 into master Nov 10, 2025
65 checks passed

Mihir-Mavalankar deleted the fixability-stop-for-autofix branch November 10, 2025 21:34

Uh oh!

feat(triage signals): Fixability based stopping point for autofix #103076

feat(triage signals): Fixability based stopping point for autofix #103076

Conversation

Mihir-Mavalankar commented Nov 10, 2025

PR Details

Uh oh!

codecov bot commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

JoshFerge Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoshFerge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mihir-Mavalankar commented Nov 10, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sentry bot commented Nov 10, 2025

Issues attributed to commits in this pull request

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Nov 10, 2025 •

edited

Loading

JoshFerge Nov 10, 2025 •

edited

Loading