New HLT tracking baseline as default by VourMa · Pull Request #231 · SegmentLinking/cmssw

VourMa · 2026-02-01T23:04:13Z

As per title.

Validations:

Legacy tracking (using the hltPhase2LegacyTracking procModifier, equivalent to the previous legacy) = two iterations, legacy quad + triplet seeding, CKF building:
https://uaf-10.t2.ucsd.edu/~evourlio/SDL/CMSSWPR231/oldBaseline_legacyPixels_noCAExt_noSingleIter_noMkFit_noLST/plots_hlt.html
Legacy tracking with Patatrack quads (using the hltPhase2LegacyTrackingPatatrackQuads procModifier, equivalent to the current default) = two iterations, legacy quad + triplet seeding, CKF building:
https://uaf-10.t2.ucsd.edu/~evourlio/SDL/CMSSWPR231/oldBaseline_noCAExt_noSingleIter_noMkFit_noLST/plots_hlt.html
New tracking baseline (no procModifiers needed after this PR) = single iteration, CA extension + LST seeding, mkFit building:
https://uaf-10.t2.ucsd.edu/~evourlio/SDL/CMSSWPR231/newBaseline_noCAExt_noSingleIter_noMkFit_noLST/plots_hlt.html
trackingLST = single iteration, CA extension, LST building:
https://uaf-10.t2.ucsd.edu/~evourlio/SDL/CMSSWPR231/trackingLST_noCAExt_noSingleIter_noMkFit_noLST/plots_hlt.html
trackingMkFitFit = single iteration, CA extension + LST seeding, mkFit building, mkFit fitting:
https://uaf-10.t2.ucsd.edu/~evourlio/SDL/CMSSWPR231/trackingMkFitFit_noCAExt_noSingleIter_noMkFit_noLST/plots_hlt.html
ngtScouting:
https://uaf-10.t2.ucsd.edu/~evourlio/SDL/CMSSWPR231/ngtScouting_noCAExt_noSingleIter_noMkFit_noLST/plots_hlt.html
ngtScouting,trackingLST:
https://uaf-10.t2.ucsd.edu/~evourlio/SDL/CMSSWPR231/ngtScouting_trackingLST_noCAExt_noSingleIter_noMkFit_noLST/plots_hlt.html

Small cleanup done:

Port here the developments from Remove alpaka procModfier from workflows in which it is no longer useful #220, because it conflicts with this PR and PdmV L2s have not bother checking that one on the cms-sw side:
- Delete alpaka procModifier where it is not needed.
- Add README entry for TICL v5 GNN workflow.
Add legacyTracking wf
Remove the hltTrackingMkFitInitialStep procModifier

VourMa · 2026-02-02T11:38:54Z

run-ci: all

github-actions · 2026-02-02T11:58:50Z

The PR was built and ran successfully in standalone mode. Here are some of the comparison plots.

The full set of validation and comparison plots can be found here.

Here is a timing comparison:

   Evt    Hits       MD       LS      T3       T5       pLS       pT5      pT3      TC       Reset    Event     Short             Rate
   avg     30.5    377.8    271.7    125.3     51.2    686.1     10.9    125.3    134.8    187.9      1.8    2003.3    1286.6+/- 301.7     613.5   explicit[s=4] (target branch)
   avg     30.7    376.3    272.4    123.9     50.9    693.7     10.9    125.8    132.1    186.5      1.8    2004.9    1280.5+/- 303.6     620.1   explicit[s=4] (this PR)

github-actions · 2026-02-02T12:06:31Z

There was a problem while building and running with CMSSW. The logs can be found here.

VourMa · 2026-02-02T12:18:33Z

There was a problem while building and running with CMSSW. The logs can be found here.

I have trouble understanding where the problem:

File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02927/el8_amd64_gcc13/cms/cmssw/CMSSW_16_1_X_2026-02-01-0000/src/Validation/RecoTrack/python/HLTmultiTrackValidator_cff.py", line 74, in <module>
    from Configuration.ProcessModifiers.singleIterPatatrack_cff import singleIterPatatrack
ModuleNotFoundError: No module named 'Configuration.ProcessModifiers.singleIterPatatrack_cff'

comes from. I can see that the procModifier is there in the master (as it should) and has been deleted in this PR (as it should).

ariostas · 2026-02-02T14:06:43Z

The standalone plots are missing because didn't fully deploy the new CI yet because I'm figuring out what's the issue with standalone GPU. You can still use the old syntax for now.

VourMa · 2026-02-02T17:54:42Z

/run all

VourMa · 2026-02-02T18:11:21Z

Static checks with the "old" CI failed with:

ls: cannot access '/cvmfs/cms-ib.cern.ch/': No such file or directory
Error: Process completed with exit code 2

I would trust the "new" CI static checks. I ran the scram code-checks in the last commit anyways.

github-actions · 2026-02-02T18:19:43Z

The PR was built and ran successfully in standalone mode. Here are some of the comparison plots.

The full set of validation and comparison plots can be found here.

Here is a timing comparison:

   Evt    Hits       MD       LS      T3       T5       pLS       pT5      pT3      TC       Reset    Event     Short             Rate
   avg     30.6    374.1    271.4    125.0     51.3    690.3     10.9    126.0    133.3    188.1      2.3    2003.4    1282.5+/- 302.8     619.9   explicit[s=4] (target branch)
   avg     30.4    374.5    272.4    124.8     51.2    694.9     10.9    126.1    132.2    187.6      1.8    2006.9    1281.6+/- 301.0     615.9   explicit[s=4] (this PR)

github-actions · 2026-02-02T18:23:45Z

There was a problem while building and running with CMSSW. The logs can be found here.

VourMa · 2026-02-02T18:26:56Z

There was a problem while building and running with CMSSW. The logs can be found here.

I have trouble understanding where the problem:
File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02927/el8_amd64_gcc13/cms/cmssw/CMSSW_16_1_X_2026-02-01-0000/src/Validation/RecoTrack/python/HLTmultiTrackValidator_cff.py", line 74, in <module>
    from Configuration.ProcessModifiers.singleIterPatatrack_cff import singleIterPatatrack
ModuleNotFoundError: No module named 'Configuration.ProcessModifiers.singleIterPatatrack_cff'
comes from. I can see that the procModifier is there in the master (as it should) and has been deleted in this PR (as it should).

@ariostas Any idea about this one? Things compiled normally locally, and it seems to me like I don't have an instance of singleIterPatatrack where it says that I have one...

ariostas · 2026-02-02T18:31:30Z

Oh it's because the CI doesn't add the Validation/RecoTrack package. I'll add the functionality to specify custom packages you want to check out.

VourMa · 2026-02-02T18:58:41Z

Oh it's because the CI doesn't add the Validation/RecoTrack package. I'll add the functionality to specify custom packages you want to check out.

Oh, OK. Maybe not needed for now (not sure how complicated its implementation is) but useful for the long run. Thanks!

slava77 · 2026-02-02T19:19:11Z

Oh it's because the CI doesn't add the Validation/RecoTrack package. I'll add the functionality to specify custom packages you want to check out.

Oh, OK. Maybe not needed for now (not sure how complicated its implementation is) but useful for the long run. Thanks!

Does it mean that if a PR changes some code, it will not be checked out by the CI unless that code is in a specific set of packages? Or is it a comment about dependencies?

slava77

I'm not sure CAExtension in hltPhase2PixelTracksCAExtension should remain in the default pre-selection track collection name.

The choice of having or not HighPurity on the pixel track collection is unclear, but perhaps there is a benefit for the downstream users to see it as hltPhase2PixelTracks (similar to hltGeneralTracks, which actually pass HP).

OTOH, perhaps I'm confused on the choice of inputs (like PV reco or scouting output: shouldn't they all consume hltPhase2PixelTracks).

These comments are beyond the LST topic review. So, I'd ask for feedback from the POG.

Configuration/ProcessModifiers/python/phase2LegacyTracking_cff.py

HLTrigger/Configuration/python/HLT_75e33/modules/hltPhase2PixelTracks_cfi.py

HLTrigger/NGTScouting/python/hltTracks_cfi.py

ariostas · 2026-02-02T19:54:26Z

Does it mean that if a PR changes some code, it will not be checked out by the CI unless that code is in a specific set of packages? Or is it a comment about dependencies?

No, it used to just check out a select list of packages. Now it should check out what's needed and you can also do it manually with packages: [some, list]. Let's see if it works.

ariostas · 2026-02-02T19:54:46Z

run-ci: all

ariostas · 2026-02-02T20:12:17Z

GitHub actions is having issues (https://www.githubstatus.com/incidents/xwn6hjps36ty), so I'll test on the self-hosted runner.

ariostas · 2026-02-02T20:12:25Z

run-ci: [standalone, cmssw]
modifiers: gpu
required-prs: 232

github-actions · 2026-02-02T20:16:29Z

There was a problem while building and running in standalone mode. The logs can be found here.

github-actions · 2026-02-02T21:30:23Z

The PR was built and ran successfully with CMSSW. Here are some plots.

OOTB All Tracks

The full set of validation and comparison plots can be found here.

VourMa · 2026-02-03T13:38:39Z

As a heads-up, after discussing with @mmasciov, it was agreed to add also a legacy configuration but with Patatrack quads instead of the legacy ones. I will work on this today and then it should be good to go right after.

ariostas · 2026-02-03T18:50:06Z

run-ci: standalone
modifiers: gpu
required-prs: 232

github-actions · 2026-02-03T19:15:22Z

The PR was built and ran successfully in standalone mode. Here are some of the comparison plots.

The full set of validation and comparison plots can be found here.

Here is a timing comparison:

   Evt    Hits       MD       LS      T3       T5       pLS       pT5      pT3      TC       Reset    Event     Short             Rate
   avg     34.8      0.7      0.6      0.7      1.0      0.4      0.8      0.8      0.5      1.7      0.0      42.0       6.8+/-  3.5     440.1   explicit[s=1]
   avg     35.5      0.4      0.4      0.5      0.9      0.3      0.6      0.7      0.4      1.4      0.0      41.1       5.3+/-  2.5      41.1   explicit[s=1]
   avg      1.3      0.5      0.6      0.7      1.1      0.3      0.9      0.9      0.4      1.9      0.0       8.7       7.0+/-  2.8       4.5   explicit[s=2]
   avg      2.4      0.9      1.0      1.2      1.5      0.4      1.4      1.5      0.6      2.8      0.0      13.7      10.9+/-  3.7       3.5   explicit[s=4]
   avg      3.7      1.3      1.5      1.7      2.1      0.6      2.0      2.0      0.9      3.9      0.0      19.6      15.3+/-  4.8       3.4   explicit[s=6]
   avg      4.7      1.9      2.0      2.5      2.8      0.7      2.5      2.8      1.2      4.7      0.0      25.8      20.3+/-  5.6       3.3   explicit[s=8]
   avg     35.5      0.4      0.4      0.5      0.9      0.3      0.6      0.7      0.4      1.4      0.0      41.1       5.3+/-  2.5      41.1   explicit[s=1]
   avg      1.3      0.5      0.6      0.7      1.1      0.3      0.9      0.9      0.4      1.9      0.0       8.7       7.0+/-  2.8       4.5   explicit[s=2]
   avg      2.4      0.9      1.0      1.2      1.5      0.4      1.4      1.5      0.6      2.8      0.0      13.7      10.9+/-  3.7       3.5   explicit[s=4]
   avg      3.7      1.3      1.5      1.7      2.1      0.6      2.0      2.0      0.9      3.9      0.0      19.6      15.3+/-  4.8       3.4   explicit[s=6]
   avg      4.7      1.9      2.0      2.5      2.8      0.7      2.5      2.8      1.2      4.7      0.0      25.8      20.3+/-  5.6       3.3   explicit[s=8] (target branch)
   avg     35.1      0.4      0.4      0.5      0.9      0.3      0.6      0.7      0.4      1.4      0.0      40.8       5.4+/-  2.6     543.4   explicit[s=1]
   avg     35.4      0.4      0.4      0.5      0.9      0.3      0.6      0.7      0.4      1.4      0.0      41.0       5.3+/-  2.6      41.0   explicit[s=1]
   avg      1.2      0.6      0.6      0.7      1.1      0.3      0.9      0.9      0.5      1.9      0.0       8.6       7.1+/-  2.9       4.4   explicit[s=2]
   avg      1.9      0.9      0.9      1.2      1.6      0.4      1.4      1.4      0.6      2.9      0.0      13.2      10.8+/-  3.8       3.4   explicit[s=4]
   avg      3.0      1.2      1.4      1.8      2.2      0.6      2.0      2.0      0.9      3.7      0.0      18.8      15.2+/-  4.5       3.3   explicit[s=6]
   avg      4.0      2.0      2.3      3.0      3.5      0.8      3.0      2.8      1.5      5.8      0.0      28.8      24.0+/-  6.1       3.7   explicit[s=8]
   avg     35.4      0.4      0.4      0.5      0.9      0.3      0.6      0.7      0.4      1.4      0.0      41.0       5.3+/-  2.6      41.0   explicit[s=1]
   avg      1.2      0.6      0.6      0.7      1.1      0.3      0.9      0.9      0.5      1.9      0.0       8.6       7.1+/-  2.9       4.4   explicit[s=2]
   avg      1.9      0.9      0.9      1.2      1.6      0.4      1.4      1.4      0.6      2.9      0.0      13.2      10.8+/-  3.8       3.4   explicit[s=4]
   avg      3.0      1.2      1.4      1.8      2.2      0.6      2.0      2.0      0.9      3.7      0.0      18.8      15.2+/-  4.5       3.3   explicit[s=6]
   avg      4.0      2.0      2.3      3.0      3.5      0.8      3.0      2.8      1.5      5.8      0.0      28.8      24.0+/-  6.1       3.7   explicit[s=8] (this PR)

ariostas · 2026-02-03T19:19:06Z

Sorry for the CI spam. I'm done now. (I'll fix the repeated timings and test somewhere else)

VourMa · 2026-02-04T00:10:02Z

As a heads-up, after discussing with @mmasciov, it was agreed to add also a legacy configuration but with Patatrack quads instead of the legacy ones. I will work on this today and then it should be good to go right after.

The legacy configuration with Patatrack quads can be enabled by using the procModifier combination hltPhase2LegacyTracking,hltPhase2LegacyTrackingPatatrackQuads. A workflow has been added and the validation can be found in the PR description.

VourMa · 2026-02-04T00:10:31Z

@slava77 @mmasciov The PR is ready to go from my side. Let me know if you have any follow up comments.

HLTrigger/Configuration/python/HLT_75e33/eventsetup/hltESPMkFit_cfi.py

Configuration/ProcessModifiers/python/hltPhase2LegacyTrackingPatatrackQuads_cff.py

VourMa · 2026-02-05T09:00:01Z

Thanks, @slava77, submitting to cms-sw!

VourMa · 2026-02-05T09:56:13Z

Configuration/ProcessModifiers/python/hltPhase2LegacyTracking_cff.py

@@ -0,0 +1,4 @@
+import FWCore.ParameterSet.Config as cms
+
+# This modifiers reverts the Tracking in the HLT Phase-2 to the Legacy algorithms from Run-2


Note to self, if more changes are needed:

Suggested change

# This modifiers reverts the Tracking in the HLT Phase-2 to the Legacy algorithms from Run-2

# This modifier reverts the Tracking in the HLT Phase-2 to the Legacy algorithms from Run-2

VourMa · 2026-02-05T09:56:27Z

Configuration/ProcessModifiers/python/hltPhase2LegacyTrackingPatatrackQuads_cff.py

@@ -0,0 +1,6 @@
+import FWCore.ParameterSet.Config as cms
+
+# This modifiers reverts the Tracking in the HLT Phase-2 to the Legacy algorithms from Run-2


Note to self, if more changes are needed:

Suggested change

# This modifiers reverts the Tracking in the HLT Phase-2 to the Legacy algorithms from Run-2

# This modifier reverts the Tracking in the HLT Phase-2 to the Legacy algorithms from Run-2

VourMa · 2026-02-05T09:57:09Z

HLTrigger/Configuration/python/HLT_75e33/sequences/HLTTrackingSequence_cfi.py

+_HLTTrackingSequenceLegacy = cms.Sequence(HLTItLocalRecoSequence+
                                   HLTOtLocalRecoSequence+
                                   hltTrackerClusterCheck+
                                   HLTPhase2PixelTracksAndVerticesSequence+
                                   HLTInitialStepSequence+
                                   HLTHighPtTripletStepSequence+
                                   hltGeneralTracks)


Note to self, fix indentation, if more changes are needed.

slava77 · 2026-02-11T16:51:40Z

HLTrigger/Configuration/python/HLT_75e33/paths/DST_HeterogeneousReco_cfi.py

+HLTLocalTrackerSequence = cms.Sequence(
+    hltPhase2SiPixelClustersSoA
+    + hltPhase2SiPixelRecHitsSoA
+    + hltSiPhase2Clusters


cross-post from #235 (comment)

it would be more practical to split the offloadable sequences in the tracking sequence definition files and pick them up here. The approach with re-definition doesn't look well maintainable

Do you think that this is an issue with this PR, or it pre-existed in the master:
https://github.com/cms-sw/cmssw/blob/master/HLTrigger/Configuration/python/HLT_75e33/paths/DST_HeterogeneousReco_cfi.py

If the latter is the case (to which I would agree), I would propose to leave it out of this PR and include it in a separate one, after having coordinated with Marco offline. It may require more thinking and redesigning to be accommodated as a side note in another PR, in my opinion.

…tatrack + LST, mkFit) as Phase 2 HLT default Make phase2CAExtension the default for HLT tracking Make singleIterPatatrack the default for HLT tracking Make mkFit the default for HLT tracking Make LST seeding the default for HLT tracking Addition of legacy HLT tracking workflows

Co-authored-by: Marco Musich <marco.musich@cern.ch>

VourMa changed the title ~~New tracking baseline as default~~ New HLT tracking baseline as default Feb 2, 2026

VourMa requested a review from slava77 February 2, 2026 11:38

slava77 approved these changes Feb 2, 2026

View reviewed changes

Configuration/ProcessModifiers/python/phase2LegacyTracking_cff.py Outdated Show resolved Hide resolved

HLTrigger/Configuration/python/HLT_75e33/modules/hltPhase2PixelTracks_cfi.py Show resolved Hide resolved

HLTrigger/NGTScouting/python/hltTracks_cfi.py Show resolved Hide resolved

SegmentLinking deleted a comment from github-actions bot Feb 3, 2026

slava77 reviewed Feb 4, 2026

View reviewed changes

HLTrigger/Configuration/python/HLT_75e33/eventsetup/hltESPMkFit_cfi.py Outdated Show resolved Hide resolved

Configuration/ProcessModifiers/python/hltPhase2LegacyTrackingPatatrackQuads_cff.py Outdated Show resolved Hide resolved

VourMa force-pushed the newTRKAsPhase2HLTDefault branch from 5d07346 to 1cb2ecc Compare February 5, 2026 08:58

VourMa commented Feb 5, 2026

View reviewed changes

VourMa force-pushed the newTRKAsPhase2HLTDefault branch 2 times, most recently from 81b2882 to e72f9c1 Compare February 9, 2026 15:53

VourMa mentioned this pull request Feb 11, 2026

Draft implementation of a CPU vs. GPU workflow and infrastructure for Phase 2 HLT tracking #235

Closed

slava77 reviewed Feb 11, 2026

View reviewed changes

VourMa force-pushed the newTRKAsPhase2HLTDefault branch 4 times, most recently from 92b5204 to 46ec72c Compare February 20, 2026 09:18

VourMa and others added 2 commits February 27, 2026 09:12

Fix vertex validation for extended pixel tracks

247705b

Co-authored-by: Marco Musich <marco.musich@cern.ch>

VourMa force-pushed the newTRKAsPhase2HLTDefault branch from 46ec72c to 247705b Compare February 27, 2026 17:30

github-actions bot merged commit 3e84fb9 into master Mar 4, 2026

VourMa mentioned this pull request Mar 11, 2026

Remove alpaka procModfier from workflows in which it is no longer useful #220

Closed

		@@ -0,0 +1,4 @@
		import FWCore.ParameterSet.Config as cms

		# This modifiers reverts the Tracking in the HLT Phase-2 to the Legacy algorithms from Run-2

	# This modifiers reverts the Tracking in the HLT Phase-2 to the Legacy algorithms from Run-2
	# This modifier reverts the Tracking in the HLT Phase-2 to the Legacy algorithms from Run-2

		@@ -0,0 +1,6 @@
		import FWCore.ParameterSet.Config as cms

		# This modifiers reverts the Tracking in the HLT Phase-2 to the Legacy algorithms from Run-2

Conversation

VourMa commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

VourMa commented Feb 2, 2026

Uh oh!

github-actions bot commented Feb 2, 2026

Uh oh!

github-actions bot commented Feb 2, 2026

Uh oh!

VourMa commented Feb 2, 2026

Uh oh!

ariostas commented Feb 2, 2026

Uh oh!

VourMa commented Feb 2, 2026

Uh oh!

VourMa commented Feb 2, 2026

Uh oh!

github-actions bot commented Feb 2, 2026

Uh oh!

github-actions bot commented Feb 2, 2026

Uh oh!

VourMa commented Feb 2, 2026

Uh oh!

ariostas commented Feb 2, 2026

Uh oh!

VourMa commented Feb 2, 2026

Uh oh!

slava77 commented Feb 2, 2026

Uh oh!

slava77 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ariostas commented Feb 2, 2026

Uh oh!

ariostas commented Feb 2, 2026

Uh oh!

ariostas commented Feb 2, 2026

Uh oh!

ariostas commented Feb 2, 2026

Uh oh!

github-actions bot commented Feb 2, 2026

Uh oh!

github-actions bot commented Feb 2, 2026

Uh oh!

VourMa commented Feb 3, 2026

Uh oh!

ariostas commented Feb 3, 2026

Uh oh!

github-actions bot commented Feb 3, 2026

Uh oh!

ariostas commented Feb 3, 2026

Uh oh!

VourMa commented Feb 4, 2026

Uh oh!

VourMa commented Feb 4, 2026

Uh oh!

Uh oh!

Uh oh!

VourMa commented Feb 5, 2026

Uh oh!

VourMa Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

VourMa Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

VourMa Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

slava77 Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

VourMa Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

VourMa commented Feb 1, 2026 •

edited

Loading

slava77 left a comment •

edited

Loading