Issue 17127365: Watch counts still marginally off on vsphere in 4.20
Description: (_Feel free to update this bug's summary to be more specific._)
 Component Readiness has found a potential regression in the following test:
 
 {code:none}[sig-arch][Late] operators should not create watch channels very often [apigroup:apiserver.openshift.io] [Suite:openshift/conformance/parallel]{code}
 
 Significant regression detected.
 Fishers Exact probability of a regression: 99.94%.
 Test pass rate dropped from 100.00% to 91.43%.
 
 Sample (being evaluated) Release: 4.19
 Start Time: 2025-06-04T00:00:00Z
 End Time: 2025-06-11T16:00:00Z
 Success Rate: 91.43%
 Successes: 32
 Failures: 3
 Flakes: 0
 
 Base (historical) Release: 4.15
 Start Time: 2024-01-29T00:00:00Z
 End Time: 2024-02-28T00:00:00Z
 Success Rate: 100.00%
 Successes: 54
 Failures: 0
 Flakes: 0
 
 View the [test details report|https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/test_details?Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&Network=ovn&Network=ovn&Platform=vsphere&Platform=vsphere&Suite=serial&Suite=serial&Topology=ha&Topology=ha&Upgrade=none&Upgrade=none&baseEndTime=2025-02-25%2023%3A59%3A59&baseRelease=4.18&baseStartTime=2025-01-26%2000%3A00%3A00&capability=Other&columnGroupBy=Architecture%2CNetwork%2CPlatform%2CTopology&component=kube-apiserver&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=Architecture%3Aamd64%20FeatureSet%3Adefault%20Installer%3Aipi%20Network%3Aovn%20Platform%3Avsphere%20Suite%3Aserial%20Topology%3Aha%20Upgrade%3Anone&flakeAsFailure=false&ignoreDisruption=true&ignoreMissing=false&includeMultiReleaseAnalysis=true&includeVariant=Architecture%3Aamd64&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Acrun&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=FeatureSet%3Atechpreview&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=JobTier%3Ablocking&includeVariant=JobTier%3Ainforming&includeVariant=JobTier%3Astandard&includeVariant=LayeredProduct%3Anone&includeVariant=Network%3Aovn&includeVariant=Owner%3Aeng&includeVariant=Owner%3Aservice-delivery&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Arosa&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2025-06-11%2023%3A59%3A59&sampleRelease=4.19&sampleStartTime=2025-06-04%2000%3A00%3A00&testBasisRelease=4.15&testId=openshift-tests%3A9ff4e9b171ea809e0d6faf721b2fe737&testName=%5Bsig-arch%5D%5BLate%5D%20operators%20should%20not%20create%20watch%20channels%20very%20often%20%5Bapigroup%3Aapiserver.openshift.io%5D%20%5BSuite%3Aopenshift%2Fconformance%2Fparallel%5D] for additional context.
             
 This is the test that was silently disabled and stopped failing and Forrest has been working steadily to try to get it healthy again with latest updated watch counts. It looks like the last attempt was just very slightly off, failing with a ratio of 1.01. We need to bump a few more slightly. Probably best to scan all test failures of this globally and accomodate.
Status: Verified

Issue 17381748: [sig-arch][Late] operators should not create watch channels very often
Description: CI Lanes for SNO presubmits serial are failing due to increase in watch counts, we need to determine if this is okay and increase the limit or if something has been incorrectly added.
 
 https://prow.ci.openshift.org/job-history/gs/test-platform-results/pr-logs/directory/pull-ci-openshift-origin-main-e2e-aws-ovn-single-node-serial

Issue 16880351: operators should not create watch channels very often regression
Description: [TRT-2049|https://issues.redhat.com/browse/TRT-2049] found that the method for collecting watch counts was changed to track only deprecated api calls.  This left a gap in monitoring until it was discovered and the logic moved to audit log analysis monitor tests. 
 {code:none}
 [sig-arch][Late] operators should not create watch channels very often [apigroup:apiserver.openshift.io] [Suite:openshift/conformance/parallel]
 {code}
 is showing [regressed|https://sippy.dptools.openshift.org/sippy-ng/component_readiness/capability?baseEndTime=2025-04-10%2023%3A59%3A59&baseRelease=4.18&baseStartTime=2025-03-11%2000%3A00%3A00&capability=Other&columnGroupBy=Architecture%2CNetwork%2CPlatform%2CTopology&component=kube-apiserver&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&flakeAsFailure=false&ignoreDisruption=true&ignoreMissing=false&includeMultiReleaseAnalysis=true&includeVariant=Architecture%3Aamd64&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Acrun&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=FeatureSet%3Atechpreview&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=JobTier%3Ablocking&includeVariant=JobTier%3Ainforming&includeVariant=JobTier%3Astandard&includeVariant=LayeredProduct%3Anone&includeVariant=Network%3Aovn&includeVariant=Owner%3Aeng&includeVariant=Owner%3Aservice-delivery&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Arosa&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&regressedModal=1&sampleEndTime=2025-04-10%2023%3A59%3A59&sampleRelease=4.19&sampleStartTime=2025-04-03%2000%3A00%3A00] for vshpere and gcp serial jobs currently.
 
 Now that we can collect the data again we need to analyze the current base line, backport the fix to 4.18 and potentially adjust the caps.
Status: Verified
Comment 28017799 by Ke Wang at 2025-09-09T11:06:10.109+0000
Let's see [last month Test report for Component|https://sippy.dptools.openshift.org/sippy-ng/component_readiness/capability?baseEndTime=2025-09-09%2023%3A59%3A59&baseRelease=4.19&baseStartTime=2025-08-09%2000%3A00%3A00&capability=Other&columnGroupBy=Architecture%2CNetwork%2CPlatform%2CTopology&component=kube-apiserver&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&flakeAsFailure=false&ignoreDisruption=true&ignoreMissing=false&includeMultiReleaseAnalysis=true&includeVariant=Architecture%3Aamd64&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Acrun&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=FeatureSet%3Atechpreview&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=JobTier%3Ablocking&includeVariant=JobTier%3Ainforming&includeVariant=JobTier%3Astandard&includeVariant=LayeredProduct%3Anone&includeVariant=Network%3Aovn&includeVariant=Owner%3Aeng&includeVariant=Owner%3Aservice-delivery&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Arosa&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2025-09-09%2023%3A59%3A59&sampleRelease=4.20&sampleStartTime=2025-08-09%2000%3A00%3A00],  test [\[sig-arch\]\[Late\] operators should not create watch channels very often \[apigroup:apiserver.openshift.io\] \[Suite:openshift/conformance/parallel\]|https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test?baseEndTime=2025-09-09%2023%3A59%3A59&baseRelease=4.19&baseStartTime=2025-08-09%2000%3A00%3A00&capability=Other&columnGroupBy=Architecture%2CNetwork%2CPlatform%2CTopology&component=kube-apiserver&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&flakeAsFailure=false&ignoreDisruption=true&ignoreMissing=false&includeMultiReleaseAnalysis=true&includeVariant=Architecture%3Aamd64&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Acrun&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=FeatureSet%3Atechpreview&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=JobTier%3Ablocking&includeVariant=JobTier%3Ainforming&includeVariant=JobTier%3Astandard&includeVariant=LayeredProduct%3Anone&includeVariant=Network%3Aovn&includeVariant=Owner%3Aeng&includeVariant=Owner%3Aservice-delivery&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Arosa&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2025-09-09%2023%3A59%3A59&sampleRelease=4.20&sampleStartTime=2025-08-09%2000%3A00%3A00&testId=openshift-tests%3A9ff4e9b171ea809e0d6faf721b2fe737] all are green, looks good, so moving the bug VERIFIED.
Comment 28091491 by UNKNOWN at 2025-09-19T13:59:48.979+0000