Issue 16460707: CI fails on "[sig-arch][Late] operators should not create watch channels very often"
Description: h2. Description of problem
 CI is flaky because of test failures such as the following:
 {noformat}
 {  fail [github.com/openshift/origin/test/extended/apiserver/api_requests.go:406]: Expected
     <[]string | len:1, cap:1>: [
         "Operator \"cluster-samples-operator\" produces more watch requests than expected: watchrequestcount=151, upperbound=118, ratio=1.28",
     ]
 to be empty
 Ginkgo exit error 1: exit with code 1}
 {noformat}
 This particular failure comes from [https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1147/pull-ci-openshift-cluster-ingress-operator-master-e2e-azure-ovn/1857099470299205632].  Search.ci has other similar failures.
 h2. Version-Release number of selected component (if applicable)
 4.18 and 4.19.
 h2. How reproducible
 Presently, [search.ci shows|https://search.dptools.openshift.org/?search=Operator+%5C%5C%22cluster-samples-operator%5C%5C%22+produces+more+watch+requests+than+expected&maxAge=48h&context=1&type=build-log&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job] the following stats for the past two days:
 {noformat}
 pull-ci-openshift-origin-master-e2e-openstack-ovn (all) - 32 runs, 25% failed, 38% of failures match = 9% impact
 pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-kube-apiserver-rollout (all) - 32 runs, 38% failed, 33% of failures match = 13% impact
 pull-ci-openshift-origin-master-e2e-metal-ipi-ovn (all) - 34 runs, 38% failed, 15% of failures match = 6% impact
 pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-ipv6 (all) - 44 runs, 30% failed, 46% of failures match = 14% impact
 pull-ci-openshift-machine-api-operator-master-e2e-metal-ipi (all) - 3 runs, 67% failed, 50% of failures match = 33% impact
 periodic-ci-openshift-release-master-ci-4.18-e2e-azure-ovn-upgrade (all) - 64 runs, 38% failed, 4% of failures match = 2% impact
 periodic-ci-openshift-release-master-nightly-4.18-e2e-gcp-ovn-serial (all) - 6 runs, 17% failed, 100% of failures match = 17% impact
 periodic-ci-openshift-release-master-nightly-4.18-e2e-vsphere-ovn-serial (all) - 6 runs, 67% failed, 25% of failures match = 17% impact
 periodic-ci-openshift-release-master-nightly-4.18-e2e-metal-ipi-ovn-techpreview (all) - 6 runs, 100% failed, 17% of failures match = 17% impact
 pull-ci-openshift-cluster-ingress-operator-master-e2e-azure-ovn (all) - 16 runs, 31% failed, 40% of failures match = 13% impact
 periodic-ci-openshift-release-master-nightly-4.18-e2e-agent-compact-fips (all) - 8 runs, 63% failed, 40% of failures match = 25% impact
 pull-ci-openshift-origin-master-e2e-agnostic-ovn-cmd (all) - 34 runs, 38% failed, 8% of failures match = 3% impact
 periodic-ci-openshift-multiarch-master-nightly-4.18-ocp-e2e-gcp-ovn-multi-a-a (all) - 5 runs, 40% failed, 50% of failures match = 20% impact
 openshift-origin-29265-nightly-4.18-e2e-metal-ipi-ovn-dualstack-techpreview (all) - 3 runs, 100% failed, 67% of failures match = 67% impact
 pull-ci-openshift-insights-operator-master-e2e-metal-ipi-ovn-ipv6 (all) - 17 runs, 29% failed, 40% of failures match = 12% impact
 periodic-ci-openshift-release-master-ci-4.18-e2e-gcp-ovn-upgrade (all) - 77 runs, 25% failed, 5% of failures match = 1% impact
 periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-compact-fips (all) - 9 runs, 44% failed, 50% of failures match = 22% impact
 periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-ha-dualstack-conformance (all) - 9 runs, 33% failed, 67% of failures match = 22% impact
 openshift-etcd-302-nightly-4.18-e2e-metal-ipi-serial-ovn-ipv6 (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
 openshift-etcd-302-nightly-4.18-e2e-metal-ipi-ovn-ipv6 (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
 periodic-ci-openshift-release-master-ci-4.19-e2e-gcp-ovn-upgrade (all) - 35 runs, 34% failed, 25% of failures match = 9% impact
 pull-ci-openshift-machine-api-operator-master-e2e-metal-ipi-ovn-ipv6 (all) - 3 runs, 67% failed, 50% of failures match = 33% impact
 pull-ci-openshift-origin-master-e2e-gcp-ovn (all) - 46 runs, 22% failed, 20% of failures match = 4% impact
 periodic-ci-openshift-release-master-nightly-4.18-e2e-metal-ipi-serial-ovn-dualstack (all) - 6 runs, 50% failed, 33% of failures match = 17% impact
 pull-ci-openshift-machine-api-operator-master-e2e-gcp-ovn (all) - 3 runs, 67% failed, 50% of failures match = 33% impact
 periodic-ci-openshift-release-master-nightly-4.18-e2e-metal-ipi-ovn-ipv6 (all) - 8 runs, 25% failed, 50% of failures match = 13% impact
 periodic-ci-openshift-release-master-nightly-4.18-e2e-metal-ipi-ovn-bm (all) - 8 runs, 25% failed, 50% of failures match = 13% impact
 periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-ovn-ipv6-techpreview (all) - 7 runs, 100% failed, 14% of failures match = 14% impact
 periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-ovn-dualstack-techpreview (all) - 7 runs, 100% failed, 14% of failures match = 14% impact
 pull-ci-openshift-machine-config-operator-master-e2e-vsphere-ovn-zones (all) - 24 runs, 29% failed, 14% of failures match = 4% impact
 pull-ci-openshift-installer-master-e2e-gcp-ovn-xpn (all) - 5 runs, 20% failed, 100% of failures match = 20% impact
 pull-ci-openshift-installer-master-e2e-aws-ovn (all) - 30 runs, 10% failed, 33% of failures match = 3% impact
 {noformat}
 h2. Steps to Reproduce
 1. Post a PR and have bad luck.
 2. Check search.ci: https://search.dptools.openshift.org/?search=Operator+%5C%5C%22cluster-samples-operator%5C%5C%22+produces+more+watch+requests+than+expected&maxAge=48h&context=1&type=build-log&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job
 h2. Actual results
 CI fails.
 h2. Expected results
 CI passes, or fails on some other test failure.

Issue 16490447: Enable ResilientWatchCacheInitialization in openshift-apiserver
Description: We've seen a high rate of failure for this test since last Thursday.
 
  
 ----
 ({_}Feel free to update this bug's summary to be more specific.{_})
 Component Readiness has found a potential regression in the following test:
 {code:none}
 [sig-arch][Late] operators should not create watch channels very often [apigroup:apiserver.openshift.io] [Suite:openshift/conformance/parallel]{code}
 Extreme regression detected.
 Fishers Exact probability of a regression: 100.00%.
 Test pass rate dropped from 100.00% to 78.33%.
 
 Sample (being evaluated) Release: 4.18
 Start Time: 2024-11-04T00:00:00Z
 End Time: 2024-11-18T23:59:59Z
 Success Rate: 78.33%
 Successes: 47
 Failures: 13
 Flakes: 0
 
 Base (historical) Release: 4.17
 Start Time: 2024-09-01T00:00:00Z
 End Time: 2024-10-01T23:59:59Z
 Success Rate: 100.00%
 Successes: 133
 Failures: 0
 Flakes: 0
 
 View the [test details report|https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Aggregation=none&Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&LayeredProduct=none&Network=ovn&Network=ovn&NetworkAccess=default&Platform=metal&Platform=metal&Procedure=none&Scheduler=default&SecurityMode=default&Suite=parallel&Suite=parallel&Topology=ha&Topology=ha&Upgrade=none&Upgrade=none&baseEndTime=2024-10-01%2023%3A59%3A59&baseRelease=4.17&baseStartTime=2024-09-01%2000%3A00%3A00&capability=Other&columnGroupBy=Architecture%2CNetwork%2CPlatform%2CTopology&component=Unknown&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20metal%20parallel%20ha%20none&ignoreDisruption=1&ignoreMissing=0&includeVariant=Architecture%3Aamd64&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Acrun&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Network%3Aovn&includeVariant=Owner%3Aeng&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2024-11-18%2023%3A59%3A59&samplePRNumber=&samplePROrg=&samplePRRepo=&sampleRelease=4.18&sampleStartTime=2024-11-04%2000%3A00%3A00&testId=openshift-tests%3A9ff4e9b171ea809e0d6faf721b2fe737&testName=%5Bsig-arch%5D%5BLate%5D%20operators%20should%20not%20create%20watch%20channels%20very%20often%20%5Bapigroup%3Aapiserver.openshift.io%5D%20%5BSuite%3Aopenshift%2Fconformance%2Fparallel%5D&view=] for additional context.
Status: CLOSED

Issue 16466199: Component Readiness: operators should not create watch channels very often
Description: We've seen a high rate of failure for this test since last Thursday.
 
  
 ----
 ({_}Feel free to update this bug's summary to be more specific.{_})
 Component Readiness has found a potential regression in the following test:
 {code:none}
 [sig-arch][Late] operators should not create watch channels very often [apigroup:apiserver.openshift.io] [Suite:openshift/conformance/parallel]{code}
 Extreme regression detected.
 Fishers Exact probability of a regression: 100.00%.
 Test pass rate dropped from 100.00% to 78.33%.
 
 Sample (being evaluated) Release: 4.18
 Start Time: 2024-11-04T00:00:00Z
 End Time: 2024-11-18T23:59:59Z
 Success Rate: 78.33%
 Successes: 47
 Failures: 13
 Flakes: 0
 
 Base (historical) Release: 4.17
 Start Time: 2024-09-01T00:00:00Z
 End Time: 2024-10-01T23:59:59Z
 Success Rate: 100.00%
 Successes: 133
 Failures: 0
 Flakes: 0
 
 View the [test details report|https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Aggregation=none&Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&LayeredProduct=none&Network=ovn&Network=ovn&NetworkAccess=default&Platform=metal&Platform=metal&Procedure=none&Scheduler=default&SecurityMode=default&Suite=parallel&Suite=parallel&Topology=ha&Topology=ha&Upgrade=none&Upgrade=none&baseEndTime=2024-10-01%2023%3A59%3A59&baseRelease=4.17&baseStartTime=2024-09-01%2000%3A00%3A00&capability=Other&columnGroupBy=Architecture%2CNetwork%2CPlatform%2CTopology&component=Unknown&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20metal%20parallel%20ha%20none&ignoreDisruption=1&ignoreMissing=0&includeVariant=Architecture%3Aamd64&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Acrun&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Network%3Aovn&includeVariant=Owner%3Aeng&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2024-11-18%2023%3A59%3A59&samplePRNumber=&samplePROrg=&samplePRRepo=&sampleRelease=4.18&sampleStartTime=2024-11-04%2000%3A00%3A00&testId=openshift-tests%3A9ff4e9b171ea809e0d6faf721b2fe737&testName=%5Bsig-arch%5D%5BLate%5D%20operators%20should%20not%20create%20watch%20channels%20very%20often%20%5Bapigroup%3Aapiserver.openshift.io%5D%20%5BSuite%3Aopenshift%2Fconformance%2Fparallel%5D&view=] for additional context.
Status: ASSIGNED

Issue 16493683: Component Readiness: operators should not create watch channels very often
Description: This is a clone of issue OCPBUGS-44693. The following is the description of the original issue:  --- We've seen a high rate of failure for this test since last Thursday.
 
  
 ----
 ({_}Feel free to update this bug's summary to be more specific.{_})
 Component Readiness has found a potential regression in the following test:
 {code:none}
 [sig-arch][Late] operators should not create watch channels very often [apigroup:apiserver.openshift.io] [Suite:openshift/conformance/parallel]{code}
 Extreme regression detected.
 Fishers Exact probability of a regression: 100.00%.
 Test pass rate dropped from 100.00% to 78.33%.
 
 Sample (being evaluated) Release: 4.18
 Start Time: 2024-11-04T00:00:00Z
 End Time: 2024-11-18T23:59:59Z
 Success Rate: 78.33%
 Successes: 47
 Failures: 13
 Flakes: 0
 
 Base (historical) Release: 4.17
 Start Time: 2024-09-01T00:00:00Z
 End Time: 2024-10-01T23:59:59Z
 Success Rate: 100.00%
 Successes: 133
 Failures: 0
 Flakes: 0
 
 View the [test details report|https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Aggregation=none&Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&LayeredProduct=none&Network=ovn&Network=ovn&NetworkAccess=default&Platform=metal&Platform=metal&Procedure=none&Scheduler=default&SecurityMode=default&Suite=parallel&Suite=parallel&Topology=ha&Topology=ha&Upgrade=none&Upgrade=none&baseEndTime=2024-10-01%2023%3A59%3A59&baseRelease=4.17&baseStartTime=2024-09-01%2000%3A00%3A00&capability=Other&columnGroupBy=Architecture%2CNetwork%2CPlatform%2CTopology&component=Unknown&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20metal%20parallel%20ha%20none&ignoreDisruption=1&ignoreMissing=0&includeVariant=Architecture%3Aamd64&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Acrun&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Network%3Aovn&includeVariant=Owner%3Aeng&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2024-11-18%2023%3A59%3A59&samplePRNumber=&samplePROrg=&samplePRRepo=&sampleRelease=4.18&sampleStartTime=2024-11-04%2000%3A00%3A00&testId=openshift-tests%3A9ff4e9b171ea809e0d6faf721b2fe737&testName=%5Bsig-arch%5D%5BLate%5D%20operators%20should%20not%20create%20watch%20channels%20very%20often%20%5Bapigroup%3Aapiserver.openshift.io%5D%20%5BSuite%3Aopenshift%2Fconformance%2Fparallel%5D&view=] for additional context.
Status: CLOSED

Issue 16504733: Component Readiness: {OCPBUGS Test Framework} test regressed
Description: Component Readiness has found a potential regression in the following tests:{code}[sig-arch] events should not repeat pathologically for ns/openshift-cluster-csi-drivers{code} Extreme regression detected. Fishers Exact probability of a regression: 100.00%. Test pass rate dropped from 100.00% to 62.50%.{code}[sig-arch][Late] operators should not create watch channels very often [apigroup:apiserver.openshift.io] [Suite:openshift/conformance/parallel]{code} Extreme regression detected. Fishers Exact probability of a regression: 100.00%. Test pass rate dropped from 100.00% to 84.48%.{code}API LBs follow /readyz of kube-apiserver and stop sending requests before server shutdowns for external clients{code} Test has a 60.00% pass rate, but 95.00% is required.{code}[sig-arch] events should not repeat pathologically for ns/openshift-cluster-csi-drivers{code} Significant regression detected. Fishers Exact probability of a regression: 99.99%. Test pass rate dropped from 100.00% to 88.00%.{code}[sig-arch][Late] clients should not use APIs that are removed in upcoming releases [apigroup:apiserver.openshift.io] [Suite:openshift/conformance/parallel]{code} Significant regression detected. Fishers Exact probability of a regression: 100.00%. Test pass rate dropped from 100.00% to 86.44%.  h4. Potentially Related Issues:  * This regression might be related to [OCPBUGS-44387|https://issues.redhat.com/rest/api/2/issue/16445723]. Feel free to link it if found related.   h4. Useful Component Readiness Links:  We are proving the following two links for your convenience:  - Click [here|https://sippy.dptools.openshift.org//sippy-ng/component_readiness/main?baseEndTime=2024-10-01T23%3A59%3A59Z&baseRelease=4.17&baseStartTime=2024-09-01T00%3A00%3A00Z&columnGroupBy=Architecture%2CNetwork%2CPlatform%2CTopology&confidence=95&ignoreDisruption=1&ignoreMissing=0&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Owner%3Aeng&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&includeVariant=CGroupMode%3Av2&includeVariant=Architecture%3Aamd64&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=ContainerRuntime%3Arunc&includeVariant=ContainerRuntime%3Acrun&includeVariant=Network%3Aovn&minFail=3&pity=5&sampleEndTime=2024-12-04T16%3A00%3A00Z&sampleRelease=4.18&sampleStartTime=2024-11-27T00%3A00%3A00Z] to access a snapshot of the component readiness page at the time this issue was generated.  - Click [here|https://sippy.dptools.openshift.org//sippy-ng/component_readiness/main?view=4.18-main] to access the component readiness page with latest data. This is useful for developers to verify their fixes.   h4. Workflow Requirement:   This is an automatically generated Jira card against the component based on stats generated from component readiness dashboard. Please follow the following requirements when dealing with this card:   * Please use this card as a placeholder for all the regressed tests for your component. A separate issue should be created for each regressed test and linked to this issue. Please only close this issue when all known regressed tests are believed fixed.   * Please do not remove 'Release Blocker' label. The bot will automatically add it back if any regressed tests continue showing for the component.
Status: New

[sig-arch][Late] operators should not create watch channels very often [Suite:openshift/conformance/parallel]

[sig-arch][Late] operators should not create watch channels very often [Suite:openshift/conformance/parallel]

# [sig-arch][Late] operators should not create watch channels very often [Suite:openshift/conformance/parallel]
fail [github.com/openshift/origin/test/extended/apiserver/api_requests.go:448]: Expected

# [sig-arch][Late] operators should not create watch channels very often [Suite:openshift/conformance/parallel]
fail [github.com/openshift/origin/test/extended/apiserver/api_requests.go:448]: Expected

Nov 25 12:58:41.062 E ns/e2e-test-cli-deployment-gndbz pod/history-limit-11-v7hbr node/ip-10-0-178-90.ec2.internal container/myapp reason/ContainerExit code/137 cause/Error
Nov 25 13:01:05.712 - 1s    E e2e-test/"[sig-arch][Late] operators should not create watch channels very often [Suite:openshift/conformance/parallel]" e2e test finished As "Failed"
Nov 25 13:01:05.847 E e2e-test/"[sig-instrumentation][Late] Alerts shouldn't report any unexpected alerts in firing or pending state [Suite:openshift/conformance/parallel]" finishedStatus/Flaked
Nov 25 13:01:07.251 E e2e-test/"[sig-arch][Late] operators should not create watch channels very often [Suite:openshift/conformance/parallel]" finishedStatus/Failed
Nov 25 13:01:07.514 E e2e-test/"[sig-arch][Late] clients should not use APIs that are removed in upcoming releases [Suite:openshift/conformance/parallel]" finishedStatus/Flaked