Issue 16291984: alert/KubeAPIErrorBudgetBurn should not be at or above info
Description: This test failed 3 times in the last week with the following error:
 {quote}{{{ KubeAPIErrorBudgetBurn was at or above info for at least 2m28s on platformidentification.JobType\{Release:"4.17", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h33m52s, firing for 2m28s: Sep 16 21:20:56.839 - 148s E namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/firing severity/critical ALERTS\{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}}}}
 {quote}
  
 
 It didn't fail a single time in the previous month on 4.17 nor in the month before we shipped 4.16 so I'm proposing this as a blocker to be investigated.  Below you have the boilerplate Component Readiness text:
 
  
 ----
 Component Readiness has found a potential regression in the following test:
 {code:java}
 [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above info{code}
 Probability of significant regression: 99.04%
 
 Sample (being evaluated) Release: 4.17
 Start Time: 2024-09-10T00:00:00Z
 End Time: 2024-09-17T23:59:59Z
 Success Rate: 85.71%
 Successes: 18
 Failures: 3
 Flakes: 0
 
 Base (historical) Release: 4.16
 Start Time: 2024-05-28T00:00:00Z
 End Time: 2024-06-27T23:59:59Z
 Success Rate: 100.00%
 Successes: 74
 Failures: 0
 Flakes: 0
 
 View the test details report at [https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Aggregation=none&Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&Network=ovn&Network=ovn&NetworkAccess=default&Platform=azure&Platform=azure&Scheduler=default&SecurityMode=default&Suite=serial&Suite=serial&Topology=ha&Topology=ha&Upgrade=none&Upgrade=none&baseEndTime=2024-06-27%2023%3A59%3A59&baseRelease=4.16&baseStartTime=2024-05-28%2000%3A00%3A00&capability=Alerts&columnGroupBy=Architecture%2CNetwork%2CPlatform&component=kube-apiserver&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20azure%20serial%20ha%20none&ignoreDisruption=1&ignoreMissing=0&includeVariant=Architecture%3Aamd64&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Owner%3Aeng&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&minFail=3&pity=5&sampleEndTime=2024-09-17%2023%3A59%3A59&samplePRNumber=&samplePROrg=&samplePRRepo=&sampleRelease=4.17&sampleStartTime=2024-09-10%2000%3A00%3A00&testId=openshift-tests%3Ad6b41cee7afca1c2a0b52f9e6975425f&testName=%5Bbz-kube-apiserver%5D%5Binvariant%5D%20alert%2FKubeAPIErrorBudgetBurn%20should%20not%20be%20at%20or%20above%20info&view=]
Status: POST
Comment 25602713 by Devan Goodwin at 2024-09-18T11:43:10.370+0000
A few pieces of data, using our [alert dashboard|https://grafana-loki.ci.openshift.org/d/PeFPT7XVk/ci-cluster-alerts?orgId=1&var-platform=aws&var-platform=azure&var-platform=gcp&var-platform=metal&var-platform=vsphere&var-percentile=P95&var-alerts=KubeAPIErrorBudgetBurn&var-namespaces=All&var-releases=4.17&var-upgrade_type=micro&var-upgrade_type=minor&var-upgrade_type=none&var-networks=sdn&var-networks=ovn&var-topologies=ha&var-architectures=amd64&var-min_job_runs=1&var-lookback=3&var-level=Critical&var-level=Warning&var-min_firing_seconds=1], specifically the job run list a little down the page, we can see this alert firing is quite novel, our first 4.17 hit ever detected on Aug 25. Looking back to 4.16 and 4.15, it seems exceedingly rare, I see just one job run on each release, but 4.17 has a couple dozen and they intensify quite a bit around Sep 12-14.

Mar 12 18:52:24.937 - 58s E namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/firing severity/critical ALERTS
{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="1h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="5m"}

Issue 16322985: alert/KubeAPIErrorBudgetBurn should not be at or above info
Description: This is a clone of issue OCPBUGS-42083. The following is the description of the original issue:  --- This test failed 3 times in the last week with the following error:
 {quote}{{{ KubeAPIErrorBudgetBurn was at or above info for at least 2m28s on platformidentification.JobType\{Release:"4.17", FromRelease:"", Platform:"azure", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 1h33m52s, firing for 2m28s: Sep 16 21:20:56.839 - 148s E namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/firing severity/critical ALERTS\{alertname="KubeAPIErrorBudgetBurn", alertstate="firing", long="6h", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="critical", short="30m"}}}}
 {quote}
  
 
 It didn't fail a single time in the previous month on 4.17 nor in the month before we shipped 4.16 so I'm proposing this as a blocker to be investigated.  Below you have the boilerplate Component Readiness text:
 
  
 ----
 Component Readiness has found a potential regression in the following test:
 {code:java}
 [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above info{code}
 Probability of significant regression: 99.04%
 
 Sample (being evaluated) Release: 4.17
 Start Time: 2024-09-10T00:00:00Z
 End Time: 2024-09-17T23:59:59Z
 Success Rate: 85.71%
 Successes: 18
 Failures: 3
 Flakes: 0
 
 Base (historical) Release: 4.16
 Start Time: 2024-05-28T00:00:00Z
 End Time: 2024-06-27T23:59:59Z
 Success Rate: 100.00%
 Successes: 74
 Failures: 0
 Flakes: 0
 
 View the test details report at [https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Aggregation=none&Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&Network=ovn&Network=ovn&NetworkAccess=default&Platform=azure&Platform=azure&Scheduler=default&SecurityMode=default&Suite=serial&Suite=serial&Topology=ha&Topology=ha&Upgrade=none&Upgrade=none&baseEndTime=2024-06-27%2023%3A59%3A59&baseRelease=4.16&baseStartTime=2024-05-28%2000%3A00%3A00&capability=Alerts&columnGroupBy=Architecture%2CNetwork%2CPlatform&component=kube-apiserver&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20azure%20serial%20ha%20none&ignoreDisruption=1&ignoreMissing=0&includeVariant=Architecture%3Aamd64&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Owner%3Aeng&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&minFail=3&pity=5&sampleEndTime=2024-09-17%2023%3A59%3A59&samplePRNumber=&samplePROrg=&samplePRRepo=&sampleRelease=4.17&sampleStartTime=2024-09-10%2000%3A00%3A00&testId=openshift-tests%3Ad6b41cee7afca1c2a0b52f9e6975425f&testName=%5Bbz-kube-apiserver%5D%5Binvariant%5D%20alert%2FKubeAPIErrorBudgetBurn%20should%20not%20be%20at%20or%20above%20info&view=]
Status: New

We have many guards making sure that there are always at least two instances of the kube-apiserver. If we ever reach a single kube-apiserver and it causes disruption for the clients, other alerts such as KubeAPIErrorBudgetBurn will fire.
KubeAPIDown is here to make sure that Prometheus and really any client can reach the kube-apiserver, which they can even when there is only one instance of kube-apiserver running. If they can't or that availability is disrupted, `KubeAPIErrorBudgetBurn` will fire.
Comment 23058588 by Marcel Härri at 2023-09-19T06:57:07.949+0000

# [bz-kube-apiserver][invariant] alert/KubeAPIErrorBudgetBurn should not be at or above pending
KubeAPIErrorBudgetBurn was at or above pending for at least 16m18s on platformidentification.JobType{Release:"4.17", FromRelease:"", Platform:"vsphere", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 16m18s, firing for 0s:
Sep 27 10:52:26.059 - 84s   I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="1d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="2h"}
Sep 27 10:52:26.059 - 894s  I namespace/openshift-kube-apiserver alert/KubeAPIErrorBudgetBurn alertstate/pending severity/warning ALERTS{alertname="KubeAPIErrorBudgetBurn", alertstate="pending", long="3d", namespace="openshift-kube-apiserver", prometheus="openshift-monitoring/k8s", severity="warning", short="6h"}