Job:
#OCPBUGS-41270issue4 weeks ago"pods should successfully create sandboxes by adding pod to network" are failing on multiple platforms Verified
Issue 16265652: "pods should successfully create sandboxes by adding pod to network" are failing on multiple platforms
Description: Component Readiness has found a potential regression in the following test:
 {code:java}
 [sig-network] pods should successfully create sandboxes by adding pod to network{code}
 Probability of significant regression: 96.41%
 
 Sample (being evaluated) Release: 4.17
 Start Time: 2024-08-27T00:00:00Z
 End Time: 2024-09-03T23:59:59Z
 Success Rate: 88.37%
 Successes: 26
 Failures: 5
 Flakes: 12
 
 Base (historical) Release: 4.16
 Start Time: 2024-05-31T00:00:00Z
 End Time: 2024-06-27T23:59:59Z
 Success Rate: 98.46%
 Successes: 43
 Failures: 1
 Flakes: 21
 
 View the test details report at [https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&Network=ovn&Network=ovn&NetworkAccess=default&Platform=metal&Platform=metal&Scheduler=default&SecurityMode=default&Suite=unknown&Suite=unknown&Topology=ha&Topology=ha&Upgrade=minor&Upgrade=minor&baseEndTime=2024-06-27%2023%3A59%3A59&baseRelease=4.16&baseStartTime=2024-05-31%2000%3A00%3A00&capability=Other&columnGroupBy=Platform%2CArchitecture%2CNetwork&component=Networking%20%2F%20cluster-network-operator&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20metal%20unknown%20ha%20minor&ignoreDisruption=true&ignoreMissing=false&includeVariant=Architecture%3Aamd64&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Owner%3Aeng&includeVariant=Platform%3Ametal&includeVariant=Topology%3Aha&minFail=3&pity=5&sampleEndTime=2024-09-03%2023%3A59%3A59&sampleRelease=4.17&sampleStartTime=2024-08-27%2000%3A00%3A00&testId=openshift-tests-upgrade%3A65e48733eb0b6115134b2b8c6a365f16&testName=%5Bsig-network%5D%20pods%20should%20successfully%20create%20sandboxes%20by%20adding%20pod%20to%20network]
 
  
 
 Here is [an example run|https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-upgrade-from-stable-4.16-e2e-metal-ipi-ovn-upgrade/1830563649966575616]. 
 
 We see the following signature for the failure:
 
  
 {code:java}
 namespace/openshift-etcd node/master-0 pod/revision-pruner-11-master-0 hmsg/b90fda805a - 111.86 seconds after deletion - firstTimestamp/2024-09-02T13:14:37Z interesting/true lastTimestamp/2024-09-02T13:14:37Z reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-11-master-0_openshift-etcd_08346d8f-7d22-4d70-ab40-538a67e21e3c_0(d4b61f9ff9f2ddfd3b64352203e8a3eafc2c3bd7c3d31a0a573bc29e4ac6da57): error adding pod openshift-etcd_revision-pruner-11-master-0 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d4b61f9ff9f2ddfd3b64352203e8a3eafc2c3bd7c3d31a0a573bc29e4ac6da57" Netns:"/var/run/netns/97dc5eb9-19da-462f-8b2e-c301cfd7f3cf" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-etcd;K8S_POD_NAME=revision-pruner-11-master-0;K8S_POD_INFRA_CONTAINER_ID=d4b61f9ff9f2ddfd3b64352203e8a3eafc2c3bd7c3d31a0a573bc29e4ac6da57;K8S_POD_UID=08346d8f-7d22-4d70-ab40-538a67e21e3c" Path:"" ERRORED: error configuring pod [openshift-etcd/revision-pruner-11-master-0] networking: Multus: [openshift-etcd/revision-pruner-11-master-0/08346d8f-7d22-4d70-ab40-538a67e21e3c]: error waiting for pod: pod "revision-pruner-11-master-0" not found  {code}
  
 
 The same signature has been reported for both [azure|https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Aggregation=none&Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&Network=ovn&Network=ovn&NetworkAccess=default&Platform=azure&Platform=azure&Scheduler=default&SecurityMode=default&Suite=serial&Suite=serial&Topology=ha&Topology=ha&Upgrade=none&Upgrade=none&baseEndTime=2024-06-27%2023%3A59%3A59&baseRelease=4.16&baseStartTime=2024-05-31%2000%3A00%3A00&capability=Other&columnGroupBy=Platform%2CArchitecture%2CNetwork&component=Networking%20%2F%20cluster-network-operator&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20azure%20serial%20ha%20none&ignoreDisruption=true&ignoreMissing=false&includeVariant=Architecture%3Aamd64&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Owner%3Aeng&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&minFail=3&pity=5&sampleEndTime=2024-08-05%2023%3A59%3A59&sampleRelease=4.17&sampleStartTime=2024-07-30%2000%3A00%3A00&testId=openshift-tests%3A65e48733eb0b6115134b2b8c6a365f16&testName=%5Bsig-network%5D%20pods%20should%20successfully%20create%20sandboxes%20by%20adding%20pod%20to%20network] and [x390x |https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-multiarch-master-nightly-4.16-ocp-e2e-ovn-remote-libvirt-s390x/1787828354724925440]as well.
 
  
 
 It is worth mentioning that sdn to ovn transition adds some complication to our analysis. From the component readiness above, you will see most of the failures are for job: periodic-ci-openshift-release-master-nightly-X.X-upgrade-from-stable-X.X-e2e-metal-ipi-ovn-upgrade. This is a new job for 4.17 and therefore miss base stats in 4.16.
 
  
 
 So we ask for:
  # An analysis of the root cause and impact of this issue
  # Team can compare relevant 4.16 sdn jobs to see if this is really a regression.
  # Given the current passing rate of 88%, what priority we should give to this?
  # Since this is affecting component readiness, and management depends on a green dashboard for release decision, we need to figure out what is the best approach for handling this issue.
Status: Verified
#OCPBUGS-43630issue5 days agopods should successfully create sandboxes by adding pod to network is failing component readiness again CLOSED
Issue 16400209: pods should successfully create sandboxes by adding pod to network is failing component readiness again
Description: Component Readiness has found a potential regression in the following test:
 {code:java}
 [sig-network] pods should successfully create sandboxes by adding pod to network{code}
 Significant regression detected.
 Fishers Exact probability of a regression: 99.95%.
 Test pass rate dropped from 99.02% to 90.91%.
 
 Sample (being evaluated) Release: 4.18
 Start Time: 2024-10-14T00:00:00Z
 End Time: 2024-10-21T23:59:59Z
 Success Rate: 90.91%
 Successes: 25
 Failures: 3
 Flakes: 5
 
 Base (historical) Release: 4.17
 Start Time: 2024-09-01T00:00:00Z
 End Time: 2024-10-01T23:59:59Z
 Success Rate: 99.02%
 Successes: 90
 Failures: 1
 Flakes: 11
 
 View the test details report at [https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&Network=ovn&Network=ovn&NetworkAccess=default&Platform=vsphere&Platform=vsphere&Scheduler=default&SecurityMode=default&Suite=unknown&Suite=unknown&Topology=ha&Topology=ha&Upgrade=minor&Upgrade=minor&baseEndTime=2024-10-01%2023%3A59%3A59&baseRelease=4.17&baseStartTime=2024-09-01%2000%3A00%3A00&capability=Other&columnGroupBy=Architecture%2CNetwork%2CPlatform&component=Networking%20%2F%20cluster-network-operator&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20vsphere%20unknown%20ha%20minor&ignoreDisruption=true&ignoreMissing=false&includeVariant=Architecture%3Aamd64&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Owner%3Aeng&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2024-10-21%2023%3A59%3A59&sampleRelease=4.18&sampleStartTime=2024-10-14%2000%3A00%3A00&testId=openshift-tests-upgrade%3A65e48733eb0b6115134b2b8c6a365f16&testName=%5Bsig-network%5D%20pods%20should%20successfully%20create%20sandboxes%20by%20adding%20pod%20to%20network]
 
  
 
 There are three jobs failing in the past week. Two of them share a common signature "pod deleted before sandbox ADD operation began"
Status: CLOSED
periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-aws-ovn-upgrade (all) - 43 runs, 33% failed, 264% of failures match = 86% impact
#1856396804069789696junit5 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
12 failures to create the sandbox
#1856396804069789696junit5 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
12 failures to create the sandbox
#1856251778677346304junit15 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
17 failures to create the sandbox
#1856251778677346304junit15 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
17 failures to create the sandbox
#1856345741769838592junit8 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
10 failures to create the sandbox
#1856345741769838592junit8 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
10 failures to create the sandbox
#1856124147340414976junit23 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
11 failures to create the sandbox
#1856124147340414976junit23 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
11 failures to create the sandbox
#1856033499685851136junit29 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
18 failures to create the sandbox
#1856033499685851136junit29 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
18 failures to create the sandbox
#1855943329439551488junit35 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
9 failures to create the sandbox
#1855943329439551488junit35 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
9 failures to create the sandbox
#1855175840594137088junit3 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
21 failures to create the sandbox
#1855175840594137088junit3 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
21 failures to create the sandbox
#1854974602103492608junit4 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
13 failures to create the sandbox
#1854974602103492608junit4 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
13 failures to create the sandbox
#1856447279838269440junit2 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
18 failures to create the sandbox
#1856447279838269440junit2 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
18 failures to create the sandbox
#1855224408419864576junit3 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855224408419864576junit3 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855224408419864576junit3 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
10 failures to create the sandbox
#1855224408419864576junit3 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
10 failures to create the sandbox
#1854649520856502272junit5 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
17 failures to create the sandbox
#1854649520856502272junit5 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
17 failures to create the sandbox
#1854740104032030720junit4 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
17 failures to create the sandbox
#1854740104032030720junit4 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
17 failures to create the sandbox
#1854595338556411904junit5 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
10 failures to create the sandbox
#1854595338556411904junit5 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
10 failures to create the sandbox
#1854415949369708544junit5 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
18 failures to create the sandbox
#1854415949369708544junit5 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
18 failures to create the sandbox
#1854541763981611008junit5 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
13 failures to create the sandbox
#1854541763981611008junit5 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
13 failures to create the sandbox
#1853976308350980096junit6 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
10 failures to create the sandbox
#1853976308350980096junit6 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
10 failures to create the sandbox
#1854168650378907648junit6 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
18 failures to create the sandbox
#1854168650378907648junit6 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
18 failures to create the sandbox
#1854116363526737920junit6 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
17 failures to create the sandbox
#1854116363526737920junit6 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
17 failures to create the sandbox
#1854028136807665664junit6 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
9 failures to create the sandbox
#1854028136807665664junit6 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
9 failures to create the sandbox
#1853836854965571584junit7 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
11 failures to create the sandbox
#1853836854965571584junit7 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
11 failures to create the sandbox
#1853926179728265216junit7 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
15 failures to create the sandbox
#1853926179728265216junit7 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
15 failures to create the sandbox
#1853746246817157120junit7 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
18 failures to create the sandbox
#1853746246817157120junit7 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
18 failures to create the sandbox
#1853655691756048384junit7 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
12 failures to create the sandbox
#1853655691756048384junit7 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
12 failures to create the sandbox
#1853562556812103680junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
16 failures to create the sandbox
#1853562556812103680junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
16 failures to create the sandbox
#1853484808353615872junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
16 failures to create the sandbox
#1853484808353615872junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
16 failures to create the sandbox
#1853348236828348416junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
15 failures to create the sandbox
#1853348236828348416junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
15 failures to create the sandbox
#1852428522274951168junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
12 failures to create the sandbox
#1852428522274951168junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
12 failures to create the sandbox
#1852339514224152576junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1852339514224152576junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1852339514224152576junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
16 failures to create the sandbox
#1852339514224152576junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
16 failures to create the sandbox
#1852240423561465856junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
19 failures to create the sandbox
#1852240423561465856junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
19 failures to create the sandbox
#1852149144722870272junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
14 failures to create the sandbox
#1852149144722870272junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
14 failures to create the sandbox
#1852056921167630336junit12 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
16 failures to create the sandbox
#1852056921167630336junit12 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
16 failures to create the sandbox
#1852005188760506368junit12 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
13 failures to create the sandbox
#1852005188760506368junit12 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
13 failures to create the sandbox
#1851955192665214976junit12 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
19 failures to create the sandbox
#1851955192665214976junit12 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
19 failures to create the sandbox
#1851863477124075520junit12 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
12 failures to create the sandbox
#1851863477124075520junit12 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
12 failures to create the sandbox
#1851604651338633216junit13 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
18 failures to create the sandbox
#1851604651338633216junit13 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
18 failures to create the sandbox
#1851445882138923008junit13 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
12 failures to create the sandbox
#1851445882138923008junit13 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
12 failures to create the sandbox
#1851495716191473664junit13 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
16 failures to create the sandbox
#1851495716191473664junit13 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
16 failures to create the sandbox

Found in 86.05% of runs (264.29% of failures) across 43 total runs and 1 jobs (32.56% failed) in 189ms - clear search | chart view - source code located on github