Job:
#OCPBUGS-41270issue4 weeks ago"pods should successfully create sandboxes by adding pod to network" are failing on multiple platforms Verified
Issue 16265652: "pods should successfully create sandboxes by adding pod to network" are failing on multiple platforms
Description: Component Readiness has found a potential regression in the following test:
 {code:java}
 [sig-network] pods should successfully create sandboxes by adding pod to network{code}
 Probability of significant regression: 96.41%
 
 Sample (being evaluated) Release: 4.17
 Start Time: 2024-08-27T00:00:00Z
 End Time: 2024-09-03T23:59:59Z
 Success Rate: 88.37%
 Successes: 26
 Failures: 5
 Flakes: 12
 
 Base (historical) Release: 4.16
 Start Time: 2024-05-31T00:00:00Z
 End Time: 2024-06-27T23:59:59Z
 Success Rate: 98.46%
 Successes: 43
 Failures: 1
 Flakes: 21
 
 View the test details report at [https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&Network=ovn&Network=ovn&NetworkAccess=default&Platform=metal&Platform=metal&Scheduler=default&SecurityMode=default&Suite=unknown&Suite=unknown&Topology=ha&Topology=ha&Upgrade=minor&Upgrade=minor&baseEndTime=2024-06-27%2023%3A59%3A59&baseRelease=4.16&baseStartTime=2024-05-31%2000%3A00%3A00&capability=Other&columnGroupBy=Platform%2CArchitecture%2CNetwork&component=Networking%20%2F%20cluster-network-operator&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20metal%20unknown%20ha%20minor&ignoreDisruption=true&ignoreMissing=false&includeVariant=Architecture%3Aamd64&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Owner%3Aeng&includeVariant=Platform%3Ametal&includeVariant=Topology%3Aha&minFail=3&pity=5&sampleEndTime=2024-09-03%2023%3A59%3A59&sampleRelease=4.17&sampleStartTime=2024-08-27%2000%3A00%3A00&testId=openshift-tests-upgrade%3A65e48733eb0b6115134b2b8c6a365f16&testName=%5Bsig-network%5D%20pods%20should%20successfully%20create%20sandboxes%20by%20adding%20pod%20to%20network]
 
  
 
 Here is [an example run|https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-upgrade-from-stable-4.16-e2e-metal-ipi-ovn-upgrade/1830563649966575616]. 
 
 We see the following signature for the failure:
 
  
 {code:java}
 namespace/openshift-etcd node/master-0 pod/revision-pruner-11-master-0 hmsg/b90fda805a - 111.86 seconds after deletion - firstTimestamp/2024-09-02T13:14:37Z interesting/true lastTimestamp/2024-09-02T13:14:37Z reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-11-master-0_openshift-etcd_08346d8f-7d22-4d70-ab40-538a67e21e3c_0(d4b61f9ff9f2ddfd3b64352203e8a3eafc2c3bd7c3d31a0a573bc29e4ac6da57): error adding pod openshift-etcd_revision-pruner-11-master-0 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d4b61f9ff9f2ddfd3b64352203e8a3eafc2c3bd7c3d31a0a573bc29e4ac6da57" Netns:"/var/run/netns/97dc5eb9-19da-462f-8b2e-c301cfd7f3cf" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-etcd;K8S_POD_NAME=revision-pruner-11-master-0;K8S_POD_INFRA_CONTAINER_ID=d4b61f9ff9f2ddfd3b64352203e8a3eafc2c3bd7c3d31a0a573bc29e4ac6da57;K8S_POD_UID=08346d8f-7d22-4d70-ab40-538a67e21e3c" Path:"" ERRORED: error configuring pod [openshift-etcd/revision-pruner-11-master-0] networking: Multus: [openshift-etcd/revision-pruner-11-master-0/08346d8f-7d22-4d70-ab40-538a67e21e3c]: error waiting for pod: pod "revision-pruner-11-master-0" not found  {code}
  
 
 The same signature has been reported for both [azure|https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Aggregation=none&Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&Network=ovn&Network=ovn&NetworkAccess=default&Platform=azure&Platform=azure&Scheduler=default&SecurityMode=default&Suite=serial&Suite=serial&Topology=ha&Topology=ha&Upgrade=none&Upgrade=none&baseEndTime=2024-06-27%2023%3A59%3A59&baseRelease=4.16&baseStartTime=2024-05-31%2000%3A00%3A00&capability=Other&columnGroupBy=Platform%2CArchitecture%2CNetwork&component=Networking%20%2F%20cluster-network-operator&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20azure%20serial%20ha%20none&ignoreDisruption=true&ignoreMissing=false&includeVariant=Architecture%3Aamd64&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Owner%3Aeng&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&minFail=3&pity=5&sampleEndTime=2024-08-05%2023%3A59%3A59&sampleRelease=4.17&sampleStartTime=2024-07-30%2000%3A00%3A00&testId=openshift-tests%3A65e48733eb0b6115134b2b8c6a365f16&testName=%5Bsig-network%5D%20pods%20should%20successfully%20create%20sandboxes%20by%20adding%20pod%20to%20network] and [x390x |https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-multiarch-master-nightly-4.16-ocp-e2e-ovn-remote-libvirt-s390x/1787828354724925440]as well.
 
  
 
 It is worth mentioning that sdn to ovn transition adds some complication to our analysis. From the component readiness above, you will see most of the failures are for job: periodic-ci-openshift-release-master-nightly-X.X-upgrade-from-stable-X.X-e2e-metal-ipi-ovn-upgrade. This is a new job for 4.17 and therefore miss base stats in 4.16.
 
  
 
 So we ask for:
  # An analysis of the root cause and impact of this issue
  # Team can compare relevant 4.16 sdn jobs to see if this is really a regression.
  # Given the current passing rate of 88%, what priority we should give to this?
  # Since this is affecting component readiness, and management depends on a green dashboard for release decision, we need to figure out what is the best approach for handling this issue.
Status: Verified
#OCPBUGS-43630issue5 days agopods should successfully create sandboxes by adding pod to network is failing component readiness again CLOSED
Issue 16400209: pods should successfully create sandboxes by adding pod to network is failing component readiness again
Description: Component Readiness has found a potential regression in the following test:
 {code:java}
 [sig-network] pods should successfully create sandboxes by adding pod to network{code}
 Significant regression detected.
 Fishers Exact probability of a regression: 99.95%.
 Test pass rate dropped from 99.02% to 90.91%.
 
 Sample (being evaluated) Release: 4.18
 Start Time: 2024-10-14T00:00:00Z
 End Time: 2024-10-21T23:59:59Z
 Success Rate: 90.91%
 Successes: 25
 Failures: 3
 Flakes: 5
 
 Base (historical) Release: 4.17
 Start Time: 2024-09-01T00:00:00Z
 End Time: 2024-10-01T23:59:59Z
 Success Rate: 99.02%
 Successes: 90
 Failures: 1
 Flakes: 11
 
 View the test details report at [https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&Network=ovn&Network=ovn&NetworkAccess=default&Platform=vsphere&Platform=vsphere&Scheduler=default&SecurityMode=default&Suite=unknown&Suite=unknown&Topology=ha&Topology=ha&Upgrade=minor&Upgrade=minor&baseEndTime=2024-10-01%2023%3A59%3A59&baseRelease=4.17&baseStartTime=2024-09-01%2000%3A00%3A00&capability=Other&columnGroupBy=Architecture%2CNetwork%2CPlatform&component=Networking%20%2F%20cluster-network-operator&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20vsphere%20unknown%20ha%20minor&ignoreDisruption=true&ignoreMissing=false&includeVariant=Architecture%3Aamd64&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Owner%3Aeng&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2024-10-21%2023%3A59%3A59&sampleRelease=4.18&sampleStartTime=2024-10-14%2000%3A00%3A00&testId=openshift-tests-upgrade%3A65e48733eb0b6115134b2b8c6a365f16&testName=%5Bsig-network%5D%20pods%20should%20successfully%20create%20sandboxes%20by%20adding%20pod%20to%20network]
 
  
 
 There are three jobs failing in the past week. Two of them share a common signature "pod deleted before sandbox ADD operation began"
Status: CLOSED
periodic-ci-openshift-release-master-ci-4.18-e2e-gcp-ovn-upgrade (all) - 570 runs, 17% failed, 73% of failures match = 12% impact
#1856424378447695872junit2 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1856424366703644672junit2 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox
#1856333056479072256junit8 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
3 failures to create the sandbox
#1856333056726536192junit8 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1856333056349048832junit8 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1856323544300916736junit8 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox
#1856242539959422976junit14 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1856242529897287680junit14 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
4 failures to create the sandbox
#1856205945466523648junit16 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1856151922046668800junit20 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1856151916170448896junit20 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855970718957703168junit32 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855970726079631360junit32 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855970718789931008junit32 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox
#1855880164559294464junit38 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855789513847083008junit44 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855789518880247808junit44 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox
#1855789523074551808junit44 hours ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855662413290934272junit2 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox
#1855662406580047872junit2 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855662414972850176junit2 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855662404063465472junit2 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
5 failures to create the sandbox
#1855572866175602688junit2 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox
#1855572872882294784junit2 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
3 failures to create the sandbox
#1855572868687990784junit2 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855452006378901504junit2 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855389294160515072junit2 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855361889391874048junit3 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox
#1855361881850515456junit3 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855361880164405248junit3 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox
#1855271605001785344junit3 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855088772706734080junit3 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1855088751735214080junit3 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox
#1854998118798462976junit4 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1854816932549627904junit4 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1854816944281096192junit4 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1854766693482500096junit4 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1854728166208180224junit4 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1854637511335219200junit5 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1854546699427516416junit5 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1854427118247612416junit5 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1854368951090286592junit5 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1854183251346722816junit6 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1854093105448357888junit6 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1854003087748894720junit6 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1853912435824529408junit7 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1853798279364481024junit7 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1853708541257322496junit7 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1853618012620328960junit7 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
3 failures to create the sandbox
#1853527079929253888junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1853527064825565184junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox
#1853527064825565184junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1853506500476014592junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1853485836025204736junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
5 failures to create the sandbox
#1853393350812504064junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox
#1853347216391933952junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1853347216274493440junit8 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
3 failures to create the sandbox
#1853100475008684032junit9 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1852668269186519040junit10 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1852577846522810368junit10 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1852398423563571200junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1852306924544987136junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1852206974448439296junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1852118172866449408junit11 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1852028283739181056junit12 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1851847062186889216junit12 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
8 failures to create the sandbox
#1851754924375805952junit12 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox
#1851665630827122688junit13 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1851483675863027712junit13 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1851483680896192512junit13 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox
#1851483686772412416junit13 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox
#1851391858446438400junit13 days ago
# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox

Found in 12.46% of runs (73.20% of failures) across 570 total runs and 1 jobs (17.02% failed) in 11.703s - clear search | chart view - source code located on github