Issue 17081653: "[sig-network] pods should successfully create sandboxes by adding pod to network" fails often on compact CI jobs
Description: This conformance test has been failing regularly in our CI:
 
 4.18 example: [https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.18-e2e-agent-compact-fips/1936031560633421824]
 
 4.17 example: [https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-agent-compact-fips/1934953258862776320]

Issue 16573555: Failed to create the sandbox-plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": EOF [Openshift-4.14]
Description: Description of problem:{code:none}
 In the 4.14 z-stream rollback job, I'm seeing test-case "[sig-network] pods should successfully create sandboxes by adding pod to network " fail. 
 
 The job link is here https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-upgrade-rollback-oldest-supported/1719037590788640768
 
 The error is:
 
 56 failures to create the sandbox
 
 ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-48-75.us-east-2.compute.internal - 3314.57 seconds after deletion - reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_prometheus-k8s-1_openshift-monitoring_95d1a457-3e1b-4ae3-8b57-8023eec5937d_0(5b36bc12b2964e85bcdbe60b275d6a12ea68cb18b81f16622a6cb686270c4eb3): error adding pod openshift-monitoring_prometheus-k8s-1 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": EOF
 ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-48-75.us-east-2.compute.internal - 3321.57 seconds after deletion - reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_prometheus-k8s-1_openshift-monitoring_95d1a457-3e1b-4ae3-8b57-8023eec5937d_0(3cc0afc5bec362566e4c3bdaf822209377102c2e39aaa8ef5d99b0f4ba795aaf): error adding pod openshift-monitoring_prometheus-k8s-1 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": dial unix /run/multus/socket/multus.sock: connect: connection refused
 
 
 {code}
 Version-Release number of selected component (if applicable):{code:none}
 4.14.0-0.nightly-2023-10-30-170011
 {code}
 How reproducible:{code:none}
 Flaky
 {code}
 Steps to Reproduce:{code:none}
 1.
 2.
 3.
 {code}
 Actual results:{code:none}
 
 {code}
 Expected results:{code:none}
 
 {code}
 Additional info:{code:none}
 The rollback test is testing by installing 4.14.0, then upgrade to the latest 4.14.nightly, at some random point, rolling back to 4.14.0
 {code}
Status: Verified

Issue 17069730: Component Readiness: pods should successfully create sandboxes by adding pod to network: expected pod UID "aa853924-c6c6-45b7-be56-e059960bc3c6" but got "ab26e0dc-d736-4945-aa02-91fa3f066cdc" from Kube API
Description: (_Feel free to update this bug's summary to be more specific._)
 Component Readiness has found a potential regression in the following test:
 
 {code:none}[sig-network] pods should successfully create sandboxes by adding pod to network{code}
 
 Significant regression detected.
 Fishers Exact probability of a regression: 100.00%.
 Test pass rate dropped from 99.42% to 93.99%.
 
 Sample (being evaluated) Release: 4.19
 Start Time: 2025-06-10T00:00:00Z
 End Time: 2025-06-17T12:00:00Z
 Success Rate: 93.99%
 Successes: 212
 Failures: 14
 Flakes: 7
 
 Base (historical) Release: 4.17
 Start Time: 2024-09-01T00:00:00Z
 End Time: 2024-10-01T00:00:00Z
 Success Rate: 99.42%
 Successes: 803
 Failures: 5
 Flakes: 54
 
 View the [test details report|https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/test_details?Architecture=amd64&FeatureSet=default&Installer=ipi&Network=ovn&Platform=azure&Suite=unknown&Topology=ha&Upgrade=minor&baseEndTime=2025-02-25%2023%3A59%3A59&baseRelease=4.18&baseStartTime=2025-01-26%2000%3A00%3A00&capability=Other&columnGroupBy=Architecture%2CNetwork%2CPlatform%2CTopology&component=Networking%20%2F%20cluster-network-operator&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=Architecture%3Aamd64%20FeatureSet%3Adefault%20Installer%3Aipi%20Network%3Aovn%20Platform%3Aazure%20Suite%3Aunknown%20Topology%3Aha%20Upgrade%3Aminor&flakeAsFailure=false&ignoreDisruption=true&ignoreMissing=false&includeMultiReleaseAnalysis=true&includeVariant=Architecture%3Aamd64&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Acrun&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=FeatureSet%3Atechpreview&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=JobTier%3Ablocking&includeVariant=JobTier%3Ainforming&includeVariant=JobTier%3Astandard&includeVariant=LayeredProduct%3Anone&includeVariant=Network%3Aovn&includeVariant=Owner%3Aeng&includeVariant=Owner%3Aservice-delivery&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Arosa&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2025-06-17%2023%3A59%3A59&sampleRelease=4.19&sampleStartTime=2025-06-10%2000%3A00%3A00&testBasisRelease=4.17&testId=openshift-tests-upgrade%3A65e48733eb0b6115134b2b8c6a365f16&testName=%5Bsig-network%5D%20pods%20should%20successfully%20create%20sandboxes%20by%20adding%20pod%20to%20network] for additional context.
 
 Pretty clearly failing worse in the last week than 4.17 and 4.18 both.
Status: POST
{code:java}
: [Monitor:legacy-networking-invariants][sig-network] pods should successfully create sandboxes by adding pod to network expand_less    0s
{  1 failures to create the sandboxnamespace/e2e-statefulset-8898 node/ci-op-v5j2gm5l-2de33-gxlm6-compute-2 pod/ss2-0 hmsg/f7be97bdce - 20.29 seconds after deletion - firstTimestamp/2025-10-22T20:19:05Z interesting/true lastTimestamp/2025-10-22T20:19:05Z reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_ss2-0_e2e-statefulset-8898_9fc21ab8-0615-41eb-a3f4-8cdb4bbfbc34_0(16285e66aa0064163ceac16a9307ac90a75ea73b97504605d176f20af095eaa0): error adding pod e2e-statefulset-8898_ss2-0 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"16285e66aa0064163ceac16a9307ac90a75ea73b97504605d176f20af095eaa0" Netns:"/var/run/netns/8fc6ba8a-debc-4b02-ad7d-bf7d13c9335e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=e2e-statefulset-8898;K8S_POD_NAME=ss2-0;K8S_POD_INFRA_CONTAINER_ID=16285e66aa0064163ceac16a9307ac90a75ea73b97504605d176f20af095eaa0;K8S_POD_UID=9fc21ab8-0615-41eb-a3f4-8cdb4bbfbc34" Path:"" ERRORED: error configuring pod [e2e-statefulset-8898/ss2-0] networking: Multus: [e2e-statefulset-8898/ss2-0/9fc21ab8-0615-41eb-a3f4-8cdb4bbfbc34]: expected pod UID "9fc21ab8-0615-41eb-a3f4-8cdb4bbfbc34" but got "45cb07e2-c60c-42fb-97b5-59da83c3eb5b" from Kube API
{code:java}
: [Monitor:legacy-networking-invariants][sig-network] pods should successfully create sandboxes by adding pod to network expand_less    0s
{  4 failures to create the sandboxnamespace/openshift-kube-apiserver node/ci-op-xhbwst1j-4ef75-tgv5t-master-2 pod/kube-apiserver-guard-ci-op-xhbwst1j-4ef75-tgv5t-master-2 hmsg/acb70413dd - 19.68 seconds after deletion - firstTimestamp/2025-10-22T11:36:34Z interesting/true lastTimestamp/2025-10-22T11:36:34Z reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kube-apiserver-guard-ci-op-xhbwst1j-4ef75-tgv5t-master-2_openshift-kube-apiserver_6cdef0e6-94ed-47ee-b5e4-873cab30d34b_0(bda407d05265951eae10c2c7eb686dee423fc83643f897862551063dbc9b4c08): error adding pod openshift-kube-apiserver_kube-apiserver-guard-ci-op-xhbwst1j-4ef75-tgv5t-master-2 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"bda407d05265951eae10c2c7eb686dee423fc83643f897862551063dbc9b4c08" Netns:"/var/run/netns/be9c9470-8076-4796-ac98-51303a554f2e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-kube-apiserver;K8S_POD_NAME=kube-apiserver-guard-ci-op-xhbwst1j-4ef75-tgv5t-master-2;K8S_POD_INFRA_CONTAINER_ID=bda407d05265951eae10c2c7eb686dee423fc83643f897862551063dbc9b4c08;K8S_POD_UID=6cdef0e6-94ed-47ee-b5e4-873cab30d34b" Path:"" ERRORED: error configuring pod [openshift-kube-apiserver/kube-apiserver-guard-ci-op-xhbwst1j-4ef75-tgv5t-master-2] networking: Multus: [openshift-kube-apiserver/kube-apiserver-guard-ci-op-xhbwst1j-4ef75-tgv5t-master-2/6cdef0e6-94ed-47ee-b5e4-873cab30d34b]: expected pod UID "6cdef0e6-94ed-47ee-b5e4-873cab30d34b" but got "fd68519c-4c45-4f98-8252-2f8951e5ca37" from Kube API

Issue 17218475: Testcase failures on ppc64le libvirt CI runs in S2S environment - upgrade jobs
Description: Description of problem:
 
 The below testcase are failing in upgrade jobs
 
 1.  [sig-network] pods should successfully create sandboxes by other
 2. [bz-monitoring][invariant] alert/Watchdog must have no gaps or changes
 3. API LBs follow /readyz of kube-apiserver and stop sending requests before server shutdowns for external clients
 4. [sig-network] pods should successfully create sandboxes by adding pod to network 
 5. [sig-instrumentation] Prometheus [apigroup:image.openshift.io] when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early][apigroup:config.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel] 
 
 Joblink - [1954878323259609088|https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/67777/rehearse-67777-periodic-ci-openshift-multiarch-master-nightly-4.19-upgrade-from-nightly-4.18-ocp-ovn-remote-s2s-libvirt-multi-p-p/1954878323259609088]
Status: New

Issue 17503931: Component Readiness: pods should successfully create sandboxes by adding pod to network: adds operator Progressing checks to the deleted pod branch
Description: Description of problem:
 {code:none}
 For deleted pods, the code never checks if the failure occurred during operator Progressing. It only checks the time difference, which results in a hard failure at > 5 seconds. According to the code's own logic, if the etcd/dns/network operator was Progressing, sandbox failures should be treated more leniently (as flakes), but this check is missing in the deleted pod branch.{code}
 Version-Release number of selected component (if applicable):
 {code:none}
 4.21{code}
 How reproducible:
 {code:none}
 Prow CI testing   {code}
 Steps to Reproduce:
 {code:none}
 https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-azure-ovn-upgrade/1980375618641989632{code}
 Actual results:
 {code:none}
    namespace/openshift-etcd node/ci-op-tz2f8kj9-fbbf2-j6brz-master-0 pod/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0 hmsg/b1ad8ffd50 - 297.49 seconds after deletion - firstTimestamp/2025-10-20T22:22:31Z interesting/true lastTimestamp/2025-10-20T22:22:31Z reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0_openshift-etcd_4f678bb2-10f6-47f8-b080-36cbae9d50a4_0(09b24379f8fe8571b7a5eda9a668692a1f31d31d76772a7dabdb19c9778a43d0): error adding pod openshift-etcd_etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"09b24379f8fe8571b7a5eda9a668692a1f31d31d76772a7dabdb19c9778a43d0" Netns:"/var/run/netns/4ab6fd45-b6fa-4709-bb88-a4e7a81b8b3e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-etcd;K8S_POD_NAME=etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0;K8S_POD_INFRA_CONTAINER_ID=09b24379f8fe8571b7a5eda9a668692a1f31d31d76772a7dabdb19c9778a43d0;K8S_POD_UID=4f678bb2-10f6-47f8-b080-36cbae9d50a4" Path:"" ERRORED: error configuring pod [openshift-etcd/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0] networking: Multus: [openshift-etcd/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0/4f678bb2-10f6-47f8-b080-36cbae9d50a4]: error waiting for pod: Get "https://api-int.ci-op-tz2f8kj9-fbbf2.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:6443/api/v1/namespaces/openshift-etcd/pods/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0?timeout=1m0s": context deadline exceeded
 ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}} {code}
 Expected results:
 {code:none}
 Testing pass without errors  {code}
 Additional info:
 {code:none}
 Failed in another CI job:https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.21-upgrade-from-stable-4.20-e2e-metal-ipi-ovn-upgrade/1979889012780830720{code}