Issue 15554882: Ignore pod sandbox creation failures due to networking when the node is NetworkUnavailable=true
Description: The test:
 
 [sig-network] pods should successfully create sandboxes by adding pod to network
 
 Failed a couple payloads today with 1-2 failures in batches of 10 aggregated jobs. [I looked at the most recent errors|https://sippy.dptools.openshift.org/sippy-ng/tests/4.15/analysis?test=%5Bsig-network%5D%20pods%20should%20successfully%20create%20sandboxes%20by%20adding%20pod%20to%20network&filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22%5Bsig-network%5D%20pods%20should%20successfully%20create%20sandboxes%20by%20adding%20pod%20to%20network%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22never-stable%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22aggregated%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D] and they seem to often be the same:
 {code:java}
 1 failures to create the sandbox
 
 ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-24-217.us-west-1.compute.internal - 475.52 seconds after deletion - reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_prometheus-k8s-1_openshift-monitoring_c712fc61-5a1e-4cec-b6fa-18c8f2e91c0a_0(46df8384ffeb433fc0e4864262aa52f2ede570265c43bf8b0900f184b27b10f1): error adding pod openshift-monitoring_prometheus-k8s-1 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": EOF
 {code}
 This [http://dummy/cni] URL looked interesting and seemed worthy of a bug.
 
 The problem is a rare failure overall, but happening quite frequently day to day, search.ci indicates lots of hits over the last two days in both 4.14 and 4.15, and seemingly ovn and sdn both:
 
 [https://search.ci.openshift.org/?search=Post+%22http%3A%2F%2Fdummy%2Fcni%22%3A+EOF&maxAge=48h&context=1&type=bug%2Bissue%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job]
 
 Some of these will show as flakes as the test gets retried at times and then passes.
 
 Additionally in 4.14 we are seeing similar failures reporting
 {code:none}
 No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?{code}
 [4.14.0-0.nightly-2023-10-12-015817|https://amd64.ocp.releases.ci.openshift.org/releasestream/4.14.0-0.nightly/release/4.14.0-0.nightly-2023-10-12-015817] show pod sandbox errors for [azure|https://sippy.dptools.openshift.org/sippy-ng/tests/4.14/analysis?filters=%257B%2522items%2522%253A%255B%257B%2522columnField%2522%253A%2522name%2522%252C%2522operatorValue%2522%253A%2522equals%2522%252C%2522value%2522%253A%2522%255Bsig-network%255D%2520pods%2520should%2520successfully%2520create%2520sandboxes%2520by%2520other%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522never-stable%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522aggregated%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522azure%2522%257D%255D%252C%2522linkOperator%2522%253A%2522and%2522%257D&test=%5Bsig-network%5D%20pods%20should%20successfully%20create%20sandboxes%20by%20other]  &  [aws|https://sippy.dptools.openshift.org/sippy-ng/tests/4.14/analysis?filters=%257B%2522items%2522%253A%255B%257B%2522columnField%2522%253A%2522name%2522%252C%2522operatorValue%2522%253A%2522equals%2522%252C%2522value%2522%253A%2522%255Bsig-network%255D%2520pods%2520should%2520successfully%2520create%2520sandboxes%2520by%2520other%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522never-stable%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522aggregated%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522aws%2522%257D%255D%252C%2522linkOperator%2522%253A%2522and%2522%257D&test=%5Bsig-network%5D%20pods%20should%20successfully%20create%20sandboxes%20by%20other]  both show a drop from the 10th which comes after our force accept
 
 [4.14.0-0.nightly-2023-10-11-141212|https://amd64.ocp.releases.ci.openshift.org/releasestream/4.14.0-0.nightly/release/4.14.0-0.nightly-2023-10-11-141212]  had a host of failures but it is what killed [aws sdn|https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/aggregated-aws-sdn-upgrade-4.14-micro-release-openshift-release-analysis-aggregator/1712109617682583552]
 
 [4.14.0-0.nightly-2023-10-11-200059|https://amd64.ocp.releases.ci.openshift.org/releasestream/4.14.0-0.nightly/release/4.14.0-0.nightly-2023-10-11-200059] [aws sdn|https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/aggregated-aws-sdn-upgrade-4.14-micro-release-openshift-release-analysis-aggregator/1712197385490272256]  as well and shows in [azure|https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/aggregated-azure-ovn-upgrade-4.14-micro-release-openshift-release-analysis-aggregator/1712197406507929600]
Status: MODIFIED

Issue 16573555: Failed to create the sandbox-plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": EOF [Openshift-4.14]
Description: Description of problem:{code:none}
 In the 4.14 z-stream rollback job, I'm seeing test-case "[sig-network] pods should successfully create sandboxes by adding pod to network " fail. 
 
 The job link is here https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-upgrade-rollback-oldest-supported/1719037590788640768
 
 The error is:
 
 56 failures to create the sandbox
 
 ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-48-75.us-east-2.compute.internal - 3314.57 seconds after deletion - reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_prometheus-k8s-1_openshift-monitoring_95d1a457-3e1b-4ae3-8b57-8023eec5937d_0(5b36bc12b2964e85bcdbe60b275d6a12ea68cb18b81f16622a6cb686270c4eb3): error adding pod openshift-monitoring_prometheus-k8s-1 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": EOF
 ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-48-75.us-east-2.compute.internal - 3321.57 seconds after deletion - reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_prometheus-k8s-1_openshift-monitoring_95d1a457-3e1b-4ae3-8b57-8023eec5937d_0(3cc0afc5bec362566e4c3bdaf822209377102c2e39aaa8ef5d99b0f4ba795aaf): error adding pod openshift-monitoring_prometheus-k8s-1 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): failed to send CNI request: Post "http://dummy/cni": dial unix /run/multus/socket/multus.sock: connect: connection refused
 
 
 {code}
 Version-Release number of selected component (if applicable):{code:none}
 4.14.0-0.nightly-2023-10-30-170011
 {code}
 How reproducible:{code:none}
 Flaky
 {code}
 Steps to Reproduce:{code:none}
 1.
 2.
 3.
 {code}
 Actual results:{code:none}
 
 {code}
 Expected results:{code:none}
 
 {code}
 Additional info:{code:none}
 The rollback test is testing by installing 4.14.0, then upgrade to the latest 4.14.nightly, at some random point, rolling back to 4.14.0
 {code}
Status: Verified

Issue 17081653: "[sig-network] pods should successfully create sandboxes by adding pod to network" fails often on compact CI jobs
Description: This conformance test has been failing regularly in our CI:
 
 4.18 example: [https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.18-e2e-agent-compact-fips/1936031560633421824]
 
 4.17 example: [https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-agent-compact-fips/1934953258862776320]

Issue 17069730: Component Readiness: pods should successfully create sandboxes by adding pod to network: expected pod UID "aa853924-c6c6-45b7-be56-e059960bc3c6" but got "ab26e0dc-d736-4945-aa02-91fa3f066cdc" from Kube API
Description: (_Feel free to update this bug's summary to be more specific._)
 Component Readiness has found a potential regression in the following test:
 
 {code:none}[sig-network] pods should successfully create sandboxes by adding pod to network{code}
 
 Significant regression detected.
 Fishers Exact probability of a regression: 100.00%.
 Test pass rate dropped from 99.42% to 93.99%.
 
 Sample (being evaluated) Release: 4.19
 Start Time: 2025-06-10T00:00:00Z
 End Time: 2025-06-17T12:00:00Z
 Success Rate: 93.99%
 Successes: 212
 Failures: 14
 Flakes: 7
 
 Base (historical) Release: 4.17
 Start Time: 2024-09-01T00:00:00Z
 End Time: 2024-10-01T00:00:00Z
 Success Rate: 99.42%
 Successes: 803
 Failures: 5
 Flakes: 54
 
 View the [test details report|https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/test_details?Architecture=amd64&FeatureSet=default&Installer=ipi&Network=ovn&Platform=azure&Suite=unknown&Topology=ha&Upgrade=minor&baseEndTime=2025-02-25%2023%3A59%3A59&baseRelease=4.18&baseStartTime=2025-01-26%2000%3A00%3A00&capability=Other&columnGroupBy=Architecture%2CNetwork%2CPlatform%2CTopology&component=Networking%20%2F%20cluster-network-operator&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=Architecture%3Aamd64%20FeatureSet%3Adefault%20Installer%3Aipi%20Network%3Aovn%20Platform%3Aazure%20Suite%3Aunknown%20Topology%3Aha%20Upgrade%3Aminor&flakeAsFailure=false&ignoreDisruption=true&ignoreMissing=false&includeMultiReleaseAnalysis=true&includeVariant=Architecture%3Aamd64&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Acrun&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=FeatureSet%3Atechpreview&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=JobTier%3Ablocking&includeVariant=JobTier%3Ainforming&includeVariant=JobTier%3Astandard&includeVariant=LayeredProduct%3Anone&includeVariant=Network%3Aovn&includeVariant=Owner%3Aeng&includeVariant=Owner%3Aservice-delivery&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Arosa&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2025-06-17%2023%3A59%3A59&sampleRelease=4.19&sampleStartTime=2025-06-10%2000%3A00%3A00&testBasisRelease=4.17&testId=openshift-tests-upgrade%3A65e48733eb0b6115134b2b8c6a365f16&testName=%5Bsig-network%5D%20pods%20should%20successfully%20create%20sandboxes%20by%20adding%20pod%20to%20network] for additional context.
 
 Pretty clearly failing worse in the last week than 4.17 and 4.18 both.
Status: New

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
10 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
10 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
4 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
4 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
8 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
8 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
4 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
4 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
11 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
11 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
5 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
5 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
5 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
5 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
5 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
5 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
2 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
5 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
5 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
10 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
10 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
1 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
7 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
7 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
5 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
5 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox

# [sig-network] pods should successfully create sandboxes by adding pod to network
6 failures to create the sandbox