Job:
periodic-ci-openshift-release-master-ci-4.9-upgrade-from-stable-4.8-e2e-aws-ovn-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
#1780922810433015808build-log.txt.gz8 hours ago
Apr 18 12:41:50.000 I ns/openshift-marketplace pod/redhat-operators-vxb5k node/ip-10-0-170-231.ec2.internal container/registry-server reason/Pulling image/registry.redhat.io/redhat/redhat-operator-index:v4.8
Apr 18 12:41:50.000 I ns/openshift-marketplace pod/redhat-operators-vxb5k reason/AddedInterface Add eth0 [10.131.0.41/23] from ovn-kubernetes
Apr 18 12:41:51.000 I ns/openshift-marketplace pod/redhat-operators-vxb5k node/ip-10-0-170-231.ec2.internal container/registry-server reason/Created
Apr 18 12:41:51.000 I ns/openshift-marketplace pod/redhat-operators-vxb5k node/ip-10-0-170-231.ec2.internal container/registry-server reason/Started
Apr 18 12:41:51.922 I ns/openshift-marketplace pod/redhat-operators-vxb5k node/ip-10-0-170-231.ec2.internal container/registry-server reason/ContainerStart duration/4.00s
Apr 18 12:41:55.000 I ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-186-171.ec2.internal node/ip-10-0-186-171 reason/TerminationMinimalShutdownDurationFinished The minimal shutdown duration of 3m30s finished
Apr 18 12:41:55.000 I ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-186-171.ec2.internal node/ip-10-0-186-171 reason/TerminationStoppedServing Server has stopped listening
Apr 18 12:41:57.000 I ns/openshift-marketplace pod/redhat-operators-vxb5k node/ip-10-0-170-231.ec2.internal container/registry-server reason/Killing
Apr 18 12:41:57.770 I ns/openshift-marketplace pod/redhat-operators-vxb5k node/ip-10-0-170-231.ec2.internal container/registry-server reason/Ready
Apr 18 12:41:57.784 I ns/openshift-marketplace pod/redhat-operators-vxb5k node/ip-10-0-170-231.ec2.internal reason/GracefulDelete duration/1s
Apr 18 12:41:58.937 I ns/openshift-marketplace pod/redhat-operators-vxb5k node/ip-10-0-170-231.ec2.internal container/registry-server reason/ContainerExit code/0 cause/Completed
#1780922810433015808build-log.txt.gz8 hours ago
Apr 18 12:46:26.536 I ns/openshift-marketplace pod/certified-operators-b54bw node/ip-10-0-170-231.ec2.internal reason/GracefulDelete duration/1s
Apr 18 12:46:27.554 I ns/openshift-marketplace pod/certified-operators-b54bw node/ip-10-0-170-231.ec2.internal container/registry-server reason/ContainerExit code/0 cause/Completed
Apr 18 12:46:39.138 I ns/openshift-marketplace pod/certified-operators-b54bw node/ip-10-0-170-231.ec2.internal reason/Deleted
Apr 18 12:47:19.000 I ns/openshift-kube-apiserver-operator deployment/kube-apiserver-operator reason/MultipleVersions multiple versions found, probably in transition: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c05098097979f753c3ce1d97baf2a0b2bac7f8edf0a003bd7e9357d4e88e04c,registry.ci.openshift.org/ocp/4.9-2023-09-10-173339@sha256:528f7053c41381399923f69e230671c2e85d23202b2390877f14e8e0c10dea58 (17 times)
Apr 18 12:47:21.000 I ns/openshift-kube-apiserver-operator deployment/kube-apiserver-operator reason/MultipleVersions multiple versions found, probably in transition: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c05098097979f753c3ce1d97baf2a0b2bac7f8edf0a003bd7e9357d4e88e04c,registry.ci.openshift.org/ocp/4.9-2023-09-10-173339@sha256:528f7053c41381399923f69e230671c2e85d23202b2390877f14e8e0c10dea58 (18 times)
Apr 18 12:47:46.000 I ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-176-20.ec2.internal node/ip-10-0-176-20 reason/TerminationMinimalShutdownDurationFinished The minimal shutdown duration of 3m30s finished
Apr 18 12:47:46.000 I ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-176-20.ec2.internal node/ip-10-0-176-20 reason/TerminationStoppedServing Server has stopped listening
Apr 18 12:48:15.000 I ns/openshift-kube-apiserver-operator deployment/kube-apiserver-operator reason/MultipleVersions multiple versions found, probably in transition: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c05098097979f753c3ce1d97baf2a0b2bac7f8edf0a003bd7e9357d4e88e04c,registry.ci.openshift.org/ocp/4.9-2023-09-10-173339@sha256:528f7053c41381399923f69e230671c2e85d23202b2390877f14e8e0c10dea58 (19 times)
Apr 18 12:48:27.000 I ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-176-20.ec2.internal node/ip-10-0-176-20.ec2.internal container/kube-scheduler-recovery-controller reason/Created
Apr 18 12:48:27.000 I ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-176-20.ec2.internal node/ip-10-0-176-20.ec2.internal container/kube-scheduler-recovery-controller reason/Pulled image/quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:258478726c54a7e6ea106d65b04dbd5b25a6d0c5427dad81ad2bcf16eb4d6a82
Apr 18 12:48:27.000 I ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-176-20.ec2.internal node/ip-10-0-176-20.ec2.internal container/kube-scheduler-recovery-controller reason/Started
#1780922810433015808build-log.txt.gz8 hours ago
Apr 18 12:53:03.642 I ns/openshift-marketplace pod/redhat-marketplace-btt9h node/ip-10-0-170-231.ec2.internal reason/GracefulDelete duration/1s
Apr 18 12:53:04.000 I ns/openshift-marketplace pod/redhat-marketplace-btt9h node/ip-10-0-170-231.ec2.internal container/registry-server reason/Killing
Apr 18 12:53:05.420 I ns/openshift-marketplace pod/redhat-marketplace-btt9h node/ip-10-0-170-231.ec2.internal container/registry-server reason/ContainerExit code/0 cause/Completed
Apr 18 12:53:09.139 I ns/openshift-marketplace pod/redhat-marketplace-btt9h node/ip-10-0-170-231.ec2.internal reason/Deleted
Apr 18 12:53:15.000 I ns/openshift-kube-apiserver-operator deployment/kube-apiserver-operator reason/MultipleVersions multiple versions found, probably in transition: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c05098097979f753c3ce1d97baf2a0b2bac7f8edf0a003bd7e9357d4e88e04c,registry.ci.openshift.org/ocp/4.9-2023-09-10-173339@sha256:528f7053c41381399923f69e230671c2e85d23202b2390877f14e8e0c10dea58 (37 times)
Apr 18 12:53:37.000 I ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-230-231.ec2.internal node/ip-10-0-230-231 reason/TerminationMinimalShutdownDurationFinished The minimal shutdown duration of 3m30s finished
Apr 18 12:53:37.000 I ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-230-231.ec2.internal node/ip-10-0-230-231 reason/TerminationStoppedServing Server has stopped listening
Apr 18 12:54:15.000 I ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-230-231.ec2.internal node/ip-10-0-230-231.ec2.internal container/kube-scheduler-recovery-controller reason/Created
Apr 18 12:54:15.000 I ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-230-231.ec2.internal node/ip-10-0-230-231.ec2.internal container/kube-scheduler-recovery-controller reason/Pulled image/quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:258478726c54a7e6ea106d65b04dbd5b25a6d0c5427dad81ad2bcf16eb4d6a82
Apr 18 12:54:15.000 I ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-230-231.ec2.internal node/ip-10-0-230-231.ec2.internal container/kube-scheduler-recovery-controller reason/Started
Apr 18 12:54:15.000 I ns/openshift-kube-apiserver-operator deployment/kube-apiserver-operator reason/MultipleVersions multiple versions found, probably in transition: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c05098097979f753c3ce1d97baf2a0b2bac7f8edf0a003bd7e9357d4e88e04c,registry.ci.openshift.org/ocp/4.9-2023-09-10-173339@sha256:528f7053c41381399923f69e230671c2e85d23202b2390877f14e8e0c10dea58 (38 times)
#1780922810433015808build-log.txt.gz8 hours ago
Apr 18 13:02:32.906 W ns/openshift-apiserver pod/apiserver-66f4f6f5f-dhjsz reason/FailedScheduling skip schedule deleting pod: openshift-apiserver/apiserver-66f4f6f5f-dhjsz
Apr 18 13:02:32.927 I ns/openshift-apiserver pod/apiserver-66f4f6f5f-dhjsz node/ reason/DeletedBeforeScheduling
Apr 18 13:02:32.951 W ns/openshift-apiserver pod/apiserver-7b7996cc44-vbmqr reason/FailedScheduling 0/6 nodes are available: 3 node(s) didn't match Pod's node affinity/selector, 3 node(s) didn't match pod anti-affinity rules.
Apr 18 13:02:32.952 I ns/openshift-apiserver pod/apiserver-7b7996cc44-vbmqr node/ reason/Created
Apr 18 13:02:36.000 I ns/openshift-apiserver-operator deployment/openshift-apiserver-operator reason/OperatorStatusChanged Status for clusteroperator/openshift-apiserver changed: Progressing message changed from "APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: observed generation is 6, desired generation is 7." to "APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: 1/3 pods have been updated to the latest generation"
Apr 18 13:02:39.000 I ns/openshift-apiserver pod/apiserver-75cf5d87fc-lsmln node/apiserver-75cf5d87fc-lsmln reason/TerminationMinimalShutdownDurationFinished The minimal shutdown duration of 10s finished
Apr 18 13:02:39.000 I ns/openshift-apiserver pod/apiserver-75cf5d87fc-lsmln node/apiserver-75cf5d87fc-lsmln reason/TerminationStoppedServing Server has stopped listening
Apr 18 13:02:48.000 W ns/openshift-apiserver pod/apiserver-75cf5d87fc-lsmln node/ip-10-0-176-20.ec2.internal reason/ProbeError Liveness probe error: Get "https://10.128.0.48:8443/healthz": dial tcp 10.128.0.48:8443: connect: connection refused\nbody: \n
Apr 18 13:02:48.000 W ns/openshift-apiserver pod/apiserver-75cf5d87fc-lsmln node/ip-10-0-176-20.ec2.internal reason/ProbeError Readiness probe error: Get "https://10.128.0.48:8443/healthz": dial tcp 10.128.0.48:8443: connect: connection refused\nbody: \n
Apr 18 13:02:48.000 W ns/openshift-apiserver pod/apiserver-75cf5d87fc-lsmln node/ip-10-0-176-20.ec2.internal reason/Unhealthy Liveness probe failed: Get "https://10.128.0.48:8443/healthz": dial tcp 10.128.0.48:8443: connect: connection refused
Apr 18 13:02:48.000 W ns/openshift-apiserver pod/apiserver-75cf5d87fc-lsmln node/ip-10-0-176-20.ec2.internal reason/Unhealthy Readiness probe failed: Get "https://10.128.0.48:8443/healthz": dial tcp 10.128.0.48:8443: connect: connection refused
#1780922810433015808build-log.txt.gz8 hours ago
Apr 18 13:04:09.000 I ns/openshift-apiserver pod/apiserver-75cf5d87fc-rj7zp node/apiserver-75cf5d87fc-rj7zp reason/TerminationStart Received signal to terminate, becoming unready, but keeping serving
Apr 18 13:04:09.033 I ns/openshift-apiserver pod/apiserver-75cf5d87fc-rj7zp node/ip-10-0-230-231.ec2.internal reason/GracefulDelete duration/70s
Apr 18 13:04:09.104 I ns/openshift-apiserver pod/apiserver-7b7996cc44-trd5p node/ reason/Created
Apr 18 13:04:09.105 W ns/openshift-apiserver pod/apiserver-7b7996cc44-trd5p reason/FailedScheduling 0/6 nodes are available: 3 node(s) didn't match Pod's node affinity/selector, 3 node(s) didn't match pod anti-affinity rules.
Apr 18 13:04:10.000 I ns/openshift-apiserver-operator deployment/openshift-apiserver-operator reason/OperatorStatusChanged Status for clusteroperator/openshift-apiserver changed: Progressing message changed from "APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: 1/3 pods have been updated to the latest generation" to "APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: 2/3 pods have been updated to the latest generation"
Apr 18 13:04:19.000 I ns/openshift-apiserver pod/apiserver-75cf5d87fc-rj7zp node/apiserver-75cf5d87fc-rj7zp reason/TerminationMinimalShutdownDurationFinished The minimal shutdown duration of 10s finished
Apr 18 13:04:19.000 I ns/openshift-apiserver pod/apiserver-75cf5d87fc-rj7zp node/apiserver-75cf5d87fc-rj7zp reason/TerminationStoppedServing Server has stopped listening
Apr 18 13:04:20.000 W ns/openshift-apiserver pod/apiserver-75cf5d87fc-rj7zp node/ip-10-0-230-231.ec2.internal reason/ProbeError Liveness probe error: Get "https://10.129.0.43:8443/healthz": dial tcp 10.129.0.43:8443: connect: connection refused\nbody: \n
Apr 18 13:04:20.000 W ns/openshift-apiserver pod/apiserver-75cf5d87fc-rj7zp node/ip-10-0-230-231.ec2.internal reason/ProbeError Readiness probe error: Get "https://10.129.0.43:8443/healthz": dial tcp 10.129.0.43:8443: connect: connection refused\nbody: \n
Apr 18 13:04:20.000 W ns/openshift-apiserver pod/apiserver-75cf5d87fc-rj7zp node/ip-10-0-230-231.ec2.internal reason/Unhealthy Liveness probe failed: Get "https://10.129.0.43:8443/healthz": dial tcp 10.129.0.43:8443: connect: connection refused
Apr 18 13:04:20.000 W ns/openshift-apiserver pod/apiserver-75cf5d87fc-rj7zp node/ip-10-0-230-231.ec2.internal reason/Unhealthy Readiness probe failed: Get "https://10.129.0.43:8443/healthz": dial tcp 10.129.0.43:8443: connect: connection refused
#1780922810433015808build-log.txt.gz8 hours ago
Apr 18 13:05:41.215 I ns/openshift-apiserver pod/apiserver-75cf5d87fc-d9x44 node/ip-10-0-186-171.ec2.internal reason/GracefulDelete duration/70s
Apr 18 13:05:41.430 W ns/openshift-apiserver pod/apiserver-7b7996cc44-b9b5t reason/FailedScheduling 0/6 nodes are available: 3 node(s) didn't match Pod's node affinity/selector, 3 node(s) didn't match pod anti-affinity rules.
Apr 18 13:05:41.431 I ns/openshift-apiserver pod/apiserver-7b7996cc44-b9b5t node/ reason/Created
Apr 18 13:05:42.000 I ns/openshift-apiserver-operator deployment/openshift-apiserver-operator reason/OperatorStatusChanged Status for clusteroperator/openshift-apiserver changed: Degraded message changed from "All is well" to "APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()",Progressing changed from True to False ("All is well")
Apr 18 13:05:42.785 W clusteroperator/openshift-apiserver condition/Progressing status/False reason/AsExpected changed: All is well
Apr 18 13:05:51.000 I ns/openshift-apiserver pod/apiserver-75cf5d87fc-d9x44 node/apiserver-75cf5d87fc-d9x44 reason/TerminationMinimalShutdownDurationFinished The minimal shutdown duration of 10s finished
Apr 18 13:05:51.000 I ns/openshift-apiserver pod/apiserver-75cf5d87fc-d9x44 node/apiserver-75cf5d87fc-d9x44 reason/TerminationStoppedServing Server has stopped listening
Apr 18 13:05:52.000 W ns/openshift-apiserver pod/apiserver-75cf5d87fc-d9x44 node/ip-10-0-186-171.ec2.internal reason/ProbeError Liveness probe error: Get "https://10.130.0.42:8443/healthz": dial tcp 10.130.0.42:8443: connect: connection refused\nbody: \n
Apr 18 13:05:52.000 W ns/openshift-apiserver pod/apiserver-75cf5d87fc-d9x44 node/ip-10-0-186-171.ec2.internal reason/ProbeError Readiness probe error: Get "https://10.130.0.42:8443/healthz": dial tcp 10.130.0.42:8443: connect: connection refused\nbody: \n
Apr 18 13:05:52.000 W ns/openshift-apiserver pod/apiserver-75cf5d87fc-d9x44 node/ip-10-0-186-171.ec2.internal reason/Unhealthy Liveness probe failed: Get "https://10.130.0.42:8443/healthz": dial tcp 10.130.0.42:8443: connect: connection refused
Apr 18 13:05:52.000 W ns/openshift-apiserver pod/apiserver-75cf5d87fc-d9x44 node/ip-10-0-186-171.ec2.internal reason/Unhealthy Readiness probe failed: Get "https://10.130.0.42:8443/healthz": dial tcp 10.130.0.42:8443: connect: connection refused

Found in 2.04% of runs (2.44% of failures) across 49 total runs and 43 jobs (83.67% failed) in 106ms - clear search | chart view - source code located on github