Job:
periodic-ci-openshift-release-master-nightly-4.12-upgrade-from-stable-4.11-e2e-metal-ipi-sdn-bm-upgrade (all) - 2 runs, 100% failed, 50% of failures match = 50% impact
#1792655696009891840junit4 hours ago
May 20 22:43:21.553 E ns/openshift-kube-apiserver pod/kube-apiserver-host3.cluster14.ocpci.eng.rdu2.redhat.com node/host3.cluster14.ocpci.eng.rdu2.redhat.com uid/52ac72b5-5492-4003-95d0-b9205a424106 container/kube-apiserver-cert-syncer reason/ContainerExit code/2 cause/Error rue} {user-serving-cert-008 true} {user-serving-cert-009 true}]\nI0520 22:42:51.768476       1 certsync_controller.go:66] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true} {control-plane-node-kubeconfig false} {check-endpoints-kubeconfig false}]\nI0520 22:42:51.768667       1 certsync_controller.go:170] Syncing secrets: [{aggregator-client false} {localhost-serving-cert-certkey false} {service-network-serving-certkey false} {external-loadbalancer-serving-certkey false} {internal-loadbalancer-serving-certkey false} {bound-service-account-signing-key false} {control-plane-node-admin-client-cert-key false} {check-endpoints-client-cert-key false} {kubelet-client false} {node-kubeconfigs false} {user-serving-cert true} {user-serving-cert-000 true} {user-serving-cert-001 true} {user-serving-cert-002 true} {user-serving-cert-003 true} {user-serving-cert-004 true} {user-serving-cert-005 true} {user-serving-cert-006 true} {user-serving-cert-007 true} {user-serving-cert-008 true} {user-serving-cert-009 true}]\nI0520 22:43:18.159779       1 certsync_controller.go:66] Syncing configmaps: [{aggregator-client-ca false} {client-ca false} {trusted-ca-bundle true} {control-plane-node-kubeconfig false} {check-endpoints-kubeconfig false}]\nI0520 22:43:18.160004       1 certsync_controller.go:170] Syncing secrets: [{aggregator-client false} {localhost-serving-cert-certkey false} {service-network-serving-certkey false} {external-loadbalancer-serving-certkey false} {internal-loadbalancer-serving-certkey false} {bound-service-account-signing-key false} {control-plane-node-admin-client-cert-key false} {check-endpoints-client-cert-key false} {kubelet-client false} {node-kubeconfigs false} {user-serving-cert true} {user-serving-cert-000 true} {user-serving-cert-001 true} {user-serving-cert-002 true} {user-serving-cert-003 true} {user-serving-cert-004 true} {user-serving-cert-005 true} {user-serving-cert-006 true} {user-serving-cert-007 true} {user-serving-cert-008 true} {user-serving-cert-009 true}]\n
May 20 22:43:21.580 E ns/openshift-sdn pod/sdn-controller-tzbb5 node/host3.cluster14.ocpci.eng.rdu2.redhat.com uid/ce27ac9d-a65a-442c-9f30-733698f70061 container/sdn-controller reason/ContainerExit code/2 cause/Error redhat.com became leader\nI0520 22:33:21.779006       1 master.go:56] Initializing SDN master\nE0520 22:33:21.779612       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:".17d15330bbc4f0b7", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ConfigMap", Namespace:"", Name:"", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"LeaderElection", Message:"host3.cluster14.ocpci.eng.rdu2.redhat.com became leader", Source:v1.EventSource{Component:"openshift-network-controller", Host:""}, FirstTimestamp:time.Date(2024, time.May, 20, 22, 33, 21, 778651319, time.Local), LastTimestamp:time.Date(2024, time.May, 20, 22, 33, 21, 778651319, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!)\nW0520 22:33:21.789365       1 master.go:155] Failed to list pods: pods is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "pods" in API group "" at the cluster scope\nW0520 22:33:21.790826       1 master.go:160] Failed to list services: services is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "services" in API group "" at the cluster scope\nI0520 22:33:21.790876       1 network_controller.go:56] Started OpenShift Network Controller\n
May 20 22:43:21.608 E ns/openshift-kube-scheduler pod/openshift-kube-scheduler-host3.cluster14.ocpci.eng.rdu2.redhat.com node/host3.cluster14.ocpci.eng.rdu2.redhat.com uid/a74930d4-478b-4826-bc6f-51d38b7f2443 container/kube-scheduler-cert-syncer reason/ContainerExit code/2 cause/Error 1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]\nI0520 22:41:16.330624       1 certsync_controller.go:66] Syncing configmaps: []\nI0520 22:41:16.330642       1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]\nI0520 22:41:35.971013       1 certsync_controller.go:66] Syncing configmaps: []\nI0520 22:41:35.971114       1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]\nI0520 22:41:42.345766       1 certsync_controller.go:66] Syncing configmaps: []\nI0520 22:41:42.345783       1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]\nI0520 22:42:01.988291       1 certsync_controller.go:66] Syncing configmaps: []\nI0520 22:42:01.988311       1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]\nI0520 22:42:08.360389       1 certsync_controller.go:66] Syncing configmaps: []\nI0520 22:42:08.360408       1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]\nI0520 22:42:27.996970       1 certsync_controller.go:66] Syncing configmaps: []\nI0520 22:42:27.996988       1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]\nI0520 22:42:34.371015       1 certsync_controller.go:66] Syncing configmaps: []\nI0520 22:42:34.371031       1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]\nI0520 22:42:54.006015       1 certsync_controller.go:66] Syncing configmaps: []\nI0520 22:42:54.006036       1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]\nI0520 22:43:00.973853       1 certsync_controller.go:66] Syncing configmaps: []\nI0520 22:43:00.973871       1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]\nI0520 22:43:20.016480       1 certsync_controller.go:66] Syncing configmaps: []\nI0520 22:43:20.016501       1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]\n
#1792655696009891840junit4 hours ago
May 20 22:48:06.836 E ns/openshift-machine-config-operator pod/machine-config-server-6bmfl node/host3.cluster14.ocpci.eng.rdu2.redhat.com uid/152f6de0-9333-4873-8915-a0f86e4b6ffd container/machine-config-server reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 20 22:48:07.915 E ns/openshift-sdn pod/sdn-controller-tzbb5 node/host3.cluster14.ocpci.eng.rdu2.redhat.com uid/ce27ac9d-a65a-442c-9f30-733698f70061 container/sdn-controller reason/ContainerExit code/2 cause/Error redhat.com became leader\nI0520 22:33:21.779006       1 master.go:56] Initializing SDN master\nE0520 22:33:21.779612       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:".17d15330bbc4f0b7", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ConfigMap", Namespace:"", Name:"", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"LeaderElection", Message:"host3.cluster14.ocpci.eng.rdu2.redhat.com became leader", Source:v1.EventSource{Component:"openshift-network-controller", Host:""}, FirstTimestamp:time.Date(2024, time.May, 20, 22, 33, 21, 778651319, time.Local), LastTimestamp:time.Date(2024, time.May, 20, 22, 33, 21, 778651319, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!)\nW0520 22:33:21.789365       1 master.go:155] Failed to list pods: pods is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "pods" in API group "" at the cluster scope\nW0520 22:33:21.790826       1 master.go:160] Failed to list services: services is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "services" in API group "" at the cluster scope\nI0520 22:33:21.790876       1 network_controller.go:56] Started OpenShift Network Controller\n
May 20 22:48:08.834 E ns/openshift-cluster-node-tuning-operator pod/tuned-5dmxl node/host3.cluster14.ocpci.eng.rdu2.redhat.com uid/30732e3e-d749-42c3-aba9-b8ccd88dc77b container/tuned reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
#1792655696009891840junit4 hours ago
E0520 22:33:21.779612       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:".17d15330bbc4f0b7", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ConfigMap", Namespace:"", Name:"", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"LeaderElection", Message:"host3.cluster14.ocpci.eng.rdu2.redhat.com became leader", Source:v1.EventSource{Component:"openshift-network-controller", Host:""}, FirstTimestamp:time.Date(2024, time.May, 20, 22, 33, 21, 778651319, time.Local), LastTimestamp:time.Date(2024, time.May, 20, 22, 33, 21, 778651319, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!)
W0520 22:33:21.789365       1 master.go:155] Failed to list pods: pods is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "pods" in API group "" at the cluster scope
W0520 22:33:21.790826       1 master.go:160] Failed to list services: services is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "services" in API group "" at the cluster scope
I0520 22:33:21.790876       1 network_controller.go:56] Started OpenShift Network Controller
periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-from-stable-4.14-e2e-aws-sdn-upgrade (all) - 1 runs, 0% failed, 100% of runs match
#1792649354545205248junit4 hours ago
I0520 21:44:44.414203       1 master.go:56] Initializing SDN master
W0520 21:44:44.429047       1 master.go:156] Failed to list pods: pods is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "pods" in API group "" at the cluster scope
W0520 21:44:44.429990       1 master.go:161] Failed to list services: services is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "services" in API group "" at the cluster scope
I0520 21:44:44.430026       1 network_controller.go:57] Started OpenShift Network Controller
pull-ci-openshift-cluster-network-operator-master-e2e-aws-live-migration-sdn-ovn-rollback (all) - 3 runs, 100% failed, 67% of failures match = 67% impact
#1792588138397306880junit8 hours ago
I0520 18:24:28.735026       1 master.go:56] Initializing SDN master
W0520 18:24:28.775716       1 master.go:156] Failed to list pods: pods is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "pods" in API group "" at the cluster scope
W0520 18:24:28.776865       1 master.go:161] Failed to list services: services is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "services" in API group "" at the cluster scope
I0520 18:24:28.776903       1 network_controller.go:57] Started OpenShift Network Controller
#1792588138397306880junit8 hours ago
I0520 17:37:03.268977       1 master.go:56] Initializing SDN master
W0520 17:37:03.344724       1 master.go:156] Failed to list pods: pods is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "pods" in API group "" at the cluster scope
W0520 17:37:03.346434       1 master.go:161] Failed to list services: services is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "services" in API group "" at the cluster scope
I0520 17:37:03.346471       1 network_controller.go:57] Started OpenShift Network Controller
#1792571341191254016junit9 hours ago
I0520 17:18:30.749612       1 master.go:56] Initializing SDN master
W0520 17:18:30.769038       1 master.go:156] Failed to list pods: pods is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "pods" in API group "" at the cluster scope
W0520 17:18:30.771683       1 master.go:161] Failed to list services: services is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "services" in API group "" at the cluster scope
I0520 17:18:30.771745       1 network_controller.go:57] Started OpenShift Network Controller
#1792571341191254016junit9 hours ago
I0520 16:33:01.288202       1 master.go:56] Initializing SDN master
W0520 16:33:01.298282       1 master.go:156] Failed to list pods: pods is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "pods" in API group "" at the cluster scope
W0520 16:33:01.299125       1 master.go:161] Failed to list services: services is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "services" in API group "" at the cluster scope
I0520 16:33:01.299169       1 network_controller.go:57] Started OpenShift Network Controller
pull-ci-openshift-cluster-network-operator-master-e2e-aws-live-migration-sdn-ovn (all) - 4 runs, 25% failed, 300% of failures match = 75% impact
#1792571338657894400junit10 hours ago
I0520 16:26:46.622766       1 master.go:56] Initializing SDN master
W0520 16:26:46.661287       1 master.go:156] Failed to list pods: pods is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "pods" in API group "" at the cluster scope
W0520 16:26:46.663362       1 master.go:161] Failed to list services: services is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "services" in API group "" at the cluster scope
I0520 16:26:46.663408       1 network_controller.go:57] Started OpenShift Network Controller
#1792567343117766656junit10 hours ago
I0520 16:12:23.575913       1 master.go:56] Initializing SDN master
W0520 16:12:23.599370       1 master.go:156] Failed to list pods: pods is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "pods" in API group "" at the cluster scope
W0520 16:12:23.601337       1 master.go:161] Failed to list services: services is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "services" in API group "" at the cluster scope
I0520 16:12:23.601493       1 network_controller.go:57] Started OpenShift Network Controller
#1792493552836546560junit15 hours ago
I0520 11:27:20.206736       1 master.go:56] Initializing SDN master
W0520 11:27:20.226374       1 master.go:156] Failed to list pods: pods is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "pods" in API group "" at the cluster scope
W0520 11:27:20.228693       1 master.go:161] Failed to list services: services is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "services" in API group "" at the cluster scope
I0520 11:27:20.228819       1 network_controller.go:57] Started OpenShift Network Controller
pull-ci-openshift-ovn-kubernetes-release-4.16-e2e-aws-live-migration-sdn-ovn (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
#1792555283466162176junit11 hours ago
cause/Error code/2 reason/ContainerExit 15:23:39.907359       1 master.go:56] Initializing SDN master
W0520 15:23:39.954356       1 master.go:156] Failed to list pods: pods is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "pods" in API group "" at the cluster scope
W0520 15:23:39.956006       1 master.go:161] Failed to list services: services is forbidden: User "system:serviceaccount:openshift-sdn:sdn-controller" cannot list resource "services" in API group "" at the cluster scope
I0520 15:23:39.956105       1 network_controller.go:57] Started OpenShift Network Controller
periodic-ci-openshift-release-master-ci-4.17-upgrade-from-stable-4.16-e2e-azure-ovn-upgrade (all) - 80 runs, 30% failed, 4% of failures match = 1% impact
#1792453507706523648junit16 hours ago
E0520 08:12:39.466973       1 main.go:93] Failed to list v1 volumesnapshots with error=Get "https://172.30.0.1:443/apis/snapshot.storage.k8s.io/v1/volumesnapshots": dial tcp 172.30.0.1:443: connect: connection refused
E0520 08:12:48.698255       1 main.go:93] Failed to list v1 volumesnapshots with error=volumesnapshots.snapshot.storage.k8s.io is forbidden: User "system:serviceaccount:openshift-cluster-storage-operator:csi-snapshot-controller" cannot list resource "volumesnapshots" in API group "snapshot.storage.k8s.io" at the cluster scope
I0520 08:12:53.984235       1 leaderelection.go:250] attempting to acquire leader lease openshift-cluster-storage-operator/snapshot-controller-leader...

Found in 0.03% of runs (0.16% of failures) across 28975 total runs and 4435 jobs (18.95% failed) in 1.064s - clear search | chart view - source code located on github