| #OCPBUGS-74135 | issue | 6 weeks ago | [4.17] oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9 POST |
Issue 17781273: [4.17] oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9
Description: This is a clone of issue OCPBUGS-73848. The following is the description of the original issue:
—
The oauth-proxy e2e-component tests fail on OCP 4.17 because RHEL 9 enforces fully qualified container image names ("short name mode").
The backend test image {{nginxdemos/nginx-hello:plain-text}} is rejected by Podman since it cannot determine which registry to pull from.
h2. Version-Release number
4.22
h2. How reproducible
Always
h2. Steps to Reproduce
Run the oauth-proxy e2e-component tests on OCP 4.22
The test creates a pod with a backend container using image {{nginxdemos/nginx-hello:plain-text}}
Observe the pod fails to start.
h2. Actual results
The backend container fails with:
{code:java}
ImageInspectError: short name mode is enforcing, but image name
nginxdemos/nginx-hello:plain-text returns ambiguous list
{code}
The pod never reaches Running state, and all e2e tests fail.
h2. Expected results
The e2e tests should pass. The backend container should start successfully using a fully qualified image reference.
h2. Additional info
_Root cause:_ RHEL 9 enforces "short name mode" for container images, requiring fully qualified registry paths.
_Fix:_ Replace {{nginxdemos/nginx-hello:plain-text}} with {{{}registry.k8s.io/e2e-test-images/agnhost:2.53{}}}, which is:
* Fully qualified (satisfies RHEL 9 short name policy)
* Multi-arch (supports ARM64)
* The standard test image adopted across OpenShift repositories ([openshift/origin PR #29672|https://github.com/openshift/origin/pull/29672])
* Mirrored in OpenShift CI infrastructure
_PR:_ [https://github.com/openshift/oauth-proxy/pull/343]
Status: POST
| |||
| #OCPBUGS-29085 | issue | 6 days ago | LoadBalancer ingressController not working with IPv6 and OVN-Octavia provider New |
Issue 15784478: LoadBalancer ingressController not working with IPv6 and OVN-Octavia provider
Description: Description of problem:
OCP4.15 running on top of OSP16.2 fails to configure an ipv6 ingress controller. The testcase that fails is:
{code}
make openstack-tests && KUBECONFIG=~/.kube/config ./openstack-tests run-test "[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should create a TCP OVN LoadBalancer when LoadBalancerService ingressController is created on Openshift"
{code}
and it should be run with below configuration applied:
{code}
$ oc get cm -n openshift-config cloud-provider-config -o json | jq -r .data.config
[Global]
secret-name = openstack-credentials
secret-namespace = kube-system
region = regionOne
ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem
[LoadBalancer]
floating-network-id = 663be8b6-09db-4be4-bb33-7fc87c0ec51a
lb-provider = ovn
lb-method = SOURCE_IP_PORT
subnet-id = 648feab4-be61-4f69-a2a7-f54a7bbe5375
create-monitor = False
monitor-delay = 10s
monitor-timeout = 10s
monitor-max-retries = 1
max-shared-lb = 2
{code}
It's observed an error in OVN-Octavia on the controller:
{code}
: 31592, 'pool_id': '1794b0ac-44c7-4450-abad-a0725bd8a84a', 'subnet_id': '648feab4-be61-4f69-a2a7-f54a7bbe5375', 'admin_state_up': True} request_handler /usr/lib/python3.6/site-packages/networking_ovn/octavia/ovn_
driver.py:510
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver [-] Exception occurred during creation of member: netaddr.core.AddrFormatError: invalid IPNetwork [fd2e:6f44:5dd8:c956::3b4]
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver Traceback (most recent call last):
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver File "/usr/lib/python3.6/site-packages/netaddr/strategy/ipv4.py", line 125, in str_to_int
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver return _struct.unpack('>I', _inet_pton(AF_INET, addr))[0]
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver OSError: illegal IP address string passed to inet_pton
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver During handling of the above exception, another exception occurred:
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver Traceback (most recent call last):
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver File "/usr/lib/python3.6/site-packages/netaddr/ip/__init__.py", line 311, in __init__
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver self._value = self._module.str_to_int(addr, flags)
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver File "/usr/lib/python3.6/site-packages/netaddr/strategy/ipv4.py", line 129, in str_to_int
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver raise AddrFormatError('%r is not a valid IPv4 address string!' % addr)
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver netaddr.core.AddrFormatError: '[fd2e:6f44:5dd8:c956::3b4]' is not a valid IPv4 address string!
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver During handling of the above exception, another exception occurred:
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver Traceback (most recent call last):
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver File "/usr/lib/python3.6/site-packages/netaddr/ip/__init__.py", line 791, in parse_ip_network
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver ip = IPAddress(val1, module.version, flags=INET_PTON)
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver File "/usr/lib/python3.6/site-packages/netaddr/ip/__init__.py", line 314, in __init__
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver % (addr, self._module.version))
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver netaddr.core.AddrFormatError: base address '[fd2e:6f44:5dd8:c956::3b4]' is not IPv4
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver During handling of the above exception, another exception occurred:
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver Traceback (most recent call last):
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver File "/usr/lib/python3.6/site-packages/netaddr/ip/__init__.py", line 928, in __init__
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver implicit_prefix, flags)
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver File "/usr/lib/python3.6/site-packages/netaddr/ip/__init__.py", line 795, in parse_ip_network
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver expanded_addr = _ipv4.expand_partial_address(val1)
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver File "/usr/lib/python3.6/site-packages/netaddr/strategy/ipv4.py", line 259, in expand_partial_address
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver raise error
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver netaddr.core.AddrFormatError: invalid partial IPv4 address: '[fd2e:6f44:5dd8:c956::3b4]'!
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver During handling of the above exception, another exception occurred:
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver Traceback (most recent call last):
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver File "/usr/lib/python3.6/site-packages/networking_ovn/octavia/ovn_driver.py", line 1883, in member_create
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver self._add_member(member, ovn_lb, pool_key)
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver File "/usr/lib/python3.6/site-packages/networking_ovn/octavia/ovn_driver.py", line 1841, in _add_member
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver self._refresh_lb_vips(ovn_lb.uuid, external_ids)
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver File "/usr/lib/python3.6/site-packages/networking_ovn/octavia/ovn_driver.py", line 1051, in _refresh_lb_vips
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver vip_ips = self._frame_vip_ips(lb_external_ids)
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver File "/usr/lib/python3.6/site-packages/networking_ovn/octavia/ovn_driver.py", line 1039, in _frame_vip_ips
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver if netaddr.IPNetwork(lb_vip).version == 6:
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver File "/usr/lib/python3.6/site-packages/netaddr/ip/__init__.py", line 938, in __init__
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver raise AddrFormatError('invalid IPNetwork %s' % addr)
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver netaddr.core.AddrFormatError: invalid IPNetwork [fd2e:6f44:5dd8:c956::3b4]
2024-02-06 15:28:23.072 13 ERROR networking_ovn.octavia.ovn_driver
{code}
However, this behaviour is only observed by the LB created through the openshift-ingress. Other tc that is creating a loadbalancer serviceType in ipv6 it's working fine, for example:
{code}
$ make openstack-tests && KUBECONFIG=~/.kube/config ./openstack-tests run-test "[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should create a UDP OVN LoadBalancer when a UDP svc with type:LoadBalancer is created on Openshift"
make: 'openstack-tests' is up to date.
Feb 6 09:29:12.460: INFO: Enabling in-tree volume drivers
Feb 6 09:29:12.954: INFO: configPath is now "/tmp/configfile48504311"
Feb 6 09:29:12.954: INFO: The user is now "e2e-test-openstack-shwmj-user"
Feb 6 09:29:12.954: INFO: Creating project "e2e-test-openstack-shwmj"
Feb 6 09:29:13.275: INFO: Waiting on permissions in project "e2e-test-openstack-shwmj" ...
Feb 6 09:29:13.504: INFO: Waiting for ServiceAccount "default" to be provisioned...
Feb 6 09:29:13.615: INFO: Waiting for ServiceAccount "deployer" to be provisioned...
Feb 6 09:29:13.736: INFO: Waiting for ServiceAccount "builder" to be provisioned...
Feb 6 09:29:13.851: INFO: Waiting for RoleBinding "system:image-pullers" to be provisioned...
Feb 6 09:29:13.864: INFO: Waiting for RoleBinding "system:image-builders" to be provisioned...
Feb 6 09:29:13.945: INFO: Waiting for RoleBinding "system:deployers" to be provisioned...
Feb 6 09:29:14.366: INFO: Project "e2e-test-openstack-shwmj" has been fully provisioned.
Feb 6 09:29:15.054: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:0, Replicas:0, UpdatedReplicas:0, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:0, Conditions:[]v1.DeploymentCondition(nil), CollisionCount:(*int32)(nil)}
Feb 6 09:29:17.064: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:2, UpdatedReplicas:2, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:2, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2024, time.February, 6, 9, 29, 15, 0, time.Local), LastTransitionTime:time.Date(2024, time.February, 6, 9, 29, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2024, time.February, 6, 9, 29, 15, 0, time.Local), LastTransitionTime:time.Date(2024, time.February, 6,
9, 29, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"lb-default-dep-749c7dd84d\" is progressing."}}, CollisionCount:(*int32)(nil)}
Feb 6 09:29:19.069: INFO: deployment status: v1.DeploymentStatus{ObservedGeneration:1, Replicas:2, UpdatedReplicas:2, ReadyReplicas:0, AvailableReplicas:0, UnavailableReplicas:2, Conditions:[]v1.DeploymentCondition{v1.DeploymentCondition{Type:"Available", Status:"False", LastUpdateTime:time.Date(2024, time.February, 6, 9, 29, 15, 0, time.Local), LastTransitionTime:time.Date(2024, time.February, 6, 9, 29, 15, 0, time.Local), Reason:"MinimumReplicasUnavailable", Message:"Deployment does not have minimum availability."}, v1.DeploymentCondition{Type:"Progressing", Status:"True", LastUpdateTime:time.Date(2024, time.February, 6, 9, 29, 15, 0, time.Local), LastTransitionTime:time.Date(2024, time.February, 6,
9, 29, 15, 0, time.Local), Reason:"ReplicaSetUpdated", Message:"ReplicaSet \"lb-default-dep-749c7dd84d\" is progressing."}}, CollisionCount:(*int32)(nil)}
Feb 6 09:29:21.245: INFO: Waiting up to 10m0s for service "lb-default-svc" to have a LoadBalancer
Feb 6 09:29:31.554: INFO: Waiting for 6 members matching status NO_MONITOR|ONLINE in the pool pool_0_kube_service_kubernetes_e2e-test-openstack-shwmj_lb-default-svc
Feb 6 09:29:47.352: INFO: Found expected number of NO_MONITOR|ONLINE members, checking again 5 seconds later to confirm that it is stable
Feb 6 09:29:52.872: INFO: Wait completed, NO_MONITOR|ONLINE members found: ["ostest-qvxqg-master-2" "ostest-qvxqg-master-1" "ostest-qvxqg-worker-0-fnk9m" "ostest-qvxqg-worker-0-njvpg" "ostest-qvxqg-master-0" "ostest-qvxqg-worker-0-lmv7g"]
Feb 6 09:29:53.240: INFO: Pods accessed after 100 requests:
map[lb-default-dep-749c7dd84d-6j6k8:48 lb-default-dep-749c7dd84d-kjvnn:52]
Feb 6 09:29:53.258: INFO: Checking if pod "lb-default-dep-749c7dd84d-6j6k8" has been accessed at least 30 number of times...
Feb 6 09:29:53.258: INFO: Checking if pod "lb-default-dep-749c7dd84d-kjvnn" has been accessed at least 30 number of times...
Feb 6 09:29:53.281: INFO: Deleted {user.openshift.io/v1, Resource=users e2e-test-openstack-shwmj-user}, err: <nil>
Feb 6 09:29:53.331: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthclients e2e-client-e2e-test-openstack-shwmj}, err: <nil>
{code}
Status: New
| |||
| #OCPBUGS-57501 | issue | 4 days ago | After the Cluster upgrade to 4.16.38 some of the iptables-alerter pod are in Create Container Error State - [runc]. ASSIGNED |
Issue 17067136: After the Cluster upgrade to 4.16.38 some of the iptables-alerter pod are in Create Container Error State - [runc].
Description: Description of problem:
{code:none}
--> After the Cluster upgrade to 4.16.38 some of the iptables-alerter pod are in Create Container Error State as shown:
pod/iptables-alerter-98mdm 0/1 CreateContainerError 0 23h
pod/iptables-alerter-bgzl7 1/1 Running 1 11d
pod/iptables-alerter-mpswn 0/1 CreateContainerError 0 5d
--> On seeing the pod yaml, the pods are only showing ContextDeadlineExceeded as shown:
containerStatuses:
- image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3c3d0b212909bc8049aa6fb9006c0990b664dd0ca5089da224e08ba909ce0a0b
imageID: ""
lastState: {}
name: iptables-alerter
ready: false
restartCount: 0
started: false
state:
waiting:
message: context deadline exceeded
reason: CreateContainerError
--> Similar error can be seen in the events as well:
LAST SEEN TYPE REASON OBJECT MESSAGE
5m Warning Failed pod/iptables-alerter-98mdm Error: context deadline exceeded
3m7s Warning Failed pod/iptables-alerter-mpswn Error: context deadline exceeded
--> Manually deleted the iptables-alerter pods, but no luck.
--> Manually restarted kubelet & crio service on the nodes, but no luck.
--> Pruned the nodes of the exited containers, no luck.
--> Hard & Soft reboots of the worker node from the VMware console, but no luck.
--> Also after wiping the CRIO for the node the issue persists.{code}
Version-Release number of selected component (if applicable):
{code:none}
4.16.38 {code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: ASSIGNED
| |||
| #OCPBUGS-50994 | issue | 7 weeks ago | GCP: Install failures due to inconsistent iam apply Verified |
Issue 16677256: GCP: Install failures due to inconsistent iam apply
Description: *Description of problem*. In Advanced Cluster Security we rely for OCP creation in our CI and recently observed [an increase of cluster creation failures|https://docs.google.com/document/d/1vE-kLPWooO75BXxGub6YZpM4NZolKBcOrvJULZ5NBw0/edit]. While we've been [advised to retry the failures|https://redhat-internal.slack.com/archives/C68TNFWA2/p1722873956362609] (and we do so now, see [ROX-25416|https://issues.redhat.com/browse/ROX-25416]), I'm afraid our use case is not so unique and others are affected as well.
We suggest upgrading terraform and provider to the latest version (possible before license changes) in openshift-installer for 4.12+. The underlying issue is [probably already fixed upstream|https://github.com/hashicorp/terraform-provider-google/issues/17332] and released in [v5.37.0|https://github.com/hashicorp/terraform-provider-google/releases/tag/v5.37.0].
Version-Release number of selected component (if applicable): TBD
How reproducible: TBD
Steps to Reproduce: TBD
Actual results: TBD
Expected results: TBD
h3. Additional info.
The most common error we see in our [JIRA issues|https://issues.redhat.com/browse/ROX-25724?jql=text%20~%20%22module.master.google_service_account.master-node-sa%22] is and that is something we could find similar issues with AWS provider too eg. OCPBUGS-4213.
{code}
level=error msg=Error: Provider produced inconsistent result after apply .... resource was present, but now absent
{code}
Summary of errors from:
{code}
3 failed to create cluster: failed to apply Terraform: error(GCPComputeBackendTimeout) from Infrastructure Provider: GCP is experiencing backend service interuptions, the compute instance failed to create in reasonable time."
3 Provider produced inconsistent result after apply\n\nWhen applying changes to\nmodule.master.google_service_account.master-node-sa[0], provider\n\"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new value: Root\nresource was present, but now absent.\n\n
6 Error waiting to create Network: Error waiting for Creating Network: timeout while waiting for state to become 'DONE' (last state: 'RUNNING', timeout: 4m0s)\n\n with module.network.google_compute_network.cluster_network[0],\n on network/network.tf line 1, in resource \"google_compute_network\" \"cluster_network\":\n 1: resource \"google_compute_network\" \"cluster_network\" {\n\n"
9 error applying Terraform configs: failed to apply Terraform: error(GCPComputeBackendTimeout) from Infrastructure Provider: GCP is experiencing backend service interuptions, the compute instance failed to create in reasonable time."
14 Provider produced inconsistent result after apply\n\nWhen applying changes to module.master.google_service_account.master-node-sa,\nprovider \"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new\nvalue: Root resource was present, but now absent.
16 Provider produced inconsistent result after apply\n\nWhen applying changes to google_service_account_key.bootstrap, provider\n\"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new value: Root\nresource was present, but now absent.
18 Provider produced inconsistent result after apply\n\nWhen applying changes to module.iam.google_service_account.worker-node-sa,\nprovider \"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new\nvalue: Root resource was present, but now absent.
34 Error creating service account key: googleapi: Error 404: Service account projects/acs-san-stackroxci/serviceAccounts/XXX@acs-san-stackroxci.iam.gserviceaccount.com does not exist., notFound\n\n with google_service_account_key.bootstrap,\n on main.tf line 38, in resource \"google_service_account_key\" \"bootstrap\":\n 38: resource \"google_service_account_key\" \"bootstrap\" {\n\n"
45 error applying Terraform configs: failed to apply Terraform: exit status 1\n\nError: Provider produced inconsistent result after apply\n\nWhen applying changes to\nmodule.master.google_service_account.master-node-sa[0], provider\n\"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new value: Root\nresource was present, but now absent.
59 error applying Terraform configs: failed to apply Terraform: exit status 1\n\nError: Provider produced inconsistent result after apply\n\nWhen applying changes to module.iam.google_service_account.worker-node-sa,\nprovider \"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new\nvalue: Root resource was present, but now absent.
100 Provider produced inconsistent result after apply\n\nWhen applying changes to google_service_account.bootstrap-node-sa, provider\n\"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new value: Root\nresource was present, but now absent.
103 Provider produced inconsistent result after apply\n\nWhen applying changes to module.iam.google_service_account.worker-node-sa[0],\nprovider \"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new\nvalue: Root resource was present, but now absent.
116 Provider produced inconsistent result after apply\n\nWhen applying changes to\nmodule.master.google_service_account.master-node-sa[0], provider\n\"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new value: Root\nresource was present, but now absent.
{code}
The openshift installer contains a bundled terraform and google-provider
* 4.12
** Terraform 1.0.11 (November 10, 2021)
** https://github.com/openshift/installer/blob/release-4.12/terraform/terraform/go.mod#L5
** Provider: v4.5.0 (Dec 20, 2021)
** https://github.com/openshift/installer/blob/release-4.12/terraform/providers/google/go.mod#L5
* 4.14+
** Terraform: 1.3.7 (January 04, 2023)
** https://github.com/openshift/installer/blob/release-4.14/terraform/terraform/go.mod#L5
** Provider: v4.74.0 (Jul 18, 2023)
** https://github.com/openshift/installer/blob/release-4.14/terraform/providers/google/go.mod#L5
* 4.18+
** Does not contain google provider at all: https://github.com/openshift/installer/pull/8723
Status: Verified
| |||
| #OCPBUGS-72360 | issue | 7 weeks ago | Ginkgo test failure: [Apiserver] MicroShift enable SCC admission for pods Verified |
Issue 17741919: Ginkgo test failure: [Apiserver] MicroShift enable SCC admission for pods
Description: Description of problem:
{code:none}
"[sig-api-machinery] API_Server on Microshift [It] Author:rgangwar-MicroShiftBoth-ConnectedOnly-Medium-55394-[Apiserver] MicroShift enable SCC admission for pods test" failed because the pod is created with a missing fsGroup
{code}
Version-Release number of selected component (if applicable):
{code:none}
4.21.0-rc.1 on x86. This is not reproducible on ARM architecture.{code}
How reproducible:
{code:none}
always on 4.21.0-rc.1{code}
Steps to Reproduce:
{code:none}
1. Run Ginkgo test case: "[sig-api-machinery] API_Server on Microshift [It] Author:rgangwar-MicroShiftBoth-ConnectedOnly-Medium-55394-[Apiserver] MicroShift enable SCC admission for pods test"
2.
3.
{code}
Actual results:
{code:none}
• [PANICKED] [222.518 seconds]
[sig-api-machinery] API_Server on Microshift [It] Author:rgangwar-MicroShiftBoth-ConnectedOnly-Medium-55394-[Apiserver] MicroShift enable SCC admission for pods
/tmp/tmp.I4ekJytyWF/openshift-tests-private/test/extended/apiserverauth/apiserver_microshift.go:480
Timeline >>
STEP: Creating a kubernetes client @ 01/08/26 04:07:05.353
I0108 04:07:05.368507 507110 client.go:486] Creating namespace "e2e-test-default-wb9lb"
I0108 04:07:05.380389 507110 client.go:491] Waiting for ServiceAccount "default" to be provisioned...
I0108 04:07:05.485060 507110 client.go:495] Configuring kubeconfig with user "e2e-test-default-wb9lb-user" certificates...
I0108 04:07:07.078970 507110 client.go:293] configPath is now "/tmp/configfile4051781145"
I0108 04:07:07.079012 507110 client.go:498] Waiting for RoleBinding "e2e-test-default-wb9lb-user" to be provisioned...
I0108 04:07:07.084506 507110 client.go:502] Waiting on permissions in namespace "e2e-test-default-wb9lb" ...
I0108 04:07:07.364467 507110 client.go:521] Namespace "e2e-test-default-wb9lb" has been fully provisioned.
I0108 04:07:07.364710 507110 apiserver_microshift.go:37] The cluster should be healthy before running case.
namespace/r6c171tq created
namespace "r6c171tq" deleted
I0108 04:07:13.076170 507110 apiserver_util.go:1892] Cluster sanity check passed
STEP: 1.Create temporary namespace @ 01/08/26 04:07:13.076
STEP: 2. Create one pod security-context-demo-ocp55394 with the specified security context. @ 01/08/26 04:07:13.183
I0108 04:07:13.183100 507110 apiserver_util.go:460] apiserver fixture dir is not initialized, start to create
I0108 04:07:13.187939 507110 apiserver_util.go:463] apiserver fixture dir is initialized: /tmp/fixture-testdata-dir2892104936/test/extended/testdata/apiserverauth
pod/security-context-demo-ocp55394 created
STEP: 3. Verify that the Pod's security-context-demo-ocp55394 Container is running @ 01/08/26 04:07:13.306
I0108 04:07:23.409925 507110 pods.go:176] Pod security-context-demo-ocp55394 is ready!
STEP: 4.1 Verify if the processes are running with the specified user ID 1000 in the pod. @ 01/08/26 04:07:23.41
I0108 04:07:38.587390 507110 apiserver_util.go:801] Attempting to execute command on pod security-context-demo-ocp55394. Output: PID USER TIME COMMAND
1 1000 0:00 sleep 1h
2 1000 0:00 ps, Error: <nil>
I0108 04:07:38.587444 507110 apiserver_util.go:832] Successfully retrieved non-empty output from pod security-context-demo-ocp55394: PID USER TIME COMMAND
1 1000 0:00 sleep 1h
2 1000 0:00 ps
I0108 04:07:38.587500 507110 apiserver_microshift.go:506] Processes are running on pod security-context-demo-ocp55394 with user id 1000 :: PID USER TIME COMMAND
1 1000 0:00 sleep 1h
2 1000 0:00 ps
STEP: 4.2 Verify that user is running with specified uid=1000 gid=3000 groups=2000 @ 01/08/26 04:07:38.587
I0108 04:07:53.775700 507110 apiserver_util.go:801] Attempting to execute command on pod security-context-demo-ocp55394. Output: uid=1000(1000) gid=3000 groups=2000,3000, Error: <nil>
I0108 04:07:53.775738 507110 apiserver_util.go:832] Successfully retrieved non-empty output from pod security-context-demo-ocp55394: uid=1000(1000) gid=3000 groups=2000,3000
I0108 04:07:53.775826 507110 apiserver_microshift.go:515] On pod security-context-demo-ocp55394 User running with :: uid=1000(1000) gid=3000 groups=2000,3000
STEP: 5. Create one pod security-context-demo-2-ocp55394 with the specified security context. @ 01/08/26 04:07:53.775
I0108 04:07:53.775904 507110 apiserver_util.go:466] apiserver fixture dir found in cache: /tmp/fixture-testdata-dir2892104936/test/extended/testdata/apiserverauth
pod/security-context-demo-2-ocp55394 created
STEP: 6. Verify that the Pod's security-context-demo-2-ocp55394 Container is running @ 01/08/26 04:07:53.967
I0108 04:08:04.065851 507110 pods.go:176] Pod security-context-demo-2-ocp55394 is ready!
STEP: 7. Verify that processes are running with the specified user ID 2000 in the pod. @ 01/08/26 04:08:04.066
I0108 04:08:19.255042 507110 apiserver_util.go:801] Attempting to execute command on pod security-context-demo-2-ocp55394. Output: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
2000 1 0.0 0.0 705960 768 ? Ssl 04:07 0:00 /hello_openshift
2000 6 0.0 0.0 1704 768 ? Rs 04:08 0:00 ps aux, Error: <nil>
I0108 04:08:19.255086 507110 apiserver_util.go:832] Successfully retrieved non-empty output from pod security-context-demo-2-ocp55394: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
2000 1 0.0 0.0 705960 768 ? Ssl 04:07 0:00 /hello_openshift
2000 6 0.0 0.0 1704 768 ? Rs 04:08 0:00 ps aux
I0108 04:08:19.255128 507110 apiserver_microshift.go:534] Processes are running on pod security-context-demo-2-ocp55394 with user id 2000 :: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
2000 1 0.0 0.0 705960 768 ? Ssl 04:07 0:00 /hello_openshift
2000 6 0.0 0.0 1704 768 ? Rs 04:08 0:00 ps aux
STEP: 8. Ensure that pods in different namespaces are launched with different UIDs. @ 01/08/26 04:08:19.255
STEP: 8.1 Create two different namespaces. @ 01/08/26 04:08:19.255
I0108 04:08:19.291544 507110 apiserver_util.go:466] apiserver fixture dir found in cache: /tmp/fixture-testdata-dir2892104936/test/extended/testdata/apiserverauth
STEP: 8.2 Create pods in both namespaces @ 01/08/26 04:08:19.291
STEP: 8.3 Verify pods should have different UID's and desc in both namespaces. @ 01/08/26 04:08:19.291
pod/pod-for-ping created
I0108 04:08:25.585028 507110 apiserver_util.go:1035] The resource returned:
pod-for-ping
I0108 04:08:25.585071 507110 apiserver_util.go:1263] Namespace testpod-namespace-1 pods are: pod-for-ping
I0108 04:08:35.699575 507110 pods.go:176] Pod pod-for-ping is ready!
I0108 04:08:41.978615 507110 apiserver_util.go:1035] The resource returned:
'pod-for-ping runAsUser: 1000580000 fsGroup: seLinuxOptions: s0:c24,c14
'
pod/pod-for-ping created
I0108 04:08:48.301631 507110 apiserver_util.go:1035] The resource returned:
pod-for-ping
I0108 04:08:48.301670 507110 apiserver_util.go:1263] Namespace testpod-namespace-2 pods are: pod-for-ping
I0108 04:08:58.471873 507110 pods.go:176] Pod pod-for-ping is ready!
I0108 04:09:06.403000 507110 apiserver_util.go:1035] The resource returned:
'pod-for-ping runAsUser: 1000590000 fsGroup: seLinuxOptions: s0:c24,c19
'
pod "security-context-demo-2-ocp55394" deleted from test-scc-ocp55394 namespace
pod "security-context-demo-ocp55394" deleted from test-scc-ocp55394 namespace
namespace "test-scc-ocp55394" deleted
[PANICKED] in [It] - /home/ec2-user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.11.linux-amd64/src/runtime/panic.go:115 @ 01/08/26 04:10:47.819
Jan 8 04:10:47.819: INFO: [1mtest dir /tmp/-OCP-microshift-apiseerver-cases-7zer0254/ is cleaned up[0m
STEP: Collecting events from namespace "e2e-test-default-wb9lb". @ 01/08/26 04:10:47.819
STEP: Found 0 events. @ 01/08/26 04:10:47.83
I0108 04:10:47.833385 507110 resource.go:151] POD NODE PHASE GRACE CONDITIONS
I0108 04:10:47.833421 507110 resource.go:161]
I0108 04:10:47.837817 507110 dump.go:81] skipping dumping cluster info - cluster too large
I0108 04:10:47.857338 507110 client.go:681] Deleted {certificates.k8s.io/v1, Resource=certificatesigningrequests e2e-test-default-wb9lb-user}, err: <nil>
STEP: Destroying namespace "e2e-test-default-wb9lb" for this suite. @ 01/08/26 04:10:47.857
<< Timeline
[PANICKED] Test Panicked
In [It] at: /home/ec2-user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.11.linux-amd64/src/runtime/panic.go:115 @ 01/08/26 04:10:47.819
runtime error: index out of range [6] with length 6
Full Stack Trace
github.com/openshift/openshift-tests-private/test/extended/apiserverauth.init.func3.8()
/tmp/tmp.I4ekJytyWF/openshift-tests-private/test/extended/apiserverauth/apiserver_microshift.go:590 +0x1bcf
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0x0?, 0x0?})
/home/ec2-user/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20240806135314-3946b2b7b2a8/internal/node.go:472 +0x13
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/home/ec2-user/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20240806135314-3946b2b7b2a8/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 1
/home/ec2-user/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20240806135314-3946b2b7b2a8/internal/suite.go:888 +0xd7b
------------------------------
Summarizing 1 Failure:
[PANICKED!] [sig-api-machinery] API_Server on Microshift [It] Author:rgangwar-MicroShiftBoth-ConnectedOnly-Medium-55394-[Apiserver] MicroShift enable SCC admission for pods
/home/ec2-user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.11.linux-amd64/src/runtime/panic.go:115
Ran 1 of 1 Specs in 222.518 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped{code}
Expected results:
{code:none}
test pass{code}
Additional info:
[ link to ginkgo log|https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-microshift-release-4.21-periodics-e2e-aws-tests-release-periodic/2009098095907835904/artifacts/e2e-aws-tests-release-periodic/openshift-microshift-e2e-metal-tests/artifacts/scenario-info/el96-lrel@ginkgo-tests/ginkgo-results/test-output.log]
{code:none}
{code}
Status: Verified
| |||
| #OCPBUGS-77196 | issue | 2 weeks ago | [openshift-4.16] ose-libvirt-machine-controllers can't be build hermetically New |
Issue 17925861: [openshift-4.16] ose-libvirt-machine-controllers can't be build hermetically
Description: Description of problem:
ART can't build ose-libvirt-machine-controllers hermetically.
The error is the following in the prefetch-dependencies konflux task:
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details. Please try running `go mod vendor` and committing the changes. Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-18387 | issue | 8 weeks ago | [CORS-2550] Installer should have pre-check for vm type, DES encryption type when install with Confidential VM ASSIGNED |
Issue 15452967: [CORS-2550] Installer should have pre-check for vm type, DES encryption type when install with Confidential VM
Description: Description of problem:
{code:none}
Install IPI cluster with confidential VM, installer should have pre-check for vm type, disk encryption type etc to avoid installation failed during infrastructure creation
1. vm type
Different security type support on different vm type
for example, set platfrom.azure.defaultMachinePlatform.type to Standard_DC8ads_v5 and platform.azure.defaultMachinePlatform.settings.securityType to TrustedLaunch, installation will be failed as Standard_DC8ads_v5 only support security type ConfidentialVM
ERROR Error: creating Linux Virtual Machine: (Name "jimaconf1-89qmp-bootstrap" / Resource Group "jimaconf1-89qmp-rg"): compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="BadRequest" Message="The VM size 'Standard_DC16ads_v5' is not supported for creation of VMs and Virtual Machine Scale Set with 'TrustedLaunch' security type."
2. Disk encryption Set
When install cluster with ConfidentialVM +securityEncryptionType:DiskWithVMGuestState, then using customer-managed key, it requires that DES encryption type is ConfidentialVmEncryptedWithCustomerKey, else installer throw error as below:
08-31 10:12:54.443 level=error msg=Error: creating Linux Virtual Machine: (Name "jima30confa-vtrm2-bootstrap" / Resource Group "jima30confa-vtrm2-rg"): compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="BadRequest" Message="The type of the Disk Encryption Set in the request is 'ConfidentialVmEncryptedWithCustomerKey', but this Disk Encryption Set was created with type 'EncryptionAtRestWithCustomerKey'." Target="/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/jima30confa-vtrm2-rg/providers/Microsoft.Compute/disks/jima30confa-vtrm2-bootstrap_OSDisk"
Installer should check vm type and DES's encryption type to make sure that expected DES is set.{code}
Version-Release number of selected component (if applicable):
{code:none}
4.14 nightly build{code}
How reproducible:
{code:none}
Always{code}
Steps to Reproduce:
{code:none}
1. Prepare install-config,
1) enable confidentialVM but use vm type which does not support Confidential VM
2) enable TrustedLaunch but use vm type which support confidentialVM
3) enable confidentialVM + securityEncryptionType: DiskWithVMGuestState, use customer-managed key to encrypt managed key, but customer-managed key's encryption type is the default one "EncryptionAtRestWithPlatformKey"
2. Create cluster
3.
{code}
Actual results:
{code:none}
Installation failed when creating infrastructure{code}
Expected results:
{code:none}
Installer should have pre-check for those scenarios, and exit with expected error message.{code}
Additional info:
{code:none}
{code}
Status: ASSIGNED
| |||
| #OCPBUGS-60159 | issue | 5 weeks ago | openshift-cluster-csi-drivers vmware-vsphere-csi-driver-operator-metrics endpoint 8445 port should return 401 instead 500 Verified |
Issue 17201621: openshift-cluster-csi-drivers vmware-vsphere-csi-driver-operator-metrics endpoint 8445 port should return 401 instead 500
Description: Description of problem:
see similar bug OCPBUGS-57585
Stole the testing result from [https://github.com/openshift/origin/pull/30014#issuecomment-3153130188]
8445 port is for openshift-cluster-csi-drivers vmware-vsphere-csi-driver-operator-metrics endpoint, did not see this port on other IAAS(example AWS,OSP)checked on vsphere, curl 8445 without auth returns 500(actually it's forbidden),
{noformat}
$ oc get infrastructure/cluster -o jsonpath={.spec.platformSpec.type}
VSphere
$ oc -n openshift-cluster-csi-drivers get ep
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME ENDPOINTS AGE
vmware-vsphere-csi-driver-controller-metrics 10.129.0.17:9203,10.130.0.21:9203,10.129.0.17:9202 + 9 more... 28m
vmware-vsphere-csi-driver-operator-metrics 10.130.0.17:8445 28m
vmware-vsphere-csi-driver-webhook-svc 10.128.0.45:8443,10.130.0.23:8443 28m
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -s -o /dev/null -w '%{http_code}' https://10.130.0.17:8445/metrics
500
{noformat}
return code is 500, expecting 401 forbidden instead of 500.
{code:java}
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k https://10.130.0.17:8445/metrics
Internal Server Error: "/metrics": subjectaccessreviews.authorization.k8s.io is forbidden: User "system:serviceaccount:openshift-cluster-csi-drivers:vmware-vsphere-csi-driver-operator" cannot create resource "subjectaccessreviews" in API group "authorization.k8s.io" at the cluster scope {code}
curl with auth, no issue
{noformat}
$ token=`oc create token prometheus-k8s -n openshift-monitoring`
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://10.130.0.17:8445/metrics' | head -n3
# HELP aggregator_discovery_aggregation_count_total [ALPHA] Counter of number of times discovery was aggregated
# TYPE aggregator_discovery_aggregation_count_total counter
aggregator_discovery_aggregation_count_total 0 {noformat}
[https://github.com/openshift/origin/pull/30014#issuecomment-3154900379]
When the bug is fixed, please remove it from this code snippet to avoid regression.
[https://github.com/openshift/origin/blob/4f183dd3427cffd8d97b44557caa782d65726416/test/extended/prometheus/prometheus.go#L64-L71]
Status: Verified
| |||
| #OCPBUGS-77198 | issue | 2 weeks ago | [openshift-4.14] ose-libvirt-machine-controllers can't be build hermetically New |
Issue 17925864: [openshift-4.14] ose-libvirt-machine-controllers can't be build hermetically
Description: Description of problem:
ART can't build ose-libvirt-machine-controllers hermetically.
The error is the following in the prefetch-dependencies konflux task:
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details. Please try running `go mod vendor` and committing the changes. Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77786 | issue | 6 days ago | [openshift-4.17] ose-sriov-rdma-cni can't be built hermetically New |
Issue 17955337: [openshift-4.17] ose-sriov-rdma-cni can't be built hermetically
Description: This is a clone of issue OCPBUGS-77775. The following is the description of the original issue: --- Description of problem:
ART can't build ose-sriov-rdma-cni hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77211 | issue | 4 days ago | [openshift-4.15] hermetic build failure - ose-ibm-vpc-block-csi-driver Verified |
Issue 17926179: [openshift-4.15] hermetic build failure - ose-ibm-vpc-block-csi-driver
Description: This is a clone of issue OCPBUGS-77210. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77209. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77207. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77206. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77205. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77203. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77167. The following is the description of the original issue: --- Description of problem:
{code:none}
IBM VPC Block CSI Driver build fails in hermetic mode due to non-deterministic GOPATH handling in Makefile. The build system rewrites $GOPATH causing the binary output path -o ${GOPATH}/bin/${EXE_DRIVER_NAME} to become invalid, resulting in build failures.{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift 4.19+ (affects hermetic build conversion) {code}
How reproducible:
{code:none}
Always - when building ibm-vpc-block-csi-driver in hermetic mode{code}
Steps to Reproduce:
{code:none}
1. Build ibm-vpc-block-csi-driver container in hermetic mode
2. Makefile uses absolute GOPATH-dependent output path: -o ${GOPATH}/bin/${EXE_DRIVER_NAME}
3. Build system rewrites $GOPATH during hermetic build process {code}
Actual results:
{code:none}
Build fails because binary cannot be found at expected GOPATH location after build system modifies $GOPATH
Error: building at STEP "COPY --from=builder /cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver /bin/ibm-vpc-block-csi-driver": checking on sources under [...]: copier: stat: "/cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver": no such file or directory{code}
Expected results:
{code:none}
Build succeeds using relative path bin/${EXE_DRIVER_NAME} that doesn't depend on GOPATH manipulation, consistent with other CSI drivers like aws-ebs-csi-driver {code}
Additional info:
{code:none}
Fix implemented: Change Makefile to use relative path -o bin/${EXE_DRIVER_NAME} instead of absolute GOPATH-dependent path. This aligns with other CSI drivers and makes builds hermetic-compatible. Comment added to indicate this is a permanent carry patch for OpenShift builds.{code}
Status: Verified
| |||
| #OCPBUGS-77854 | issue | 5 days ago | [openshift-4.17] multus-cni can't be built hermetically New |
Issue 17959379: [openshift-4.17] multus-cni can't be built hermetically
Description: This is a clone of issue OCPBUGS-77853. The following is the description of the original issue: --- Description of problem:
ART can't build multus-cni hermetically. Same applies for multus-cni-container-microshift
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir. {code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-70155 | issue | 6 weeks ago | iDRAC 10 BIOS setting succeeded but with servicing error New |
Issue 17714675: iDRAC 10 BIOS setting succeeded but with servicing error
Description: Description of problem:
Though iDRAC 10 bios settings update using HFS succeeded, oc describe shows servicing error. in addition, when node returned to Ready state, servicing error disappeared
Normal PowerOff 11m metal3-baremetal-controller Host soft powered off Normal ServicingError 3m18s metal3-baremetal-controller Node 08387329-dc15-4cef-932b-cc2cb323738d failed step {'args': {'settings': [
{'name': 'XptPrefetch', 'value': 'Disabled'}
]}, 'interface': 'bios', 'step': 'apply_configuration', 'abortable': False, 'priority': 0}: Redfish exception occurred. Error: In system 4c4c4544-0053-3410-804e-c8c04f4b3434 for node 08387329-dc15-4cef-932b-cc2cb323738d all managers failed: set virtual boot device. Errors: ['Manager 34344b4f-c0c8-4e80-3410-00534c4c4544: Resource https://10.26.16.132/redfish/v1/TaskService/TaskMonitors/JID_663271577404 not found'] Normal ServicingError 108s metal3-baremetal-controller Node 08387329-dc15-4cef-932b-cc2cb323738d failed step {'args': {'settings': [
{'name': 'XptPrefetch', 'value': 'Disabled'}
]}, 'interface': 'bios', 'step': 'apply_configuration', 'abortable': False, 'priority': 0}: Redfish exception occurred. Error: In system 4c4c4544-0053-3410-804e-c8c04f4b3434 for node 08387329-dc15-4cef-932b-cc2cb323738d all managers failed: set virtual boot device. Errors: ['Manager 34344b4f-c0c8-4e80-3410-00534c4c4544: Resource https://10.26.16.132/redfish/v1/TaskService/TaskMonitors/JID_663271577404 not found']
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1. Perform bios settings change on IDRAC 10 server using HFS
2. Check "oc describe bmh -n openshift-machine-api <host>" command output
3.
{code}
Actual results:
{code:none}
passed with servicing errors{code}
Expected results:
{code:none}
To pass without servicing errors{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-73850 | issue | 7 weeks ago | oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9 New |
Issue 17769399: oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9
Description: The oauth-proxy e2e-component tests fail on OCP 4.20 because RHEL 9 enforces fully qualified container image names ("short name mode"). The backend test image {{nginxdemos/nginx-hello:plain-text}} is rejected by Podman since it cannot determine which registry to pull from.
h2. Version-Release number
4.20
h2. How reproducible
Always
h2. Steps to Reproduce
Run the oauth-proxy e2e-component tests on OCP 4.20
The test creates a pod with a backend container using image {{nginxdemos/nginx-hello:plain-text}}
Observe the pod fails to start
h2. Actual results
The backend container fails with:
{code:java}
ImageInspectError: short name mode is enforcing, but image name
nginxdemos/nginx-hello:plain-text returns ambiguous list
{code}
The pod never reaches Running state, and all e2e tests fail.
h2. Expected results
The e2e tests should pass. The backend container should start successfully using a fully qualified image reference.
h2. Additional info
_Root cause:_ RHEL 9 enforces "short name mode" for container images, requiring fully qualified registry paths.
_Fix:_ Replace {{nginxdemos/nginx-hello:plain-text}} with {{{}registry.k8s.io/e2e-test-images/agnhost:2.53{}}}.
_Backport of:_ [OCPBUGS-73848|https://issues.redhat.com/browse/OCPBUGS-73848]
Status: New
| |||
| #OCPBUGS-78180 | issue | 9 hours ago | [openshift-4.12] ose-azure-cluster-api-controllers can't be built hermetically New |
Issue 17974852: [openshift-4.12] ose-azure-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78179. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78178. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78177. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78176. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77983. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77905. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77904. The following is the description of the original issue: --- Description of problem:
ART can't build ose-azure-cluster-api-controllers
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-72546 | issue | 3 weeks ago | cgroupsv2: failed to write on cpuset.cpus.exclusive ASSIGNED |
Issue 17747711: cgroupsv2: failed to write on cpuset.cpus.exclusive
Description: Description of problem:
{code:none}
When deploying a workload that makes use of cpuset.cpus.exclusive, it fails {code}
Version-Release number of selected component (if applicable):
{code:none}
4.18.11{code}
How reproducible:
{code:none}
Happens always with this particular workload using helm chart {code}
Steps to Reproduce:
{code:none}
1.Deploy the workload by using the customer helm chart
{code}
Actual results:
{code:none}
Error: failed to run pre-start hook for container "ctr-du-vppxx": set CPU load balancing: failed to write "7-16": write /sys/fs/cgroup/kubepods.slice/kubepods-podbf0bb53f_790a_4f26_ada7_52axxxxf43c.slice/crio-ctr-du-vppxx.scope/cpuset.cpus.exclusive: invalid argument
{code}
Expected results:
{code:none}
Pods can be deployed successfully{code}
Additional info:
{code:none}
{code}
Status: ASSIGNED
| |||
| #OCPBUGS-78183 | issue | 9 hours ago | [openshift-4.12] ose-machine-api-provider-gcp can't be built hermetically New |
Issue 17974914: [openshift-4.12] ose-machine-api-provider-gcp can't be built hermetically
Description: This is a clone of issue OCPBUGS-78182. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78181. The following is the description of the original issue: --- Description of problem:
ART can't build ose-machine-api-provider-gcp hermetically
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-78175 | issue | 9 hours ago | [openshift-4.18] ose-aws-cluster-api-controllers can't be built hermetically New |
Issue 17974775: [openshift-4.18] ose-aws-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-77903. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77902. The following is the description of the original issue: --- Description of problem:
ART can't build ose-aws-cluster-api-controllers
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.
{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-78110 | issue | 16 hours ago | [openshift-4.12] openshift-enterprise-haproxy-router can't be built hermetically New |
Issue 17972087: [openshift-4.12] openshift-enterprise-haproxy-router can't be built hermetically
Description: This is a clone of issue OCPBUGS-78109. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78108. The following is the description of the original issue: --- Description of problem:
ART can't build openshift-enterprise-haproxy-router & ose-haproxy-router-base hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-74487 | issue | 2 weeks ago | Agent service is failing with input/output error Verified |
Issue 17801998: Agent service is failing with input/output error Description: *Description of the problem:* - Unable to discover hosts using BMC because the agent service is failing to start with this error: Error: netavark: failed to start process: fork/exec /usr/libexec/podman/netavark: input/output error *How reproducible:* Always *Steps to reproduce:* 1. Boot a system using BMC/BMH 2. ssh the node and check the agent service logs *Actual results:* The agent service fails *Expected results:* The Agent service should start and the hosts should get discovered. *Additional Info:* This issue is not faced when: 1. Hosts are discovered using a discovery iso 2. When we disable converged_flow Status: Verified | |||
| #OCPBUGS-76557 | issue | 34 hours ago | RHCOS10: toolbox failed to get current user's default shell New |
Issue 17865189: RHCOS10: toolbox failed to get current user's default shell
Description: When running toolbox on an RHCOS 10.1 host it fails to launch with the following.
{code:none}
oc debug node/ip-10-0-10-196.us-east-2.compute.internal
Starting pod/ip-10-0-10-196us-east-2computeinternal-debug-nm4qh ...
To use host binaries, run `chroot /host`. Instead, if you need to access host namespaces, run `nsenter -a -t 1`.
Pod IP: 10.0.10.196
If you don't see a command prompt, try pressing enter.
sh-5.1# chroot /host
sh-5.2# toolbox
Error: missing command
create Create a new Toolbx container
enter Enter an existing Toolbx container
list List all existing Toolbx containers and images
Run 'toolbox --help' for usage.
sh-5.2# toolbox enter
Error: failed to get the current user's default shell
sh-5.2# toolbox create
Image required to create Toolbx container.
Download registry.access.redhat.com/ubi10/toolbox:10.1 (336.9MB)? [y/N]: y
Error: failed to get the current user's default shell
{code}
Status: New
| |||
| #OCPBUGS-77210 | issue | 12 hours ago | [openshift-4.16] hermetic build failure - ose-ibm-vpc-block-csi-driver Verified |
Issue 17926437: [openshift-4.16] hermetic build failure - ose-ibm-vpc-block-csi-driver
Description: This is a clone of issue OCPBUGS-77209. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77207. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77206. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77205. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77203. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77167. The following is the description of the original issue: --- Description of problem:
{code:none}
IBM VPC Block CSI Driver build fails in hermetic mode due to non-deterministic GOPATH handling in Makefile. The build system rewrites $GOPATH causing the binary output path -o ${GOPATH}/bin/${EXE_DRIVER_NAME} to become invalid, resulting in build failures.{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift 4.19+ (affects hermetic build conversion) {code}
How reproducible:
{code:none}
Always - when building ibm-vpc-block-csi-driver in hermetic mode{code}
Steps to Reproduce:
{code:none}
1. Build ibm-vpc-block-csi-driver container in hermetic mode
2. Makefile uses absolute GOPATH-dependent output path: -o ${GOPATH}/bin/${EXE_DRIVER_NAME}
3. Build system rewrites $GOPATH during hermetic build process {code}
Actual results:
{code:none}
Build fails because binary cannot be found at expected GOPATH location after build system modifies $GOPATH
Error: building at STEP "COPY --from=builder /cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver /bin/ibm-vpc-block-csi-driver": checking on sources under [...]: copier: stat: "/cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver": no such file or directory{code}
Expected results:
{code:none}
Build succeeds using relative path bin/${EXE_DRIVER_NAME} that doesn't depend on GOPATH manipulation, consistent with other CSI drivers like aws-ebs-csi-driver {code}
Additional info:
{code:none}
Fix implemented: Change Makefile to use relative path -o bin/${EXE_DRIVER_NAME} instead of absolute GOPATH-dependent path. This aligns with other CSI drivers and makes builds hermetic-compatible. Comment added to indicate this is a permanent carry patch for OpenShift builds.{code}
Status: Verified
| |||
| #OCPBUGS-78170 | issue | 9 hours ago | [openshift-4.16] ose-machine-api-provider-openstack can't be built hermetically New |
Issue 17974767: [openshift-4.16] ose-machine-api-provider-openstack can't be built hermetically
Description: This is a clone of issue OCPBUGS-78169. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78168. The following is the description of the original issue: --- Description of problem:
ART can't build ose-machine-api-provider-openstack hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77194 | issue | 2 weeks ago | [openshift-4.18] ose-libvirt-machine-controllers can't be build hermetically New |
Issue 17925858: [openshift-4.18] ose-libvirt-machine-controllers can't be build hermetically
Description: Description of problem:
ART can't build ose-libvirt-machine-controllers hermetically.
The error is the following in the prefetch-dependencies konflux task:
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details. Please try running `go mod vendor` and committing the changes. Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77488 | issue | 5 days ago | OCP 4.20.z: metal3-baremetal-operator pod not honoring NO_PROXY and routing metal3-state service traffic via proxy (works in 4.19.z) Verified |
Issue 17941975: OCP 4.20.z: metal3-baremetal-operator pod not honoring NO_PROXY and routing metal3-state service traffic via proxy (works in 4.19.z)
Description: *Summary:*
In OpenShift Container Platform (OCP) *4.20.z* versions, the *{{metal3-baremetal-operator}}* pod is not honoring the configured *{{NO_PROXY}}* environment variable and is attempting to connect to the internal *{{metal3-state}} service* via the configured HTTP/HTTPS proxy. {*}This behavior is not observed in OCP 4.19.z{*}, where the operator correctly bypasses the proxy for internal service communication.
*Environment:*
* OCP Version: 4.20.z (multiple z-streams observed)
* Working Version: 4.19.z
* Platform: Baremetal (IPI)
* Proxy configuration enabled (cluster-wide proxy configured)
* Affected Pod: {{metal3-baremetal-operator}}
* Target Service: {{metal3-state}} (ClusterIP service)
*Please find the reproducer details below. All cluster information is from my test environment and does not contain any sensitive data. I will also share the cluster must-gather shortly.*
{code:java}
> oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.20.14 True False 6h8m Cluster version is 4.20.14
> oc get proxy -oyaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
kind: Proxy
metadata:
creationTimestamp: "2026-02-27T07:44:48Z"
generation: 2
name: cluster
resourceVersion: "154487"
uid: dc9dbe2b-645a-4b2b-98d1-41ab0ef7eebb
spec:
httpProxy: http://10.74.234.38:3128
httpsProxy: http://10.74.234.38:3128
noProxy: example.com
trustedCA:
name: ""
status:
httpProxy: http://10.74.234.38:3128
httpsProxy: http://10.74.234.38:3128
noProxy: .cluster.local,.svc,10.128.0.0/14,10.74.232.0/21,127.0.0.1,172.30.0.0/16,api-int.ayush.example.com,example.com,localhost
kind: List
metadata:
resourceVersion: ""
> oc get infrastructure/cluster -oyaml
apiVersion: config.openshift.io/v1
kind: Infrastructure
metadata:
creationTimestamp: "2026-02-27T07:44:46Z"
generation: 1
name: cluster
resourceVersion: "558"
uid: 446c09a0-2b33-4ef1-bdf9-683d2cb00203
spec:
cloudConfig:
name: ""
platformSpec:
baremetal:
apiServerInternalIPs:
- 10.74.236.126
ingressIPs:
- 10.74.236.127
machineNetworks:
- 10.74.232.0/21
type: BareMetal
status:
apiServerInternalURI: https://api-int.ayush.example.com:6443
apiServerURL: https://api.ayush.example.com:6443
controlPlaneTopology: HighlyAvailable
cpuPartitioning: None
etcdDiscoveryDomain: ""
infrastructureName: ayush-hvsdf
infrastructureTopology: HighlyAvailable
platform: BareMetal
platformStatus:
baremetal:
apiServerInternalIP: 10.74.236.126
apiServerInternalIPs:
- 10.74.236.126
ingressIP: 10.74.236.127
ingressIPs:
- 10.74.236.127
loadBalancer:
type: OpenShiftManagedDefault
machineNetworks:
- 10.74.232.0/21
type: BareMetal
> oc get pod
NAME READY STATUS RESTARTS AGE
cluster-autoscaler-operator-794cbb6fb-qfqmm 2/2 Running 0 14m
cluster-baremetal-operator-5d44678794-bvdk6 2/2 Running 0 14m
control-plane-machine-set-operator-675b5f7b45-xzxt8 1/1 Running 0 14m
ironic-proxy-b8km9 1/1 Running 0 14m
ironic-proxy-flvtt 1/1 Running 1 14m
ironic-proxy-vmbjm 1/1 Running 1 14m
machine-api-controllers-6687f5fd9f-5mj5t 7/7 Running 0 10m
machine-api-operator-bc77b9d7b-trm8j 2/2 Running 0 14m
metal3-6b6ddc8cf9-nl6hn 3/3 Running 0 14m
metal3-baremetal-operator-84f9846555-cqpfx 1/1 Running 0 10m
metal3-image-customization-b4847b9d6-7hlfz 1/1 Running 0 14m
> oc logs metal3-baremetal-operator-84f9846555-cqpfx | tail -n5
{"level":"info","ts":1772211615.0165863,"logger":"controllers.BareMetalHost","msg":"provisioner is not ready","baremetalhost":{"name":"master2.ayush.example.com","namespace":"openshift-machine-api"},"Error":"Not ready","RequeueAfter":30}
{"level":"info","ts":1772211615.0166247,"logger":"provisioner.ironic","msg":"error caught while checking endpoint, will retry","host":"openshift-machine-api~compute1.ayush.example.com","endpoint":"https://metal3-state.openshift-machine-api.svc.cluster.local.:6388/v1/","error":"Get \"https://metal3-state.openshift-machine-api.svc.cluster.local.:6388/v1/\": Internal Server Error"}
{"level":"info","ts":1772211615.0166388,"logger":"controllers.BareMetalHost","msg":"provisioner is not ready","baremetalhost":{"name":"compute1.ayush.example.com","namespace":"openshift-machine-api"},"Error":"Not ready","RequeueAfter":30}
{"level":"info","ts":1772211615.0168898,"logger":"provisioner.ironic","msg":"error caught while checking endpoint, will retry","host":"openshift-machine-api~master0.ayush.example.com","endpoint":"https://metal3-state.openshift-machine-api.svc.cluster.local.:6388/v1/","error":"Get \"https://metal3-state.openshift-machine-api.svc.cluster.local.:6388/v1/\": Internal Server Error"}
{"level":"info","ts":1772211615.0169315,"logger":"controllers.BareMetalHost","msg":"provisioner is not ready","baremetalhost":{"name":"master0.ayush.example.com","namespace":"openshift-machine-api"},"Error":"Not ready","RequeueAfter":30}
> oc rsh metal3-baremetal-operator-84f9846555-cqpfx
sh-5.1$
sh-5.1$ curl -kv https://metal3-state.openshift-machine-api.svc.cluster.local.:6388/v1/
* Uses proxy env variable NO_PROXY == '.cluster.local,.svc,10.128.0.0/14,10.74.232.0/21,127.0.0.1,172.30.0.0/16,api-int.ayush.example.com,example.com,localhost,'
* Uses proxy env variable HTTPS_PROXY == 'http://10.74.234.38:3128'
* Trying 10.74.234.38:3128...
* Connected to 10.74.234.38 (10.74.234.38) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to metal3-state.openshift-machine-api.svc.cluster.local:6388
> CONNECT metal3-state.openshift-machine-api.svc.cluster.local:6388 HTTP/1.1
> Host: metal3-state.openshift-machine-api.svc.cluster.local:6388
> User-Agent: curl/7.76.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 500 Internal Server Error
< Server: squid/5.5
< Mime-Version: 1.0
< Date: Fri, 27 Feb 2026 17:00:39 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3450
< X-Squid-Error: ERR_CANNOT_FORWARD 0
< Vary: Accept-Language
< Content-Language: en
<
* Received HTTP code 500 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 500 from proxy after CONNECT
sh-5.1$
sh-5.1$
sh-5.1$
sh-5.1$
sh-5.1$ unset HTTPS_PROXY
sh-5.1$
sh-5.1$ curl -kv https://metal3-state.openshift-machine-api.svc.cluster.local.:6388/v1/
* Uses proxy env variable NO_PROXY == '.cluster.local,.svc,10.128.0.0/14,10.74.232.0/21,127.0.0.1,172.30.0.0/16,api-int.ayush.example.com,example.com,localhost,'
* Trying 172.30.114.184:6388...
* Connected to metal3-state.openshift-machine-api.svc.cluster.local (172.30.114.184) port 6388 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=localhost
* start date: Feb 27 07:51:49 2026 GMT
* expire date: Feb 27 07:51:50 2028 GMT
* issuer: CN=metal3-ironic
* SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET /v1/ HTTP/1.1
> Host: metal3-state.openshift-machine-api.svc.cluster.local:6388
> User-Agent: curl/7.76.1
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Unknown (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 27 Feb 2026 17:00:51 GMT
< Server: ironic_api
< X-OpenStack-Ironic-API-Minimum-Version: 1.1
< X-OpenStack-Ironic-API-Maximum-Version: 1.99
< X-OpenStack-Ironic-API-Version: 1.1
< Content-Length: 1029
< Content-Type: application/json
< Openstack-Request-Id: req-1d437637-879b-4af3-a8dd-9d7f9d11b172
<
{"id": "v1", "links": [{"href": "https://10.74.236.31:6385/v1/", "rel": "self"}, {"href": "https://docs.openstack.org//ironic/latest/contributor//webapi.html", "rel": "describedby", "type": "text/html"}], "media_types": {"base": "application/json", "type": "application/vnd.openstack.ironic.v1+json"}, "chassis": [{"href": "https://10.74.236.31:6385/v1/chassis/", "rel": "self"}, {"href": "https://10.74.236.31:6385/chassis/", "rel": "bookmark"}], "nodes": [{"href": "https://10.74.236.31:6385/v1/nodes/", "rel": "self"}, {"href": "https://10.74.236.31:6385/nodes/", "rel": "bookmark"}], "ports": [{"href": "https://10.74.236.31:6385/v1/ports/", "rel": "self"}, {"href": "https://10.74.236.31:6385/ports/", "rel": "bookmark"}], "drivers": [{"href": "https://10.74.236.31:6385/v1/drivers/", "rel": "self"}, {"href": "https://10.74.236.31:6385/drivers/", "rel": "bookmark"}], "version": {"id": "v1", "links": [{"href": "https://10.74.236.31:6385/v1/", "rel": "self"}], "status": "CURRENT", "min_version": "1.1", "version": "1.* Connection #0 to host metal3-state.openshift-machine-api.svc.cluster.local left intact
{code}
We can ignore the curl behavior for now, as this is expected. Curl does not consistently honor the {{no_proxy}} variables in certain scenarios. The output is shared solely to illustrate that the same response is observed when accessing the service via the proxy as when the pod process connects to the service.
Status: Verified
| |||
| #OCPBUGS-78039 | issue | 34 hours ago | [openshift-4.14] ose-powervs-machine-controllers can't be built hermetically POST |
Issue 17969611: [openshift-4.14] ose-powervs-machine-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78038. The following is the description of the original issue: --- Description of problem:
ART can't build ose-powervs-machine-controllers hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-77952 | issue | 2 days ago | [CI watcher] cannot import golang devfile - 40% impact New |
Issue 17966207: [CI watcher] cannot import golang devfile - 40% impact
Description: Due to some sort of upstream change in devfile and/or some bug in devfile golang library and/or some bug in our backend
[https://registry.devfile.io/viewer/devfiles/community/go-basic]
Create Sample Application from Add page: GS-03-TC05 (example #2): Create the different workloads from Add page Create Sample Application from Add page: GS-03-TC05 (example #2) expand_less 1m43s
{Timed out waiting for [data-test-id="submit-button"] to become enabled Error Error: Timed out waiting for [data-test-id="submit-button"] to become enabled at checkEnabled (webpack://@dev-console/integration-tests/./support/commands/app.ts:146:17)}
[pull-ci-openshift-console-main-e2e-gcp-console|https://prow.ci.openshift.org/job-history/test-platform-results/pr-logs/directory/pull-ci-openshift-console-main-e2e-gcp-console] [(all)|https://search.dptools.openshift.org/?context=1&excludeName=&groupByJob=job&maxAge=336h0m0s&maxBytes=20971520&maxMatches=5&mode=text&name=%5Epull-ci-openshift-console-main-e2e-gcp-console%24&search=%22submit-button%22&searchType=build-log&wrapLines=false] - _39 runs, 100% failed, 46% of failures match = 46% impact_
[https://search.dptools.openshift.org/?search=%22submit-button%22&maxAge=24h&context=1&type=build-log&name=pull-ci-openshift-.*&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job]
Status: New
| |||
| #OCPBUGS-78172 | issue | 9 hours ago | [openshift-4.14] ose-machine-api-provider-openstack can't be built hermetically New |
Issue 17974701: [openshift-4.14] ose-machine-api-provider-openstack can't be built hermetically
Description: This is a clone of issue OCPBUGS-78171. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78170. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78169. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78168. The following is the description of the original issue: --- Description of problem:
ART can't build ose-machine-api-provider-openstack hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-35478 | issue | 3 months ago | console never force user log out when accessTokenInactivityTimeoutSeconds reached CLOSED |
The error is app.jsx:652 unhandled error: Uncaught TypeError: B.b.logoutOpenShift is not a function TypeError: B.b.logoutOpenShift is not a function at https://console-openshift-console.apps.qe-uidaily-1015.qe.azure.devcluster.openshift.com/static/main-chunk-2447484….min.js:1:1309163 TypeError: B.b.logoutOpenShift is not a function at masthead-toolbar.jsx:614:17 masthead-toolbar.jsx:614 Uncaught TypeError: B.b.logoutOpenShift is not a function at masthead-toolbar.jsx:614:17 Comment 25791250 by UNKNOWN at 2024-10-16T07:55:12.044+0000 | |||
| #OCPBUGS-66046 | issue | 5 weeks ago | The gather job restart loop when username/password field is missing in the Secret Verified |
Issue 17613506: The gather job restart loop when username/password field is missing in the Secret
Description: Description of problem:
{code:none}
If the password field is missing in the Secret, the pod will restart repeatedly.{code}
{code:none}
mustgather-83956-figl-b5v2s 0/2 Completed 0 10m
mustgather-83956-figl-hl4rh 0/2 Completed 0 18m
mustgather-83956-figl-rq7m8 2/2 Running 0 81s{code}
{code:none}
jitli@jitlideMacBook-Pro:~/work/src/test/must-gather$ oc logs -f pod/mustgather-83956-figl-hl4rh -c upload
waiting for gathers to complete ...
waiting for gathers to complete ...
waiting for gathers to complete ...
no gather is running (0 / 4)
no gather is running (1 / 4)
no gather is running (2 / 4)
no gather is running (3 / 4)
no gather is running (4 / 4)
Error: Required Parameters have not been provided. Exiting...{code}
{code:none}
build/bin/upload:63-67
if [ -z "${caseid}" ] || [ -z "${username}" ] || [ -z "${password}" ];
then
echo "Error: Required Parameters have not been provided. Exiting..."
exit 1
fi{code}
{code:none}
The script checks three variables: caseid, username, and password. If any one of them is empty, it will exit and trigger a restart loop.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
Create a MustGather CR with a no-password Secret {code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
mustgather-83956-figl-b5v2s 0/2 Completed 0 10m
mustgather-83956-figl-hl4rh 0/2 Completed 0 18m
mustgather-83956-figl-rq7m8 2/2 Running 0 81s{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: Verified
| |||
| #OCPBUGS-76501 | issue | 3 weeks ago | coreos-installer iso kargs show broken on OVE Agent ISO ASSIGNED |
Issue 17858359: coreos-installer iso kargs show broken on OVE Agent ISO
Description: This issue is a manually created clone of OCPBUGS-64929 to backport the PR [https://github.com/openshift/appliance/pull/656] manually to 4.21
----------------
In the 4.20.2 build, attempts to inspect or modify kernel arguments (kargs) in the generated OVE ISO fail.
{code:none}
$ coreos-installer iso kargs show agent-ove.x86_64.iso
Error: No karg embed areas found; old or corrupted CoreOS ISO image.{code}
Previously, the appliance build did not use the ISO directly as built, it was unpacked and repacked using osirrox / xorriso. The recent change now uses the ISO produced by the appliance directly, which appears to expose an existing issue in how karg embed areas are handled.
[https://github.com/openshift/agent-installer-utils/commit/6d64a0891291701aa0b38b57c8ac9cfa81a69fe4]
Version-Release number of selected component (if applicable):
{code:none}
4.20.2{code}
How reproducible:
{code:none}
Always {code}
Status: ASSIGNED
| |||
| #OCPBUGS-77209 | issue | 5 days ago | [openshift-4.17] hermetic build failure - ose-ibm-vpc-block-csi-driver Verified |
Issue 17926347: [openshift-4.17] hermetic build failure - ose-ibm-vpc-block-csi-driver
Description: This is a clone of issue OCPBUGS-77207. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77206. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77205. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77203. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77167. The following is the description of the original issue: --- Description of problem:
{code:none}
IBM VPC Block CSI Driver build fails in hermetic mode due to non-deterministic GOPATH handling in Makefile. The build system rewrites $GOPATH causing the binary output path -o ${GOPATH}/bin/${EXE_DRIVER_NAME} to become invalid, resulting in build failures.{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift 4.19+ (affects hermetic build conversion) {code}
How reproducible:
{code:none}
Always - when building ibm-vpc-block-csi-driver in hermetic mode{code}
Steps to Reproduce:
{code:none}
1. Build ibm-vpc-block-csi-driver container in hermetic mode
2. Makefile uses absolute GOPATH-dependent output path: -o ${GOPATH}/bin/${EXE_DRIVER_NAME}
3. Build system rewrites $GOPATH during hermetic build process {code}
Actual results:
{code:none}
Build fails because binary cannot be found at expected GOPATH location after build system modifies $GOPATH
Error: building at STEP "COPY --from=builder /cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver /bin/ibm-vpc-block-csi-driver": checking on sources under [...]: copier: stat: "/cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver": no such file or directory{code}
Expected results:
{code:none}
Build succeeds using relative path bin/${EXE_DRIVER_NAME} that doesn't depend on GOPATH manipulation, consistent with other CSI drivers like aws-ebs-csi-driver {code}
Additional info:
{code:none}
Fix implemented: Change Makefile to use relative path -o bin/${EXE_DRIVER_NAME} instead of absolute GOPATH-dependent path. This aligns with other CSI drivers and makes builds hermetic-compatible. Comment added to indicate this is a permanent carry patch for OpenShift builds.{code}
Status: Verified
| |||
| #OCPBUGS-78043 | issue | 33 hours ago | [openshift-4.12] ose-powervs-machine-controllers can't be built hermetically New |
Issue 17969720: [openshift-4.12] ose-powervs-machine-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78041. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78039. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78038. The following is the description of the original issue: --- Description of problem:
ART can't build ose-powervs-machine-controllers hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-78102 | issue | 17 hours ago | [openshift-4.13] baremetal-machine-controller can't be built hermetically New |
Issue 17972066: [openshift-4.13] baremetal-machine-controller can't be built hermetically
Description: This is a clone of issue OCPBUGS-78101. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78100. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78099. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78098. The following is the description of the original issue: --- Description of problem:
ART can't build baremetal-machine-controller hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-78163 | issue | 9 hours ago | [openshift-4.13] ose-metallb-operator can't be built hermetically New |
Issue 17974597: [openshift-4.13] ose-metallb-operator can't be built hermetically
Description: This is a clone of issue OCPBUGS-78162. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78161. The following is the description of the original issue: --- Description of problem:
ART can't build ose-metallb-operator hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-78009 | issue | 35 hours ago | [openshift-4.13] sriov-network-* images can't be built hermetically POST |
Issue 17969326: [openshift-4.13] sriov-network-* images can't be built hermetically
Description: This is a clone of issue OCPBUGS-78007. The following is the description of the original issue: --- Description of problem:
ART can't build sriov-network-* images hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-77905 | issue | 38 hours ago | [openshift-4.18] ose-azure-cluster-api-controllers can't be built hermetically Verified |
Issue 17962500: [openshift-4.18] ose-azure-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-77904. The following is the description of the original issue: --- Description of problem:
ART can't build ose-azure-cluster-api-controllers
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: Verified
| |||
| #OCPBUGS-76539 | issue | 5 days ago | [4.20] Agent service is failing with input/output error Verified |
Issue 17862601: [4.20] Agent service is failing with input/output error Description: *Description of the problem:* - Unable to discover hosts using BMC because the agent service is failing to start with this error: Error: netavark: failed to start process: fork/exec /usr/libexec/podman/netavark: input/output error *How reproducible:* Always *Steps to reproduce:* 1. Boot a system using BMC/BMH 2. ssh the node and check the agent service logs *Actual results:* The agent service fails *Expected results:* The Agent service should start and the hosts should get discovered. *Additional Info:* This issue is not faced when: 1. Hosts are discovered using a discovery iso 2. When we disable converged_flow Status: Verified | |||
| #OCPBUGS-77651 | issue | 6 days ago | [openshift-4.22] hermetic build failure - node-feature-discovery Verified |
Issue 17948666: [openshift-4.22] hermetic build failure - node-feature-discovery
Description: Description of problem:
{code:none}
Node Feature Discovery build fails in hermetic mode due to non-deterministic binary path handling in Dockerfile. The build uses "go install" command which places binaries in unpredictable GOPATH locations, and the Dockerfile copies from multiple inconsistent paths causing binary path conflicts.
{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift Container Platform 4.19+
node-feature-discovery component across all affected versions
{code}
How reproducible:
{code:none}
Always - when building node-feature-discovery in hermetic mode
{code}
Steps to Reproduce:
{code:none}
1. Build node-feature-discovery container in hermetic mode
2. Dockerfile copies binaries from multiple paths: /go/bin/* and /go/node-feature-discovery/bin/*
3. "go install" command places binaries in non-deterministic GOPATH locations during hermetic build process
{code}
Actual results:
{code:none}
Build fails because binaries are placed in unpredictable locations by "go install" and Dockerfile attempts to copy from multiple conflicting paths
Error: Inconsistent binary placement causing copy failures during Docker image build process when switching from "go install" to "go build" in
hermetic mode
{code}
Expected results:
{code:none}
Build succeeds using deterministic binary paths. Should use "go build" with explicit output paths instead of "go install" and copy binaries from single consistent location in Dockerfile.
{code}
Additional info:
{code:none}
Root cause: "go install" command places binaries in GOPATH-dependent locations that are not deterministic in hermetic builds, unlike "go build"
which allows explicit output path control.
Fix implemented: Remove redundant /go/bin/* copy command from Dockerfile and keep only the specific /go/node-feature-discovery/bin/* path. This
ensures consistent binary placement and eliminates path conflicts during hermetic builds.
Reference upstream work: https://github.com/openshift/node-feature-discovery/pull/163/changes
Related hermetic conversion tracking: ART-14586
{code}
Status: Verified
| |||
| #OCPBUGS-77205 | issue | 5 days ago | [openshift-4.20] hermetic build failure - ose-ibm-vpc-block-csi-driver Verified |
Issue 17926331: [openshift-4.20] hermetic build failure - ose-ibm-vpc-block-csi-driver
Description: This is a clone of issue OCPBUGS-77203. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77167. The following is the description of the original issue: --- Description of problem:
{code:none}
IBM VPC Block CSI Driver build fails in hermetic mode due to non-deterministic GOPATH handling in Makefile. The build system rewrites $GOPATH causing the binary output path -o ${GOPATH}/bin/${EXE_DRIVER_NAME} to become invalid, resulting in build failures.{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift 4.19+ (affects hermetic build conversion) {code}
How reproducible:
{code:none}
Always - when building ibm-vpc-block-csi-driver in hermetic mode{code}
Steps to Reproduce:
{code:none}
1. Build ibm-vpc-block-csi-driver container in hermetic mode
2. Makefile uses absolute GOPATH-dependent output path: -o ${GOPATH}/bin/${EXE_DRIVER_NAME}
3. Build system rewrites $GOPATH during hermetic build process {code}
Actual results:
{code:none}
Build fails because binary cannot be found at expected GOPATH location after build system modifies $GOPATH
Error: building at STEP "COPY --from=builder /cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver /bin/ibm-vpc-block-csi-driver": checking on sources under [...]: copier: stat: "/cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver": no such file or directory{code}
Expected results:
{code:none}
Build succeeds using relative path bin/${EXE_DRIVER_NAME} that doesn't depend on GOPATH manipulation, consistent with other CSI drivers like aws-ebs-csi-driver {code}
Additional info:
{code:none}
Fix implemented: Change Makefile to use relative path -o bin/${EXE_DRIVER_NAME} instead of absolute GOPATH-dependent path. This aligns with other CSI drivers and makes builds hermetic-compatible. Comment added to indicate this is a permanent carry patch for OpenShift builds.{code}
Status: Verified
| |||
| #OCPBUGS-77203 | issue | 8 hours ago | [openshift-4.21] hermetic build failure - ose-ibm-vpc-block-csi-driver CLOSED |
Issue 17926162: [openshift-4.21] hermetic build failure - ose-ibm-vpc-block-csi-driver
Description: This is a clone of issue OCPBUGS-77167. The following is the description of the original issue: --- Description of problem:
{code:none}
IBM VPC Block CSI Driver build fails in hermetic mode due to non-deterministic GOPATH handling in Makefile. The build system rewrites $GOPATH causing the binary output path -o ${GOPATH}/bin/${EXE_DRIVER_NAME} to become invalid, resulting in build failures.{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift 4.19+ (affects hermetic build conversion) {code}
How reproducible:
{code:none}
Always - when building ibm-vpc-block-csi-driver in hermetic mode{code}
Steps to Reproduce:
{code:none}
1. Build ibm-vpc-block-csi-driver container in hermetic mode
2. Makefile uses absolute GOPATH-dependent output path: -o ${GOPATH}/bin/${EXE_DRIVER_NAME}
3. Build system rewrites $GOPATH during hermetic build process {code}
Actual results:
{code:none}
Build fails because binary cannot be found at expected GOPATH location after build system modifies $GOPATH
Error: building at STEP "COPY --from=builder /cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver /bin/ibm-vpc-block-csi-driver": checking on sources under [...]: copier: stat: "/cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver": no such file or directory{code}
Expected results:
{code:none}
Build succeeds using relative path bin/${EXE_DRIVER_NAME} that doesn't depend on GOPATH manipulation, consistent with other CSI drivers like aws-ebs-csi-driver {code}
Additional info:
{code:none}
Fix implemented: Change Makefile to use relative path -o bin/${EXE_DRIVER_NAME} instead of absolute GOPATH-dependent path. This aligns with other CSI drivers and makes builds hermetic-compatible. Comment added to indicate this is a permanent carry patch for OpenShift builds.{code}
Status: CLOSED
| |||
| #OCPBUGS-71238 | issue | 6 weeks ago | Component Readiness: [Installer / openshift-installer] [Other] metal IPI ipv6 Bootstrap failed. Verified |
{noformat}
time="2026-01-08T05:22:05Z" level=warning msg="Failed, retrying in 1s ... (1/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp [2600:1f18:483:cf02:c1fc:f41a:9516:ad63]:443: connect: network is unreachable"
time="2026-01-08T05:22:09Z" level=warning msg="Failed, retrying in 1s ... (1/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp [2600:1f18:483:cf02:c1fc:f41a:9516:ad63]:443: connect: network is unreachable"{noformat}
IPv6 jobs run with mirroring because of this issue with reaching Quay through RH networks, but it seems that MCO ignores ImageContentSourcePolicy? In another job, I see references to OSImageStream:
| |||
| #OCPBUGS-77994 | issue | 35 hours ago | [openshift-4.14] openshift-enterprise-console-operator can't be built hermetically New |
Issue 17968945: [openshift-4.14] openshift-enterprise-console-operator can't be built hermetically
Description: This is a clone of issue OCPBUGS-77993. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77992. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77862. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77861. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77860. The following is the description of the original issue: --- Description of problem:
ART can't build openshift-enterprise-console-operator hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-78158 | issue | 9 hours ago | [openshift-4.12] ose-nutanix-machine-controllers can't be built hermetically New |
Issue 17974724: [openshift-4.12] ose-nutanix-machine-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78157. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78156. The following is the description of the original issue: --- Description of problem:
ART can't build ose-nutanix-machine-controllers hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-54795 | issue | 13 days ago | hcp cli "--render" option should work without cluster connectivity New |
Issue 15662484: hcp cli "--render" option should work without cluster connectivity
Description: Running {{hcp create cluster agent --render}} ... connects to a remote cluster to validate the existence of the agent namespace:
{code:sh}
$ hcp create cluster agent --render --agent-namespace example
2023-12-06T14:30:15-05:00 INFO detected "10.30.8.82" from node "wrk-72" as external-api-server-address
2023-12-06T14:30:15-05:00 ERROR Failed to create cluster {"error": "failed to get agent namespace: namespaces \"example\" not found"}
github.com/openshift/hypershift/product-cli/cmd/cluster/agent.NewCreateCommand.func1
/remote-source/app/product-cli/cmd/cluster/agent/create.go:37
github.com/spf13/cobra.(*Command).execute
/remote-source/app/vendor/github.com/spf13/cobra/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
/remote-source/app/vendor/github.com/spf13/cobra/command.go:1044
github.com/spf13/cobra.(*Command).Execute
/remote-source/app/vendor/github.com/spf13/cobra/command.go:968
github.com/spf13/cobra.(*Command).ExecuteContext
/remote-source/app/vendor/github.com/spf13/cobra/command.go:961
main.main
/remote-source/app/product-cli/main.go:60
runtime.main
/usr/lib/golang/src/runtime/proc.go:250
Error: failed to get agent namespace: namespaces "example" not found
failed to get agent namespace: namespaces "example" not found
{code}
{{hcp create cluster --render}} should run without requiring remote connectivity.
Note that this behavior is *different* from that implemented by {{hcp create cluster kubevirt}}, which when run with {{--render}} does not make a connection to the remote cluster.
Status: New
| |||
| #OCPBUGS-62819 | issue | 8 days ago | Running OpenShift 4.19.10, in Azure, with IPI and ACM 2.14 on a Hub-A cluster. Azure appends '-internal' to the load balance ASSIGNED |
Issue 17457429: Running OpenShift 4.19.10, in Azure, with IPI and ACM 2.14 on a Hub-A cluster. Azure appends '-internal' to the load balance
Description: Description of problem:{code:none}
Running OpenShift 4.19.10, in Azure, with IPI and ACM on a Hub-A cluster.
Start with the following MachinePool yaml
``` yaml
apiVersion: hive.openshift.io/v1
kind: MachinePool
metadata:
annotations:
argocd.argoproj.io/sync-wave: '2'
labels:
app.kubernetes.io/instance: cluster-azncusnprd
name: azncusnprd-infra
namespace: azncusnprd
spec:
clusterDeploymentRef:
name: azncusnprd
labels:
node-role.kubernetes.io/infra: ''
name: infra
platform:
azure:
computeSubnet: snet-openshift-dev-ncus-compute
networkResourceGroupName: az-openshift-dev-001
osDisk:
diskSizeGB: 256
type: Standard_D4s_v3
virtualNetwork: vnet-openshift-dev-ncus-001
zones:
- '1'
- '2'
- '3'
replicas: 3
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/infra
value: reserved
```
The MachinePool is created on Hub-A cluster and successfully generates the corresponding MachineSets on the spoke cluster.
```yaml
- lastProbeTime: '2025-09-30T17:54:44Z'
lastTransitionTime: '2025-09-30T17:54:44Z'
message: MachineSets generated successfully
reason: MachineSetGenerationSucceeded
status: 'True'
type: MachineSetsGenerated
```
Issue 1 pops up in the Machine status conditions:
```yaml
status:
phase: Provisioning
providerStatus:
conditions:
- lastTransitionTime: '2025-09-30T17:57:07Z'
message: 'failed to create nic azncusnprd-zcwrb-infra-northcentralus1-b8fwr-nic for machine azncusnprd-zcwrb-infra-northcentralus1-b8fwr: unable to create VM network interface: load balancer azncusnprd-zcwrb not found: network.LoadBalancersClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The Resource ''Microsoft.Network/loadBalancers/azncusnprd-zcwrb'' under resource group ''azncusnprd-zcwrb-rg'' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"'
reason: MachineCreationFailed
status: 'False'
type: MachineCreated
```
The balancer in Azure has appended `-internal` to the name, so the Machine controller is looking for the wrong name. I do not see an option to pass a custom name into the MachinePool to match the balancer name. This may even have to be an object created in the default Azure resource group before cluster creation.
If I manually create a balancer without the `-internal` suffix the Machine gets further and hits Issue 2:
```yaml
errorMessage: |
failed to reconcile machine "azncusnprd-zcwrb-infra-northcentralus1-b8fwr": failed to create vm azncusnprd-zcwrb-infra-northcentralus1-b8fwr: failed to create VM: cannot create vm: PUT https://management.azure.com/subscriptions/56a334d6-620c-49b9-a0b7-08fd25ebf304/resourceGroups/azncusnprd-zcwrb-rg/providers/Microsoft.Compute/virtualMachines/azncusnprd-zcwrb-infra-northcentralus1-b8fwr
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotSupportAvailabilityZones
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotSupportAvailabilityZones",
"message": "The resource 'Microsoft.Compute/virtualMachines/azncusnprd-zcwrb-infra-northcentralus1-b8fwr' does not support availability zones at location 'northcentralus'."
}
}
--------------------------------------------------------------------------------
errorReason: InvalidConfiguration
lastUpdated: '2025-09-30T18:11:49Z'
phase: Failed
```
Azure documentation states that the VM does support zones in `northcentralus`.
If I go into one of the MachineSets and manually remove the spec.template.spec.providerSpec.value.zone: 1 from azncusnprd-zcwrb-infra-northcentralus1 then the machine Provisions successfully and eventually starts up as a node.
```yaml
status:
providerStatus:
conditions:
- lastTransitionTime: '2025-09-30T18:18:31Z'
message: machine successfully created
reason: MachineCreationSucceeded
status: 'True'
type: MachineCreated
```
If I delete the MachinePool, remove the zones section from the yaml, and recreate the MachinePool then OpenShift on Hub-A that creates the MachinePool errors out because of requiring zones.
```yaml
status:
conditions:
- lastProbeTime: '2025-09-30T18:34:09Z'
lastTransitionTime: '2025-09-30T18:34:09Z'
message: 'could not generate machinesets: zero zones returned for region northcentralus'
reason: MachineSetGenerationFailed
status: 'False'
type: MachineSetsGenerated
```
1. unsure about the load balancer naming issue.
2. Openshift MachinePool requires a zone for northcentralus.
Azure requires no zone for northcentralus.
Describe the impact to you or the business
This is testing for future use cases.
{code}
Version-Release number of selected component (if applicable):{code:none}
{code}
How reproducible:{code:none}
{code}
Steps to Reproduce:{code:none}
See above
{code}
Actual results:{code:none}
Requires manual intervention to fix the install issues
{code}
Expected results:{code:none}
No manual intervention
{code}
Additional info:{code:none}
{code}
Status: ASSIGNED
| |||
| #OCPBUGS-78020 | issue | 34 hours ago | [openshift-4.16] ose-vsphere-cluster-api-controllers can't be built hermetically New |
Issue 17969352: [openshift-4.16] ose-vsphere-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78019. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78018. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78017. The following is the description of the original issue: --- Description of problem:
ART can't build ose-vsphere-cluster-api-controllers hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-78019 | issue | 34 hours ago | [openshift-4.17] ose-vsphere-cluster-api-controllers can't be built hermetically New |
Issue 17969423: [openshift-4.17] ose-vsphere-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78018. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78017. The following is the description of the original issue: --- Description of problem:
ART can't build ose-vsphere-cluster-api-controllers hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-72371 | issue | 8 weeks ago | Installation behind proxy failed when enable TP in 4.22 New |
Issue 17742056: Installation behind proxy failed when enable TP in 4.22
Description: Description of problem:
{code:none}
https://reportportal-openshift.apps.dno.ocp-hub.prod.psi.redhat.com/ui/#prow/launches/1578/823195
this prow job failed and i try to reproduce. finally i found the installation behind proxy will failed when enable TP in 4.22. the bootstrap was failed, the master machines was created but can't fetch the ignition.
The machine-config-controller has been continuously rolling back.{code}
Version-Release number of selected component (if applicable):
{code:none}
4.22.0-0.nightly-2026-01-06-164201{code}
How reproducible:
{code:none}
install cluster enable TP and proxy.{code}
Steps to Reproduce:
{code:none}
1.install a cluster behind proxy and enable tp.
2.
3.
{code}
Actual results:
{code:none}
installation failed.{code}
Expected results:
{code:none}
installation successful. {code}
Additional info:
{code:none}
time="2026-01-08T06:33:46Z" level=warning msg="Failed, retrying in 1s ... (1/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 52.203.142.72:443: i/o timeout"
E0108 06:34:16.850064 1 imagestream_source.go:62] error inspecting ImageStream Image for OSImageStream generation. quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53f09f0ee42099faddae15e2a3364f6467396c55f34660e6cb8492e6a9b9f3ae image will is ignored. context deadline exceeded
E0108 06:34:16.850074 1 imagestream_source.go:62] error inspecting ImageStream Image for OSImageStream generation. quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec23b7e42ea89169a7f5c0f42bcc78cfb64d7297833ea989db33d9ead97e322 image will is ignored. error getting image source for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec23b7e42ea89169a7f5c0f42bcc78cfb64d7297833ea989db33d9ead97e322: error getting image source for //quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec23b7e42ea89169a7f5c0f42bcc78cfb64d7297833ea989db33d9ead97e322: pinging container registry quay.io: Get "https://quay.io/v2/": context deadline exceeded
F0108 06:34:16.850456 1 bootstrap.go:47] error running MCC[BOOTSTRAP]: error inspecting available OSImageStreams: unable to retrieve any OSImageStream from the configured sources{code}
Status: New
| |||
| #OCPBUGS-78112 | issue | 16 hours ago | [openshift-4.14] ose-agent-installer-api-server can't be built hermetically POST |
Issue 17972532: [openshift-4.14] ose-agent-installer-api-server can't be built hermetically
Description: This is a clone of issue OCPBUGS-78111. The following is the description of the original issue: --- Description of problem:
ART can't build ose-agent-installer-api-server hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-77195 | issue | 2 weeks ago | [openshift-4.17] ose-libvirt-machine-controllers can't be build hermetically New |
Issue 17925859: [openshift-4.17] ose-libvirt-machine-controllers can't be build hermetically
Description: Description of problem:
ART can't build ose-libvirt-machine-controllers hermetically.
The error is the following in the prefetch-dependencies konflux task:
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details. Please try running `go mod vendor` and committing the changes. Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-78177 | issue | 9 hours ago | [openshift-4.15] ose-azure-cluster-api-controllers can't be built hermetically New |
Issue 17974782: [openshift-4.15] ose-azure-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78176. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77983. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77905. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77904. The following is the description of the original issue: --- Description of problem:
ART can't build ose-azure-cluster-api-controllers
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-76642 | issue | 6 days ago | TestImageRegistryPods test failing in CMO end-to-end test suite Verified |
Issue 17871754: TestImageRegistryPods test failing in CMO end-to-end test suite
Description: This is a clone of issue OCPBUGS-76488. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-66047. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-63235. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-63123. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-63017. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-62453. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-62308. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-62109. The following is the description of the original issue: --- We can't merge pull requests in github.com/openshift/cluster-monitoring because of the test failure. Example of failed job run:
https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-monitoring-operator/2649/pull-ci-openshift-cluster-monitoring-operator-main-e2e-agnostic-operator/1970148446123855872
{code:none}
=== RUN TestUserWorkloadMonitoringAlerting/assert_containers_images_registry_is_the_same_than_CMO
image_registry_test.go:52:
Error Trace: /go/src/github.com/openshift/cluster-monitoring-operator/test/e2e/image_registry_test.go:52
/go/src/github.com/openshift/cluster-monitoring-operator/test/e2e/image_registry_test.go:18
Error: Not equal:
expected: "registry.build05.ci.openshift.org"
actual : "quay-proxy.ci.openshift.org"
Diff:
--- Expected
+++ Actual
@@ -1 +1 @@
-registry.build05.ci.openshift.org
+quay-proxy.ci.openshift.org
Test: TestUserWorkloadMonitoringAlerting/assert_containers_images_registry_is_the_same_than_CMO
=== RUN TestUserWorkloadMonitoringAlerting/assert_cross-namespace_rules_are_not_allowed_from_CMO_config
--- FAIL: TestUserWorkloadMonitoringAlerting (219.63s)
--- PASS: TestUserWorkloadMonitoringAlerting/assert_user_workload_rules (0.07s)
--- PASS: TestUserWorkloadMonitoringAlerting/assert_tenancy_model_is_enforced_for_rules_and_alerts (10.29s)
--- PASS: TestUserWorkloadMonitoringAlerting/assert_rules_without_namespace_enforcement (50.01s)
--- PASS: TestUserWorkloadMonitoringAlerting/assert_prometheus_is_not_deployed_in_user_namespace (0.00s)
--- PASS: TestUserWorkloadMonitoringAlerting/assert_alertmanager_is_not_deployed_in_user_namespace (0.01s)
--- FAIL: TestUserWorkloadMonitoringAlerting/assert_containers_images_registry_is_the_same_than_CMO (2.01s)
--- PASS: TestUserWorkloadMonitoringAlerting/assert_cross-namespace_rules_are_not_allowed_from_CMO_config (85.02s)
{code}
Status: Verified
| |||
| #OCPBUGS-77191 | issue | 2 weeks ago | [openshift-4.21] ose-libvirt-machine-controllers can't be build hermetically New |
Issue 17925851: [openshift-4.21] ose-libvirt-machine-controllers can't be build hermetically
Description: Description of problem:
ART can't build ose-libvirt-machine-controllers hermetically.
The error is the following in the prefetch-dependencies konflux task:
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details. Please try running `go mod vendor` and committing the changes. Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77152 | issue | 5 days ago | [openshift-4.20] ose-gcp-cloud-controller-manager can't be build hermetically Verified |
Issue 17918009: [openshift-4.20] ose-gcp-cloud-controller-manager can't be build hermetically
Description: Description of problem:
ART can't build ose-gcp-cloud-controller-manager hermetically.
The error is the following in the prefetch-dependencies konflux task:
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details. Please try running `go mod vendor` and committing the changes. Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
This happens only for openshift-4.20
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: Verified
| |||
| #OCPBUGS-76535 | issue | 2 weeks ago | [openshift-4.22] ose-libvirt-machine-controllers can't be build hermetically POST |
Issue 17861873: [openshift-4.22] ose-libvirt-machine-controllers can't be build hermetically
Description: Description of problem:
ART can't build ose-libvirt-machine-controllers hermetically.
The error is the following in the prefetch-dependencies konflux task:
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details. Please try running `go mod vendor` and committing the changes. Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-77993 | issue | 35 hours ago | [openshift-4.15] openshift-enterprise-console-operator can't be built hermetically New |
Issue 17968941: [openshift-4.15] openshift-enterprise-console-operator can't be built hermetically
Description: This is a clone of issue OCPBUGS-77992. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77862. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77861. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77860. The following is the description of the original issue: --- Description of problem:
ART can't build openshift-enterprise-console-operator hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77991 | issue | 33 hours ago | Component Readiness: nested container should pass podman localsystem test - suite failing POST |
{code:java}
Error: building at STEP "ADD https://github.com/containers/podman/blob/main/README.md /tmp/": reading "https://github.com/containers/podman/blob/main/README.md": invalid response status 429{code}
The error indicates that [https://github.com/containers/podman/blob/main/README.md] returns 429 TooManyRequests.
| |||
| #OCPBUGS-77784 | issue | 31 hours ago | [openshift-4.18] ose-sriov-rdma-cni can't be built hermetically New |
Issue 17955267: [openshift-4.18] ose-sriov-rdma-cni can't be built hermetically
Description: This is a clone of issue OCPBUGS-77775. The following is the description of the original issue: --- Description of problem:
ART can't build ose-sriov-rdma-cni hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-662 | issue | 2 months ago | GitLab runner operator fails to install. New |
Issue 14818715: GitLab runner operator fails to install. Description: Description of problem: The Certified GitLab Operator fails to install and results in below error: ~~~ 10s Warning Failed pod/gitlab-runner-controller-manager-86668cd5c8-tvmnr Failed to pull image "registry.gitlab.com/gitlab-org/gl-openshift/gitlab-runner-operator/gitlab-runner-operator@sha256:ead726da6f75d80ff7b389dac853096d8680b9910cd5079d3aaf77a31846bf7b": rpc error: code = Unknown desc = reading manifest sha256:ead726da6f75d80ff7b389dac853096d8680b9910cd5079d3aaf77a31846bf7b in registry.gitlab.com/gitlab-org/gl-openshift/gitlab-runner-operator/gitlab-runner-operator: manifest unknown: manifest unknown 10s Warning Failed pod/gitlab-runner-controller-manager-86668cd5c8-tvmnr Error: ErrImagePull 24s Normal BackOff pod/gitlab-runner-controller-manager-86668cd5c8-tvmnr Back-off pulling image "registry.gitlab.com/gitlab-org/gl-openshift/gitlab-runner-operator/gitlab-runner-operator@sha256:ead726da6f75d80ff7b389dac853096d8680b9910cd5079d3aaf77a31846bf7b" 24s Warning Failed pod/gitlab-runner-controller-manager-86668cd5c8-tvmnr Error: ImagePullBackOff ~~~ The pod is in ImagePullBackOff state: ~~~ $ oc get pods NAME READY STATUS RESTARTS AGE gitlab-runner-controller-manager-86668cd5c8-tvmnr 1/2 ImagePullBackOff 0 3m53s ~~~ Version-Release number of selected component (if applicable): OCP 4.10 How reproducible: Always Steps to Reproduce: 1. Go to Operator Hub 2. Search for GitLab Runner 3. Install 4. Monitor the Events and pod. Actual results: Pod in ImagePullBackOff Expected results: Operator should work as expected. Additional info: We tried installing the Community GitLab runner and it works as expected. Status: New | |||
| #OCPBUGS-78178 | issue | 9 hours ago | [openshift-4.14] ose-azure-cluster-api-controllers can't be built hermetically New |
Issue 17974849: [openshift-4.14] ose-azure-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78177. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78176. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77983. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77905. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77904. The following is the description of the original issue: --- Description of problem:
ART can't build ose-azure-cluster-api-controllers
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-49422 | issue | 5 weeks ago | Unable to setup AMD GPU operator in an air-gapped env New |
Issue 16626817: Unable to setup AMD GPU operator in an air-gapped env
Description: h3.Description
Installing AMD GPU operator (amd-gpu-operator.v1.1.1) from certified-operators (channel 'alpha') in an air-gapped env (AWS).
h3.Problem
The problem seems to be with the {{DeviceConfig}} which triggers a "build" pod (pls correct me if I'm wrong), in my case
{code}
dc-internal-registry-build-6rvp6-build
{code}
Unfortunately, one of the steps in that build is
{code}
[1/2] STEP 5/9: RUN dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm -y && crb enable && sed -i "s/\$releasever/9/g" /etc/yum.repos.d/epel*.repo && dnf install dnf-plugin-config-manager -y && dnf clean all
{code}
which fails on
{code}
Curl error (28): Timeout was reached for https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi9/9/x86_64/baseos/os/repodata/repomd.xml [Failed to connect to cdn-ubi.redhat.com port 443: Connection timed out]
Error: Failed to download metadata for repo 'ubi-9-baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
error: build error: building at STEP "RUN dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm -y && crb enable && sed -i "s/\$releasever/9/g" /etc/yum.repos.d/epel*.repo && dnf install dnf-plugin-config-manager -y && dnf clean all": while running runtime: exit status 1
{code}
Obviously, we cannot do such steps in a disconnected env.
EDIT:
To avoid the building step, we could use a pre-compiled image as described at
https://dcgpu.docs.amd.com/projects/gpu-operator/en/main/drivers/precompiled-driver.html#using-pre-compiled-images
Status: New
| |||
| #OCPBUGS-58473 | issue | 7 months ago | [CI][Suite:openshift/openstack][lb][Serial] The Openstack platform LoadBalancer tests are failing CLOSED |
Issue 17124490: [CI][Suite:openshift/openstack][lb][Serial] The Openstack platform LoadBalancer tests are failing
Description: *Description of problem:*
The Openstack platform LoadBalancer tests are failing only on [periodic-ci-shiftstack-ci-release-4.20-e2e-openstack-ccpmso|https://prow.ci.openshift.org/job-history/gs/test-platform-results/logs/periodic-ci-shiftstack-ci-release-4.20-e2e-openstack-ccpmso] jobs by authentication issues.
The failed tests are:
{code:none}
[sig-installer][Suite:openshift/openstack] Bugfix ocpbug_1765: [Serial] noAllowedAddressPairs on one port should not affect other ports
[sig-installer][Suite:openshift/openstack] The OpenShift cluster should allow the manual setting of enable_topology to false [Serial]
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should apply lb-method on TCP Amphora LoadBalancer when a TCP svc with monitors and ETP:Local is created on Openshift
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should apply lb-method on TCP OVN LoadBalancer when a TCP svc with monitors and ETP:Local is created on Openshift
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should apply lb-method on UDP Amphora LoadBalancer when a UDP svc with monitors and ETP:Local is created on Openshift
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should apply lb-method on UDP OVN LoadBalancer when a UDP svc with monitors and ETP:Local is created on Openshift
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should create a TCP Amphora LoadBalancer when LoadBalancerService ingressController is created on Openshift
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should create a TCP OVN LoadBalancer when LoadBalancerService ingressController is created on Openshift
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should create a TCP OVN LoadBalancer when a TCP svc with type:LoadBalancer is created on Openshift
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should create a UDP Amphora LoadBalancer when a UDP svc with type:LoadBalancer is created on Openshift
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should create a UDP OVN LoadBalancer when a UDP svc with type:LoadBalancer is created on Openshift
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should create an UDP Amphora LoadBalancer using a pre-created FIP when an UDP LoadBalancer svc setting the LoadBalancerIP spec is created on Openshift
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should create an UDP OVN LoadBalancer using a pre-created FIP when an UDP LoadBalancer svc setting the LoadBalancerIP spec is created on Openshift
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should limit service access on an UDP OVN LoadBalancer when an UDP LoadBalancer svc setting the loadBalancerSourceRanges spec is created on Openshift
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should not re-use an existing UDP Amphora LoadBalancer if shared internal svc is created
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should re-use an existing UDP Amphora LoadBalancer when new svc is created on Openshift with the proper annotation
[sig-installer][Suite:openshift/openstack][lb][Serial] The Openstack platform should re-use an existing UDP OVN LoadBalancer when new svc is created on Openshift with the proper annotation {code}
*Version-Release number of selected component (if applicable):*
{code:none}
4.20 {code}
*How reproducible:*
[https://steps.ci.openshift.org/job?org=shiftstack&repo=ci&branch=release-4.20&test=e2e-openstack-ccpmso]
*Steps to Reproduce:*
{code:none}
1.
2.
3.
{code}
*Actual results:*
{code:none}
Failing {code}
Expected results:
{code:none}
Pass {code}
*Additional info:*
From this log [https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-shiftstack-ci-release-4.20-e2e-openstack-ccpmso/1942220378185142272/artifacts/e2e-openstack-ccpmso/openstack-test-openstack/build-log.txt]:
{code:none}
fail [github.com/openshift/openstack-test/test/extended/openstack/loadbalancers.go:76]: Failed to build the OpenStack client
Unexpected error:
<gophercloud.ErrUnexpectedResponseCode>:
Expected HTTP response code [201 202] when accessing [POST https://identity.rdo.mtl2.vexxhost.net/v3/auth/tokens], but got 401 instead: {"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}
{
BaseError: {DefaultErrString: "", Info: ""},
URL: "https://identity.rdo.mtl2.vexxhost.net/v3/auth/tokens",
Method: "POST",
Expected: [201, 202],
Actual: 401, {code}
Status: CLOSED
| |||
| #OCPBUGS-78099 | issue | 5 hours ago | [openshift-4.16] baremetal-machine-controller can't be built hermetically MODIFIED |
Issue 17972332: [openshift-4.16] baremetal-machine-controller can't be built hermetically
Description: This is a clone of issue OCPBUGS-78098. The following is the description of the original issue: --- Description of problem:
ART can't build baremetal-machine-controller hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: ON_QA
| |||
| #OCPBUGS-66942 | issue | 2 months ago | sos report hit error during collect datas for openshift_ovn plugin New |
Issue 17648864: sos report hit error during collect datas for openshift_ovn plugin
Description: Description of problem:
{code:none}
sos report hit error during collect datas for openshift_ovn plugin {code}
Version-Release number of selected component (if applicable):
{code:none}
ocp version: 4.21
sos version:
sh-5.1# rpm -qa | grep sos
sos-4.10.0-4.el9_4.noarch{code}
How reproducible:
{code:none}
always{code}
Steps to Reproduce:
{code:none}
1. setup 4.21 cluster
2. debug into one of the node
3. run sos report to collect datas for openshift_ovn plugin
sh-5.1# sos report --only-plugins=openshift_ovn
sos report (version 4.10.0)
WARNING: unable to set option for disabled or non-existing plugin (crio).
WARNING: unable to set option for disabled or non-existing plugin (podman).
This command will collect diagnostic and configuration information from
this Red Hat Enterprise Linux system and installed applications.
An archive containing the collected information will be generated in
/host/var/tmp/sos.k_j0wvlj and may be provided to a Red Hat support
representative.
Any information provided to Red Hat will be treated in accordance with
the published support policies at:
Distribution Website : https://www.redhat.com/
Commercial Support : https://access.redhat.com/
The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.
No changes will be made to system configuration.
Press ENTER to continue, or CTRL-C to quit.
Optionally, please enter the case id that you are generating this report for:
Setting up archive ...
Setting up plugins ...
caught exception in plugin method "openshift_ovn.setup()"
writing traceback to sos_logs/openshift_ovn-plugin-errors.txt
Running plugins. Please wait ...
Starting 1/1 openshift_ovn [Running: openshift_ovn]
Finished running plugins
Creating compressed archive...
Your sos report has been generated and saved in:
/host/var/tmp/sosreport-ip-10-0-33-119-2025-12-08-tjievgx.tar.xz
Size 345.62KiB
Owner root
sha256 b009e4bcb2cf4d183e8c795d1329f5ad7979b4e53153f60adaeabab4a62c2e0c
Please send this file to your support representative.
sh-5.1# tar -xvf /host/var/tmp/sosreport-ip-10-0-33-119-2025-12-08-tjievgx.tar.xz
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/version.txt
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/sos_reports/
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/sos_reports/manifest.json
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/sos_reports/sos.html
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/sos_reports/sos.json
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/sos_reports/sos.txt
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/sos_logs/
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/sos_logs/openshift_ovn-plugin-errors.txt
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/sos_logs/sos.log
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/sos_logs/ui.log
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/environment
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/sos_commands/
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/lib/
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/lib/openvswitch/
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/lib/openvswitch/etc/
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/lib/openvswitch/etc/keys/
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/lib/openvswitch/etc/keys/ipsec-cacert.pem
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/lib/openvswitch/etc/keys/ipsec-cert.pem
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/lib/openvswitch/etc/keys/ipsec-privkey.pem
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/lib/openvswitch/etc/keys/ipsec-req.pem
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/lib/ovn-ic/
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/lib/ovn-ic/etc/
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/lib/ovn-ic/etc/libovsdb.log
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/lib/ovn-ic/etc/ovnnb_db.db
sosreport-ip-10-0-33-119-2025-12-08-tjievgx/var/lib/ovn-ic/etc/ovnsb_db.db {code}
Actual results:
{code:none}
There is no directory openshift_ovn under sos_commands directory:
sh-5.1# ls -l sosreport-ip-10-0-33-119-2025-12-08-tjievgx/sos_commands/
total 0
And there is errors for collecting openshift_ovn plugin:
sh-5.1# cat sosreport-ip-10-0-33-119-2025-12-08-tjievgx/sos_logs/openshift_ovn-plugin-errors.txt Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/sos/report/__init__.py", line 1260, in setup plug.setup() File "/usr/lib/python3.9/site-packages/sos/report/plugins/openshift_ovn.py", line 74, in setup self.add_cmd_output([ File "/usr/lib/python3.9/site-packages/sos/report/plugins/__init__.py", line 2212, in add_cmd_output cmd = self.fmt_container_cmd(container, cmd, runtime=runtime, File "/usr/lib/python3.9/site-packages/sos/report/plugins/__init__.py", line 2970, in fmt_container_cmd return _runtime.fmt_container_cmd(container, cmd, quotecmd) UnboundLocalError: local variable '_runtime' referenced before assignment{code}
Expected results:
{code:none}
sos report can collect datas for openshift_ovn plugins{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77087 | issue | 6 days ago | multus-networkpolicy not building successfully in Konflux CLOSED |
Issue 17910000: multus-networkpolicy not building successfully in Konflux Description: multus-networkpolicy hasn't built successfully in Konflux since [https://github.com/openshift/multus-networkpolicy/pull/77] went in adding nftables package dependency Example here [https://art-build-history-art-build-history.apps.artc2023.pc3z.p1.openshiftapps.com/logs?nvr=ose-multus-networkpolicy-container-v4.22.0-202602191351.p2.g25d18cf.assembly.stream.el9&record_id=ca61f00c-c694-859d-a8c9-08afbcc2e55c&group=openshift-4.22] Logs end at [2/2] STEP 1/19: FROM quay.io/redhat-user-workloads/ocp-art-tenant/art-images@sha256:955b4479354cd94013ca7b6707a3affafb0806c8d99f70cd76e4a0d2f929c9b6 [2/2] STEP 2/19: ENV ART_BUILD_ENGINE=konflux [2/2] STEP 3/19: ENV ART_BUILD_DEPS_METHOD=cachi2 [2/2] STEP 4/19: ENV ART_BUILD_NETWORK=hermetic [2/2] STEP 5/19: RUN . /cachi2/cachi2.env && go clean -cache || true /bin/sh: line 1: go: command not found [2/2] STEP 6/19: ENV ART_BUILD_DEPS_MODE=default [2/2] STEP 7/19: ENV __doozer=update BUILD_RELEASE=202602191351.p2.g25d18cf.assembly.stream.el9 BUILD_VERSION=v4.22.0 OS_GIT_MAJOR=4 OS_GIT_MINOR=22 OS_GIT_PATCH=0 OS_GIT_TREE_STATE=clean OS_GIT_VERSION=4.22.0-202602191351.p2.g25d18cf.assembly.stream.el9 SOURCE_GIT_TREE_STATE=clean __doozer_group=openshift-4.22 __doozer_key=multus-networkpolicy __doozer_uuid_tag=ose-multus-networkpolicy-rhel9-v4.22.0-20260219.142513 __doozer_version=v4.22.0 [2/2] STEP 8/19: ENV __doozer=merge OS_GIT_COMMIT=25d18cf OS_GIT_VERSION=4.22.0-202602191351.p2.g25d18cf.assembly.stream.el9-25d18cf SOURCE_DATE_EPOCH=1771275974 SOURCE_GIT_COMMIT=25d18cf8247ad6ba1a6cc8e37cc63f236eaee689 SOURCE_GIT_TAG=25d18cf8 SOURCE_GIT_URL=https://github.com/openshift/multus-networkpolicy [2/2] STEP 9/19: RUN . /cachi2/cachi2.env && dnf install -y nftables && dnf clean all error: No package matches 'nftables' subprocess exited with status 1 subprocess exited with status 1 Error: building at STEP "RUN . /cachi2/cachi2.env && dnf install -y nftables && dnf clean all": exit status 1 Status: CLOSED | |||
| #OCPBUGS-77983 | issue | 38 hours ago | [openshift-4.17] ose-azure-cluster-api-controllers can't be built hermetically POST |
Issue 17968065: [openshift-4.17] ose-azure-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-77905. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77904. The following is the description of the original issue: --- Description of problem:
ART can't build ose-azure-cluster-api-controllers
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-30345 | issue | 2 weeks ago | OperatorHub.yaml is missing from the ztp source-crs Verified |
Issue 15860647: OperatorHub.yaml is missing from the ztp source-crs
Description: This is a clone of issue OCPBUGS-27893. The following is the description of the original issue: --- Description of problem:
{code:none}
gitops deployment fails - Policies are not syncing because OperatorHub.yaml is missing from the ztp source-crs.{code}
[kni@registry.kni-qe-27 ~]$ oc get apps -A
NAMESPACE NAME SYNC STATUS HEALTH STATUS
openshift-gitops clusters Synced Healthy
openshift-gitops policies Unknown Healthy
[kni@registry.kni-qe-27 ~]$ oc get apps -n openshift-gitops policies -o yaml
[...]
status:
conditions:
- lastTransitionTime: "2024-01-24T15:08:10Z"
message: |-
Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = `kustomize build <path to cached source>/policygentemplates --enable-alpha-plugins` failed exit status 1: 2024/01/24 15:08:10 Could not build the entire policy defined by /tmp/kust-plugin-config-673779590: OperatorHub.yaml is not found both in Git path: <path to cached source>/policygentemplates/source-crs/OperatorHub.yaml and in the ztp container path: /.config/kustomize/plugin/ran.openshift.io/v1/policygentemplate/source-crs/OperatorHub.yaml
Error: failure in plugin configured via /tmp/kust-plugin-config-673779590; exit status 1: exit status 1
type: ComparisonError
Version-Release number of selected component (if applicable):
{code:none}
OCP version: 4.15
ztp-site-generator:v4.15.0-30{code}
How reproducible:
{code:none}
100%{code}
Steps to Reproduce:
{code:none}
1. Deploy 4.15 SNO
2. Deploy gitops-ztp
3. Check apps
[kni@registry.kni-qe-27 ~]$ oc get apps -A
NAMESPACE NAME SYNC STATUS HEALTH STATUS
openshift-gitops clusters Synced Healthy
openshift-gitops policies Unknown Healthy
[kni@registry.kni-qe-27 ~]$ oc get apps -n openshift-gitops policies -o yaml
[...]
status:
conditions:
- lastTransitionTime: "2024-01-24T15:08:10Z"
message: |-
Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = `kustomize build <path to cached source>/policygentemplates --enable-alpha-plugins` failed exit status 1: 2024/01/24 15:08:10 Could not build the entire policy defined by /tmp/kust-plugin-config-673779590: OperatorHub.yaml is not found both in Git path: <path to cached source>/policygentemplates/source-crs/OperatorHub.yaml and in the ztp container path: /.config/kustomize/plugin/ran.openshift.io/v1/policygentemplate/source-crs/OperatorHub.yaml
Error: failure in plugin configured via /tmp/kust-plugin-config-673779590; exit status 1: exit status 1
type: ComparisonError {code}
Actual results:
{code:none}
openshift-gitops policies Unknown Healthy message: |- Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = `kustomize build <path to cached source>/policygentemplates --enable-alpha-plugins` failed exit status 1: 2024/01/24 15:08:10 Could not build the entire policy defined by /tmp/kust-plugin-config-673779590: OperatorHub.yaml is not found both in Git path: <path to cached source>/policygentemplates/source-crs/OperatorHub.yaml and in the ztp container path: /.config/kustomize/plugin/ran.openshift.io/v1/policygentemplate/source-crs/OperatorHub.yaml{code}
Expected results:
{code:none}
openshift-gitops policies sync{code}
Additional info:
{code:none}
{code}
Status: Verified
| |||
| #OCPBUGS-78171 | issue | 9 hours ago | [openshift-4.15] ose-machine-api-provider-openstack can't be built hermetically New |
Issue 17974839: [openshift-4.15] ose-machine-api-provider-openstack can't be built hermetically
Description: This is a clone of issue OCPBUGS-78170. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78169. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78168. The following is the description of the original issue: --- Description of problem:
ART can't build ose-machine-api-provider-openstack hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-67013 | issue | 4 days ago | Unexpected + character in apirequestcounts CRD ASSIGNED |
Issue 17653802: Unexpected + character in apirequestcounts CRD
Description: Description of problem:
{code:none}
$ oc get --raw /apis/apiserver.openshift.io/v1/apirequestcounts
Error from server: invalid character '+' looking for beginning of value
$ oc logs kube-apiserver-master-1 kube-apiserver | less
2025-11-21T13:35:07.009092150Z E1121 13:35:07.009007 18 status.go:71] apiserver received an error that is not an metav1.Status: &json.SyntaxError{msg:"invalid character '+' looking for beginning of value", Offset:3190}: invalid character '+' looking for beginning of value
We used a python script to detect the json syntax stored in etcd for apirequestcount and we found:
Key: /kubernetes.io/apiserver.openshift.io/apirequestcounts/consolesamples.v1.console.openshift.io
Error: Expecting value: line 1 column 3190 (char 3189)
Bad Content Snippet: ","requestCount":8},+"byUser":[{"byVerb"
There is an unexpected + character in front of "byUser". After deleting the bad apirequestcount from etcd, the api works{code}
Version-Release number of selected component (if applicable):
{code:none}
4.16{code}
How reproducible:
{code:none}
Unknown{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: ASSIGNED
| |||
| #OCPBUGS-77673 | issue | 7 days ago | [CI Failure] ocp-spoke-assisted-operator-cim-ui-deploy - Build #4427 - Product Code - MCE Backend 500 Errors CLOSED |
Issue 17950639: [CI Failure] ocp-spoke-assisted-operator-cim-ui-deploy - Build #4427 - Product Code - MCE Backend 500 Errors
Description: {panel:title=CI Failure Analysis|borderStyle=solid|borderColor=#ccc|titleBGColor=#F7D6C1|bgColor=#FFFFCE} _Job:_ ocp-spoke-assisted-operator-cim-ui-deploy _Build:_ #4427 _Classification:_ Product Code _Environment:_ OCP 4.21.0 Nightly / MCE 2.11.0 / IPv6 Disconnected h3. Root Cause UI Test timed out waiting for networking fields because the MCE backend returned 500 Internal Server Errors. h3. Error Details {code} AssertionError: Timed out retrying after 30000ms: Expected to find element: #form-input-clusterNetworks-0-cidr-field {code} The backend returned: {code} Status: 500 {code} {panel}
Status: CLOSED
| |||
| #OCPBUGS-77976 | issue | 40 hours ago | [openshift-4.12] ose-cluster-api can't be built hermetically New |
Issue 17967571: [openshift-4.12] ose-cluster-api can't be built hermetically
Description: This is a clone of issue OCPBUGS-77975. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77974. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77973. The following is the description of the original issue: --- Description of problem:
ART can't build ose-cluster-api hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-72595 | issue | 4 weeks ago | Ginkgo test failure: [Apiserver] MicroShift enable SCC admission for pods Verified |
Issue 17751602: Ginkgo test failure: [Apiserver] MicroShift enable SCC admission for pods
Description: This is a clone of issue OCPBUGS-72360. The following is the description of the original issue: --- Description of problem:
{code:none}
"[sig-api-machinery] API_Server on Microshift [It] Author:rgangwar-MicroShiftBoth-ConnectedOnly-Medium-55394-[Apiserver] MicroShift enable SCC admission for pods test" failed because the pod is created with a missing fsGroup
{code}
Version-Release number of selected component (if applicable):
{code:none}
4.21.0-rc.1 on x86. This is not reproducible on ARM architecture.{code}
How reproducible:
{code:none}
always on 4.21.0-rc.1{code}
Steps to Reproduce:
{code:none}
1. Run Ginkgo test case: "[sig-api-machinery] API_Server on Microshift [It] Author:rgangwar-MicroShiftBoth-ConnectedOnly-Medium-55394-[Apiserver] MicroShift enable SCC admission for pods test"
2.
3.
{code}
Actual results:
{code:none}
• [PANICKED] [222.518 seconds]
[sig-api-machinery] API_Server on Microshift [It] Author:rgangwar-MicroShiftBoth-ConnectedOnly-Medium-55394-[Apiserver] MicroShift enable SCC admission for pods
/tmp/tmp.I4ekJytyWF/openshift-tests-private/test/extended/apiserverauth/apiserver_microshift.go:480
Timeline >>
STEP: Creating a kubernetes client @ 01/08/26 04:07:05.353
I0108 04:07:05.368507 507110 client.go:486] Creating namespace "e2e-test-default-wb9lb"
I0108 04:07:05.380389 507110 client.go:491] Waiting for ServiceAccount "default" to be provisioned...
I0108 04:07:05.485060 507110 client.go:495] Configuring kubeconfig with user "e2e-test-default-wb9lb-user" certificates...
I0108 04:07:07.078970 507110 client.go:293] configPath is now "/tmp/configfile4051781145"
I0108 04:07:07.079012 507110 client.go:498] Waiting for RoleBinding "e2e-test-default-wb9lb-user" to be provisioned...
I0108 04:07:07.084506 507110 client.go:502] Waiting on permissions in namespace "e2e-test-default-wb9lb" ...
I0108 04:07:07.364467 507110 client.go:521] Namespace "e2e-test-default-wb9lb" has been fully provisioned.
I0108 04:07:07.364710 507110 apiserver_microshift.go:37] The cluster should be healthy before running case.
namespace/r6c171tq created
namespace "r6c171tq" deleted
I0108 04:07:13.076170 507110 apiserver_util.go:1892] Cluster sanity check passed
STEP: 1.Create temporary namespace @ 01/08/26 04:07:13.076
STEP: 2. Create one pod security-context-demo-ocp55394 with the specified security context. @ 01/08/26 04:07:13.183
I0108 04:07:13.183100 507110 apiserver_util.go:460] apiserver fixture dir is not initialized, start to create
I0108 04:07:13.187939 507110 apiserver_util.go:463] apiserver fixture dir is initialized: /tmp/fixture-testdata-dir2892104936/test/extended/testdata/apiserverauth
pod/security-context-demo-ocp55394 created
STEP: 3. Verify that the Pod's security-context-demo-ocp55394 Container is running @ 01/08/26 04:07:13.306
I0108 04:07:23.409925 507110 pods.go:176] Pod security-context-demo-ocp55394 is ready!
STEP: 4.1 Verify if the processes are running with the specified user ID 1000 in the pod. @ 01/08/26 04:07:23.41
I0108 04:07:38.587390 507110 apiserver_util.go:801] Attempting to execute command on pod security-context-demo-ocp55394. Output: PID USER TIME COMMAND
1 1000 0:00 sleep 1h
2 1000 0:00 ps, Error: <nil>
I0108 04:07:38.587444 507110 apiserver_util.go:832] Successfully retrieved non-empty output from pod security-context-demo-ocp55394: PID USER TIME COMMAND
1 1000 0:00 sleep 1h
2 1000 0:00 ps
I0108 04:07:38.587500 507110 apiserver_microshift.go:506] Processes are running on pod security-context-demo-ocp55394 with user id 1000 :: PID USER TIME COMMAND
1 1000 0:00 sleep 1h
2 1000 0:00 ps
STEP: 4.2 Verify that user is running with specified uid=1000 gid=3000 groups=2000 @ 01/08/26 04:07:38.587
I0108 04:07:53.775700 507110 apiserver_util.go:801] Attempting to execute command on pod security-context-demo-ocp55394. Output: uid=1000(1000) gid=3000 groups=2000,3000, Error: <nil>
I0108 04:07:53.775738 507110 apiserver_util.go:832] Successfully retrieved non-empty output from pod security-context-demo-ocp55394: uid=1000(1000) gid=3000 groups=2000,3000
I0108 04:07:53.775826 507110 apiserver_microshift.go:515] On pod security-context-demo-ocp55394 User running with :: uid=1000(1000) gid=3000 groups=2000,3000
STEP: 5. Create one pod security-context-demo-2-ocp55394 with the specified security context. @ 01/08/26 04:07:53.775
I0108 04:07:53.775904 507110 apiserver_util.go:466] apiserver fixture dir found in cache: /tmp/fixture-testdata-dir2892104936/test/extended/testdata/apiserverauth
pod/security-context-demo-2-ocp55394 created
STEP: 6. Verify that the Pod's security-context-demo-2-ocp55394 Container is running @ 01/08/26 04:07:53.967
I0108 04:08:04.065851 507110 pods.go:176] Pod security-context-demo-2-ocp55394 is ready!
STEP: 7. Verify that processes are running with the specified user ID 2000 in the pod. @ 01/08/26 04:08:04.066
I0108 04:08:19.255042 507110 apiserver_util.go:801] Attempting to execute command on pod security-context-demo-2-ocp55394. Output: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
2000 1 0.0 0.0 705960 768 ? Ssl 04:07 0:00 /hello_openshift
2000 6 0.0 0.0 1704 768 ? Rs 04:08 0:00 ps aux, Error: <nil>
I0108 04:08:19.255086 507110 apiserver_util.go:832] Successfully retrieved non-empty output from pod security-context-demo-2-ocp55394: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
2000 1 0.0 0.0 705960 768 ? Ssl 04:07 0:00 /hello_openshift
2000 6 0.0 0.0 1704 768 ? Rs 04:08 0:00 ps aux
I0108 04:08:19.255128 507110 apiserver_microshift.go:534] Processes are running on pod security-context-demo-2-ocp55394 with user id 2000 :: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
2000 1 0.0 0.0 705960 768 ? Ssl 04:07 0:00 /hello_openshift
2000 6 0.0 0.0 1704 768 ? Rs 04:08 0:00 ps aux
STEP: 8. Ensure that pods in different namespaces are launched with different UIDs. @ 01/08/26 04:08:19.255
STEP: 8.1 Create two different namespaces. @ 01/08/26 04:08:19.255
I0108 04:08:19.291544 507110 apiserver_util.go:466] apiserver fixture dir found in cache: /tmp/fixture-testdata-dir2892104936/test/extended/testdata/apiserverauth
STEP: 8.2 Create pods in both namespaces @ 01/08/26 04:08:19.291
STEP: 8.3 Verify pods should have different UID's and desc in both namespaces. @ 01/08/26 04:08:19.291
pod/pod-for-ping created
I0108 04:08:25.585028 507110 apiserver_util.go:1035] The resource returned:
pod-for-ping
I0108 04:08:25.585071 507110 apiserver_util.go:1263] Namespace testpod-namespace-1 pods are: pod-for-ping
I0108 04:08:35.699575 507110 pods.go:176] Pod pod-for-ping is ready!
I0108 04:08:41.978615 507110 apiserver_util.go:1035] The resource returned:
'pod-for-ping runAsUser: 1000580000 fsGroup: seLinuxOptions: s0:c24,c14
'
pod/pod-for-ping created
I0108 04:08:48.301631 507110 apiserver_util.go:1035] The resource returned:
pod-for-ping
I0108 04:08:48.301670 507110 apiserver_util.go:1263] Namespace testpod-namespace-2 pods are: pod-for-ping
I0108 04:08:58.471873 507110 pods.go:176] Pod pod-for-ping is ready!
I0108 04:09:06.403000 507110 apiserver_util.go:1035] The resource returned:
'pod-for-ping runAsUser: 1000590000 fsGroup: seLinuxOptions: s0:c24,c19
'
pod "security-context-demo-2-ocp55394" deleted from test-scc-ocp55394 namespace
pod "security-context-demo-ocp55394" deleted from test-scc-ocp55394 namespace
namespace "test-scc-ocp55394" deleted
[PANICKED] in [It] - /home/ec2-user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.11.linux-amd64/src/runtime/panic.go:115 @ 01/08/26 04:10:47.819
Jan 8 04:10:47.819: INFO: [1mtest dir /tmp/-OCP-microshift-apiseerver-cases-7zer0254/ is cleaned up[0m
STEP: Collecting events from namespace "e2e-test-default-wb9lb". @ 01/08/26 04:10:47.819
STEP: Found 0 events. @ 01/08/26 04:10:47.83
I0108 04:10:47.833385 507110 resource.go:151] POD NODE PHASE GRACE CONDITIONS
I0108 04:10:47.833421 507110 resource.go:161]
I0108 04:10:47.837817 507110 dump.go:81] skipping dumping cluster info - cluster too large
I0108 04:10:47.857338 507110 client.go:681] Deleted {certificates.k8s.io/v1, Resource=certificatesigningrequests e2e-test-default-wb9lb-user}, err: <nil>
STEP: Destroying namespace "e2e-test-default-wb9lb" for this suite. @ 01/08/26 04:10:47.857
<< Timeline
[PANICKED] Test Panicked
In [It] at: /home/ec2-user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.11.linux-amd64/src/runtime/panic.go:115 @ 01/08/26 04:10:47.819
runtime error: index out of range [6] with length 6
Full Stack Trace
github.com/openshift/openshift-tests-private/test/extended/apiserverauth.init.func3.8()
/tmp/tmp.I4ekJytyWF/openshift-tests-private/test/extended/apiserverauth/apiserver_microshift.go:590 +0x1bcf
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0x0?, 0x0?})
/home/ec2-user/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20240806135314-3946b2b7b2a8/internal/node.go:472 +0x13
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/home/ec2-user/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20240806135314-3946b2b7b2a8/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 1
/home/ec2-user/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20240806135314-3946b2b7b2a8/internal/suite.go:888 +0xd7b
------------------------------
Summarizing 1 Failure:
[PANICKED!] [sig-api-machinery] API_Server on Microshift [It] Author:rgangwar-MicroShiftBoth-ConnectedOnly-Medium-55394-[Apiserver] MicroShift enable SCC admission for pods
/home/ec2-user/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.11.linux-amd64/src/runtime/panic.go:115
Ran 1 of 1 Specs in 222.518 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped{code}
Expected results:
{code:none}
test pass{code}
Additional info:
[ link to ginkgo log|https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-microshift-release-4.21-periodics-e2e-aws-tests-release-periodic/2009098095907835904/artifacts/e2e-aws-tests-release-periodic/openshift-microshift-e2e-metal-tests/artifacts/scenario-info/el96-lrel@ginkgo-tests/ginkgo-results/test-output.log]
{code:none}
{code}
Status: Verified
| |||
| #OCPBUGS-77299 | issue | 11 days ago | [openshift-4.16] openshift-enterprise-hyperkube can't be built hermetically New |
Issue 17932658: [openshift-4.16] openshift-enterprise-hyperkube can't be built hermetically
Description: Description of problem:
ART can't build openshift-enterprise-hyperkube hermetically. Same applies for ose-installer-kube-apiserver-artifacts
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.
{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-59807 | issue | 8 days ago | Inspection error on Dell XR8620t ASSIGNED |
Issue 17173358: Inspection error on Dell XR8620t
Description: Description of problem:
{code:none}
After deprovisioning cluster and deleting BMH for my xr8620t, I recreated the BMH and hit an inspection failure.
errorMessage: 'Failed to inspect hardware. Reason: unable to start inspection: HTTP
POST https://10.x.x.x/redfish/v1/Systems/System.Embedded.1/VirtualMedia/1/Actions/VirtualMedia.InsertMedia
returned code 500. Base.1.12.GeneralError: A general error has occurred. See ExtendedInfo
for more information Extended information: [{''Message'': ''Virtual Media is detached
or Virtual Media devices are already in use.'', ''MessageArgs'': [], ''MessageArgs@odata.count'':
0, ''MessageId'': ''IDRAC.2.9.VRM0021'', ''RelatedProperties'': [], ''RelatedProperties@odata.count'':
0, ''Resolution'': ''Change the Virtual Media Attach Mode to Attach or Auto-Attach.Stop
existing Virtual Media sessions and retry the operation.'', ''Severity'': ''Informational''}]'
errorType: inspection error
I've confirmed that there is no virtual media attached. Setting online=true has no effect, and manually powering on the node from idrac ends up with the node quickly being powered off again.{code}
Version-Release number of selected component (if applicable):
{code:none}
OCP: 4.20.0-0.nightly-2025-07-20-021531
ACM: 2.14.0-131
MCE: 2.9.0-102{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: ASSIGNED
{code:java}
{"level":"info","ts":1765324228.08247,"logger":"provisioner.ironic","msg":"starting provisioning","host":"lab2-bmh~node-4","lastError":"Failed to prepare to deploy. Exception: HTTP POST https://10.237.132.17/redfish/v1/Systems/System.Embedded.1/VirtualMedia/1/Actions/VirtualMedia.InsertMedia returned code 500. Base.1.12.GeneralError: A general error has occurred. See ExtendedInfo for more information Extended information: [{'Message': 'Virtual Media is detached or Virtual Media devices are already in use.', 'MessageArgs': [], 'MessageArgs@odata.count': 0, 'MessageId': 'IDRAC.2.9.VRM0021', 'RelatedProperties': [], 'RelatedProperties@odata.count': 0, 'Resolution': 'Change the Virtual Media Attach Mode to Attach or Auto-Attach.Stop existing Virtual Media sessions and retry the operation.', 'Severity': 'Informational'}]","current":"deploy failed","target":"active","deploy step":{}} {code}
| |||
| #OCPBUGS-76930 | issue | 12 days ago | GCP: Provider produced inconsistent result after apply - google_service_account ASSIGNED |
Issue 17884795: GCP: Provider produced inconsistent result after apply - google_service_account
Description: Description of problem:
{code:none}
The fix given for issue https://issues.redhat.com/browse/OCPBUGS-42014(pointing to same problem) was at service account key creation. but the actual problem lies at service account creation itself.
We suggest upgrading terraform and provider to the latest version in openshift-installer for 4.12+. The underlying issue is already fixed upstream and released in v6.45.0.
Note: we see this problem more often in lower openshift versions 4.12, 4.13 than in 4.16.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
Intermittent{code}
Steps to Reproduce:
{code:none}
1.Install the UPI openshift cluster in GCP platform
2.
3.
{code}
Actual results:
{code:none}
level=error msg=Error: Provider produced inconsistent result after apply
level=error
level=error msg=When applying changes to
level=error msg=module.master.google_service_account.master-node-sa[0], provider
level=error msg="provider[\"openshift/local/google\"]" produced an unexpected new value: Root
level=error msg=resource was present, but now absent.
level=error
level=error msg=This is a bug in the provider, which should be reported in the provider's own{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-77308 | issue | 7 days ago | [openshift-4.22] hermetic build failure - nmstate-console-plugin ON_QA |
Issue 17933419: [openshift-4.22] hermetic build failure - nmstate-console-plugin
Description: Description of problem:
{code:none}
nmstate-console-plugin fails to build in hermetic mode due to yarn package manager incompatibility and Cypress test framework attempting network downloads during build process.
{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift Container Platform 4.14+
nmstate-console-plugin component across all affected versions
{code}
How reproducible:
{code:none}
Always - occurs when converting any nmstate-console-plugin build from network_mode: open to hermetic mode
{code}
Steps to Reproduce:
{code:none}
1. Remove network_mode: open from nmstate-console-plugin.yml configuration
2. Attempt hermetic build via Konflux build system
3. Build fails during npm install phase when Cypress tries to download binaries
{code}
Actual results:
{code:none}
npm error Installing Cypress (version: 12.17.4)
npm error The Cypress App could not be downloaded.
npm error URL: https://download.cypress.io/desktop/12.17.4?platform=linux&arch=x64
npm error Error: getaddrinfo ENOTFOUND download.cypress.io
Build fails because:
- Yarn package manager not supported by cachi2 for hermetic dependency management
- Cypress testing framework requires network access to download.cypress.io
- Custom yarn artifacts and dockerfile modifications conflict with hermetic requirements
{code}
Expected results:
{code:none}
nmstate-console-plugin builds successfully in hermetic mode without requiring network access during build process. Component should use npm package manager and disable Cypress binary
downloads via CYPRESS_INSTALL_BINARY=0 environment variable.
{code}
Additional info:
{code:none}
Root cause: Component uses yarn package manager which is incompatible with cachi2 hermetic builds, plus Cypress test framework downloads binaries during build.
Solution requires:
1. Upstream yarn→npm migration (similar to networking-console-plugin)
2. Set CYPRESS_INSTALL_BINARY=0 to disable test framework downloads
3. Remove yarn-specific artifacts and dockerfile modifications
4. Update ocp-build-data package manager configuration
Reference upstream work: https://github.com/openshift/nmstate-console-plugin/pull/169
Similar successful conversion: networking-console-plugin PRs #347 and #331
Tracking: ART-14616 (parent task with technical analysis)
{code}
Status: ON_QA
| |||
| #OCPBUGS-77995 | issue | 35 hours ago | [openshift-4.13] openshift-enterprise-console-operator can't be built hermetically New |
Issue 17968948: [openshift-4.13] openshift-enterprise-console-operator can't be built hermetically
Description: This is a clone of issue OCPBUGS-77994. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77993. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77992. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77862. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77861. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77860. The following is the description of the original issue: --- Description of problem:
ART can't build openshift-enterprise-console-operator hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-76479 | issue | 13 hours ago | Openshift pods stay in a "CreateContainerError" status and show the executable file not found in $PATH" error. New |
Issue 17856179: Openshift pods stay in a "CreateContainerError" status and show the executable file not found in $PATH" error.
Description: $ oc get pods
NAME READY STATUS RESTARTS AGE
cluster-59fd4ffbfd-pvws6 1/1 Running 0 1d
gitops-plugin-895cc759-wth87 1/1 Running 0 1d
openshift-gitops-application-controller-0 0/1 CrashLoopBackOff 121 1d
openshift-gitops-applicationset-controller-8bb69d85d-xdqzs 0/1 CrashLoopBackOff 311 1d
openshift-gitops-dex-server-5bf4bf4f8c-vd298 1/1 Running 0 20d
openshift-gitops-dex-server-6876d69597-kl8xp 0/1 Init:CreateContainerError 0 1d
openshift-gitops-redis-7848d4b859-t8k2m 1/1 Running 0 20d
openshift-gitops-repo-server-56fdcdd764-w6866 1/1 Running 0 20d
openshift-gitops-repo-server-6f4b8564bd-znhd5 0/1 Init:CreateContainerError 0 1d
openshift-gitops-server-5f87669c76-cfg9w 0/1 CrashLoopBackOff 120 1d
$ oc get events
LAST SEEN TYPE REASON OBJECT MESSAGE
37m Normal Pulled pod/openshift-gitops-application-controller-0 Container image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:69b28d34a1a6ae55a6fcca5980bf4f6bdf5f901240b411c8a56205027671e4af" already present on machine
2m51s Warning BackOff pod/openshift-gitops-application-controller-0 Back-off restarting failed container argocd-application-controller in pod openshift-gitops-application-controller-0_openshift-gitops(c1a7c45c-2772-4010-a2c5-164a18dde3b1)
42m Normal Pulled pod/openshift-gitops-applicationset-controller-8bb69d85d-xdqzs Container image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:69b28d34a1a6ae55a6fcca5980bf4f6bdf5f901240b411c8a56205027671e4af" already present on machine
2m48s Warning BackOff pod/openshift-gitops-applicationset-controller-8bb69d85d-xdqzs Back-off restarting failed container argocd-applicationset-controller in pod openshift-gitops-applicationset-controller-8bb69d85d-xdqzs_openshift-gitops(8105b135-261e-4f08-8e01-864b0ea0ae5d)
7m Normal Pulled pod/openshift-gitops-dex-server-6876d69597-kl8xp Container image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:69b28d34a1a6ae55a6fcca5980bf4f6bdf5f901240b411c8a56205027671e4af" already present on machine
2m30s Warning Failed pod/openshift-gitops-dex-server-6876d69597-kl8xp Error: container create failed: executable file `cp` not found in $PATH: No such file or directory
7m Normal Pulled pod/openshift-gitops-repo-server-6f4b8564bd-znhd5 Container image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:69b28d34a1a6ae55a6fcca5980bf4f6bdf5f901240b411c8a56205027671e4af" already present on machine
2m56s Warning Failed pod/openshift-gitops-repo-server-6f4b8564bd-znhd5 Error: container create failed: executable file `cp` not found in $PATH: No such file or directory
12m Normal Pulled pod/openshift-gitops-server-5f87669c76-cfg9w Container image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:69b28d34a1a6ae55a6fcca5980bf4f6bdf5f901240b411c8a56205027671e4af" already present on machine
2m44s Warning BackOff pod/openshift-gitops-server-5f87669c76-cfg9w Back-off restarting failed container argocd-server in pod openshift-gitops-server-5f87669c76-cfg9w_openshift-gitops(f9a45aac-1af4-47a2-9397-24dc97364798)
$ oc get pods
NAME READY STATUS RESTARTS AGE
cluster-logging-operator-5bff654c87-64xbd 1/1 Running 0 2d
instance-dqg5g 1/1 Running 0 3d
instance-grktk 1/1 Running 0 2d
instance-nkjc5 0/1 CreateContainerError 0 2d
instance-nsvpw 0/1 CreateContainerError 0 2d
instance-rnbhs 1/1 Running 0 2d
$ oc get events
LAST SEEN TYPE REASON OBJECT MESSAGE
1m37s Normal Pulled pod/instance-nkjc5 Container image "registry.redhat.io/openshift-logging/vector-rhel9@sha256:fa2cfa2ed336ce105c8dea5bfe0825407e37ef296193ae162f515213fe43c8d5" already present on machine
56m Warning Failed pod/instance-nkjc5 Error: container create failed: executable file `sh` not found in $PATH: No such file or directory
2m23s Normal Pulled pod/instance-nsvpw Container image "registry.redhat.io/openshift-logging/vector-rhel9@sha256:fa2cfa2ed336ce105c8dea5bfe0825407e37ef296193ae162f515213fe43c8d5" already present on machine
17m Warning Failed pod/instance-nsvpw Error: container create failed: executable file `sh` not found in $PATH: No such file or directory
Similar to this was also experienced in v4.6 for which bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1950536] and is closed.
The temporary workaround is to delete all the images from {{/var/lib/containers/storage}} directories and reboot which doesn't helps as a permanent solution.
Status: New
| |||
| #OCPBUGS-77974 | issue | 15 hours ago | [openshift-4.14] ose-cluster-api can't be built hermetically ON_QA |
Issue 17967569: [openshift-4.14] ose-cluster-api can't be built hermetically
Description: This is a clone of issue OCPBUGS-77973. The following is the description of the original issue: --- Description of problem:
ART can't build ose-cluster-api hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: Verified
| |||
| #OCPBUGS-64929 | issue | 7 days ago | coreos-installer iso kargs show broken on OVE Agent ISO Verified |
Issue 17560657: coreos-installer iso kargs show broken on OVE Agent ISO
Description: In the 4.20.2 build, attempts to inspect or modify kernel arguments (kargs) in the generated OVE ISO fail.
{code:none}
$ coreos-installer iso kargs show agent-ove.x86_64.iso
Error: No karg embed areas found; old or corrupted CoreOS ISO image.{code}
Previously, the appliance build did not use the ISO directly as built, it was unpacked and repacked using osirrox / xorriso. The recent change now uses the ISO produced by the appliance directly, which appears to expose an existing issue in how karg embed areas are handled.
[https://github.com/openshift/agent-installer-utils/commit/6d64a0891291701aa0b38b57c8ac9cfa81a69fe4]
Version-Release number of selected component (if applicable):
{code:none}
4.20.2{code}
How reproducible:
{code:none}
Always {code}
Status: Verified
| |||
| #OCPBUGS-77975 | issue | 40 hours ago | [openshift-4.13] ose-cluster-api can't be built hermetically New |
Issue 17967457: [openshift-4.13] ose-cluster-api can't be built hermetically
Description: This is a clone of issue OCPBUGS-77974. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77973. The following is the description of the original issue: --- Description of problem:
ART can't build ose-cluster-api hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-73848 | issue | 6 weeks ago | oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9 Verified |
Issue 17769418: oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9
Description: The oauth-proxy e2e-component tests fail on OCP 4.22 because RHEL 9 enforces fully qualified container image names ("short name mode").
The backend test image {{nginxdemos/nginx-hello:plain-text}} is rejected by Podman since it cannot determine which registry to pull from.
h2. Version-Release number
4.22
h2. How reproducible
Always
h2. Steps to Reproduce
Run the oauth-proxy e2e-component tests on OCP 4.22
The test creates a pod with a backend container using image {{nginxdemos/nginx-hello:plain-text}}
Observe the pod fails to start.
h2. Actual results
The backend container fails with:
{code:java}
ImageInspectError: short name mode is enforcing, but image name
nginxdemos/nginx-hello:plain-text returns ambiguous list
{code}
The pod never reaches Running state, and all e2e tests fail.
h2. Expected results
The e2e tests should pass. The backend container should start successfully using a fully qualified image reference.
h2. Additional info
_Root cause:_ RHEL 9 enforces "short name mode" for container images, requiring fully qualified registry paths.
_Fix:_ Replace {{nginxdemos/nginx-hello:plain-text}} with {{{}registry.k8s.io/e2e-test-images/agnhost:2.53{}}}, which is:
* Fully qualified (satisfies RHEL 9 short name policy)
* Multi-arch (supports ARM64)
* The standard test image adopted across OpenShift repositories ([openshift/origin PR #29672|https://github.com/openshift/origin/pull/29672])
* Mirrored in OpenShift CI infrastructure
_PR:_ [https://github.com/openshift/oauth-proxy/pull/343]
Status: Verified
| |||
| #OCPBUGS-77764 | issue | 6 days ago | [openshift-4.20] hermetic build failure - node-feature-discovery Verified |
Issue 17953456: [openshift-4.20] hermetic build failure - node-feature-discovery
Description: This is a clone of issue OCPBUGS-77763. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77651. The following is the description of the original issue: --- Description of problem:
{code:none}
Node Feature Discovery build fails in hermetic mode due to non-deterministic binary path handling in Dockerfile. The build uses "go install" command which places binaries in unpredictable GOPATH locations, and the Dockerfile copies from multiple inconsistent paths causing binary path conflicts.
{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift Container Platform 4.19+
node-feature-discovery component across all affected versions
{code}
How reproducible:
{code:none}
Always - when building node-feature-discovery in hermetic mode
{code}
Steps to Reproduce:
{code:none}
1. Build node-feature-discovery container in hermetic mode
2. Dockerfile copies binaries from multiple paths: /go/bin/* and /go/node-feature-discovery/bin/*
3. "go install" command places binaries in non-deterministic GOPATH locations during hermetic build process
{code}
Actual results:
{code:none}
Build fails because binaries are placed in unpredictable locations by "go install" and Dockerfile attempts to copy from multiple conflicting paths
Error: Inconsistent binary placement causing copy failures during Docker image build process when switching from "go install" to "go build" in
hermetic mode
{code}
Expected results:
{code:none}
Build succeeds using deterministic binary paths. Should use "go build" with explicit output paths instead of "go install" and copy binaries from single consistent location in Dockerfile.
{code}
Additional info:
{code:none}
Root cause: "go install" command places binaries in GOPATH-dependent locations that are not deterministic in hermetic builds, unlike "go build"
which allows explicit output path control.
Fix implemented: Remove redundant /go/bin/* copy command from Dockerfile and keep only the specific /go/node-feature-discovery/bin/* path. This
ensures consistent binary placement and eliminates path conflicts during hermetic builds.
Reference upstream work: https://github.com/openshift/node-feature-discovery/pull/163/changes
Related hermetic conversion tracking: ART-14586
{code}
Status: Verified
| |||
| #OCPBUGS-67364 | issue | 7 weeks ago | SignatureValidationFailed when creating a pod after applying an image policy with PKI. New |
Issue 17670663: SignatureValidationFailed when creating a pod after applying an image policy with PKI.
Description: Description of problem:
{code:none}
SignatureValidationFailed when creating a pod after applying an image policy with PKI.
The image used is signed with cosign PKI. (refer to https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-86101){code}
Version-Release number of selected component (if applicable):
{code:none}
4.21.0-0.nightly-2025-12-10-193741{code}
How reproducible:
{code:none}
Always {code}
Steps to Reproduce:
{code:none}
1. Set crio log level as "debug"
#oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
name: set-loglevel
spec:
machineConfigPoolSelector:
matchLabels:
pools.operator.machineconfiguration.openshift.io/worker: ""
containerRuntimeConfig:
logLevel: debug
EOF
2. create a user namespace
% oc new-project testnamespace
3. create an imagepolicy crd
apiVersion: config.openshift.io/v1
kind: ImagePolicy
metadata:
name: byopkipolicy # BYOPKI with Root CA and Intermediate CA
namespace: testnamespace
spec:
scopes:
- quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
policy:
rootOfTrust:
policyType: PKI
pki:
caRootsData: LS0tLS1CRUdJT....Q0FURS0tLS0t
caIntermediatesData: LS0tLS1CRUd....JQ0FURS0tLS0t
pkiCertificateSubject:
email: minmli@redhat.com
hostname: myhost.example.com
4. wait a few minutes, check the policy take effect on worker
sh-5.1# cat /etc/crio/policies/testnamespace.json
sh-5.1# cat /etc/containers/registries.d/sigstore-registries.yaml
docker:
quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3:
use-sigstore-attachments: true
5. create a pod with the signed image
#oc apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
name: testpod-1
labels:
app: testpod
spec:
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: hello-pod
image: quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
ports:
- containerPort: 80
EOF
{code}
Actual results:
{code:none}
5 failed to pull image due to error : SignatureValidationFailed
% oc get pod
NAME READY STATUS RESTARTS AGE
testpod-1 0/1 SignatureValidationFailed 0 5s
% oc describe pod testpod-1
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedInterface 24s multus Add eth0 [10.128.2.22/23] from ovn-kubernetes
Normal BackOff 23s kubelet Back-off pulling image "quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3"
Warning Failed 23s kubelet Error: ImagePullBackOff
Normal Pulling 9s (x2 over 24s) kubelet Pulling image "quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3"
Warning Failed 8s (x2 over 24s) kubelet Failed to pull image "quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3": SignatureValidationFailed: unable to pull image or OCI artifact: pull image err: Source image rejected: Signature for identity "quay.io/minmli/quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" is not accepted; artifact err: provided artifact is a container image
Warning Failed 8s (x2 over 24s) kubelet Error: SignatureValidationFailed
{code}
Expected results:
{code:none}
5 the image can be pulled successfully and the pod became running{code}
Additional info:
{code:none}
On the pod's node, I can pull image successfully using podman or crictl.
sh-5.1# crictl pull quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
Image is up to date for quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
sh-5.1#
sh-5.1# crictl images | grep testpkisignedimage-1
quay.io/minmli/testpkisignedimage-1 <none> 230662f80b00e 1.39MB
sh-5.1# podman pull quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
Trying to pull quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3...
Getting image source signatures
Checking if image destination supports signatures
Copying blob f70adabe43c0 done |
Copying config 230662f80b done |
Writing manifest to image destination
Storing signatures
230662f80b00efe44d7f65d4d111f5fe9914f3dad02e92a9120995059f860408
The crio logs show:
Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.302403078Z" level=info msg="Checking image status: quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="server/image_status.go:31" id=2dca236a-6acc-497f-b2f3-16f5a20e2496 name=/runtime.v1.ImageService/ImageStatus
Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.30259278Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.skip_mount_home=true]quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\" does not resolve to an image ID" file="storage/storage_reference.go:154"
Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.302723181Z" level=debug msg="Can't find quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="server/image_status.go:126" id=2dca236a-6acc-497f-b2f3-16f5a20e2496 name=/runtime.v1.ImageService/ImageStatus
Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.302769481Z" level=info msg="Image quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3 not found" file="server/image_status.go:145" id=2dca236a-6acc-497f-b2f3-16f5a20e2496 name=/runtime.v1.ImageService/ImageStatus
Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.302860582Z" level=info msg="Neither image nor artfiact quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3 found" file="server/image_status.go:47" id=2dca236a-6acc-497f-b2f3-16f5a20e2496 name=/runtime.v1.ImageService/ImageStatus
...
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.708198128Z" level=info msg="Pulling image: quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="server/image_pull.go:44" id=5f889081-5de4-4b13-b48e-1a4a2e2540ce name=/runtime.v1.ImageService/PullImage
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.708241229Z" level=debug msg="Using pull policy path for image quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3: \"\"" file="server/image_pull.go:147" id=5f889081-5de4-4b13-b48e-1a4a2e2540ce name=/runtime.v1.ImageService/PullImage
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750278117Z" level=info msg="Trying to access \"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\"" file="docker/docker_image_src.go:96"
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750438218Z" level=debug msg="Found credentials for quay.io/minmli/testpkisignedimage-1 in credential helper containers-auth.json in file /var/lib/kubelet/config.json" file="config/config.go:263"
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750478118Z" level=debug msg=" Lookaside configuration: using \"docker\" namespace quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="docker/registries_d.go:210"
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750513919Z" level=debug msg=" No signature storage configuration found for quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3, using built-in default file:///var/lib/containers/sigstore" file="docker/registries_d.go:183"
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750597619Z" level=debug msg=" Sigstore attachments: using \"docker\" namespace quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="docker/registries_d.go:243"
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.771852216Z" level=debug msg="GET https://quay.io/v2/auth?account=openshift-release-dev%2Bart_dev_priv_visitor_pull&scope=repository%3Aminmli%2Ftestpkisignedimage-1%3Apull&service=quay.io" file="docker/docker_client.go:845"
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.919422177Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/manifests/sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="docker/docker_client.go:616"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.462189484Z" level=debug msg="IsRunningImageAllowed for image docker:quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="signature/policy_eval.go:274"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.462468686Z" level=debug msg="Downloading /v2/minmli/testpkisignedimage-1/blobs/sha256:230662f80b00efe44d7f65d4d111f5fe9914f3dad02e92a9120995059f860408" file="docker/docker_client.go:1051"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.462543987Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/blobs/sha256:230662f80b00efe44d7f65d4d111f5fe9914f3dad02e92a9120995059f860408" file="docker/docker_client.go:616"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.598953145Z" level=debug msg="Reading /var/lib/containers/sigstore/minmli/testpkisignedimage-1@sha256=7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3/signature-1" file="docker/docker_image_src.go:543"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.599085247Z" level=debug msg="Looking for sigstore attachments in quay.io/minmli/testpkisignedimage-1:sha256-7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3.sig" file="docker/docker_client.go:1142"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.599138947Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/manifests/sha256-7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3.sig" file="docker/docker_client.go:616"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.678931183Z" level=debug msg="Downloading /v2/minmli/testpkisignedimage-1/blobs/sha256:7381d1e6c50aa97c5934e473e226db9a7c159754f7ad0d578f4e64c2852b395c" file="docker/docker_client.go:1051"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.678972183Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/blobs/sha256:7381d1e6c50aa97c5934e473e226db9a7c159754f7ad0d578f4e64c2852b395c" file="docker/docker_client.go:616"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.820416688Z" level=debug msg="Downloading /v2/minmli/testpkisignedimage-1/blobs/sha256:f70adabe43c0cccffbae8785406d490e26855b8748fc982d14bc2b20c778b929" file="docker/docker_client.go:1051"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.820485589Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/blobs/sha256:f70adabe43c0cccffbae8785406d490e26855b8748fc982d14bc2b20c778b929" file="docker/docker_client.go:616"
Dec 12 09:08:40 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:40.178773594Z" level=debug msg="added name \"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\" to image \"230662f80b00efe44d7f65d4d111f5fe9914f3dad02e92a9120995059f860408\"" file="storage/storage_dest.go:1624"
Dec 12 09:08:40 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:40.179327099Z" level=info msg="Pulled image: quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="server/image_pull.go:126" id=5f889081-5de4-4b13-b48e-1a4a2e2540ce name=/runtime.v1.ImageService/PullImage
Dec 12 09:08:40 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:40.1794169Z" level=debug msg="Response: *v1.PullImageResponse: image_ref:\"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\"" file="interceptors/interceptors.go:76" id=5f889081-5de4-4b13-b48e-1a4a2e2540ce name=/runtime.v1.ImageService/PullImage
...
Dec 12 09:08:52 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:52.317771557Z" level=debug msg="IsRunningImageAllowed for image docker:quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="signature/policy_eval.go:274"
Dec 12 09:08:52 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:52.317809258Z" level=debug msg=" Using transport \"docker\" policy section \"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\"" file="signature/policy_eval.go:143"
Dec 12 09:08:52 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:52.319713675Z" level=debug msg="Response error: checking signature of \"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\": verifying signatures: SignatureValidationFailed: Signature for identity \"quay.io/minmli/quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\" is not accepted" file="interceptors/interceptors.go:73" id=91708134-7f8b-4453-9f09-5e881b9fc1ca name=/runtime.v1.RuntimeService/CreateContainer
{code}
Status: New
| |||
| #OCPBUGS-77007 | issue | 7 days ago | [4.19] openshift-cluster-csi-drivers vmware-vsphere-csi-driver-operator-metrics endpoint 8445 port should return 401 instead 500 Verified |
Issue 17895615: [4.19] openshift-cluster-csi-drivers vmware-vsphere-csi-driver-operator-metrics endpoint 8445 port should return 401 instead 500
Description: This is a clone of issue OCPBUGS-75912. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-74569. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-60159. The following is the description of the original issue: --- Description of problem:
see similar bug OCPBUGS-57585
Stole the testing result from [https://github.com/openshift/origin/pull/30014#issuecomment-3153130188]
8445 port is for openshift-cluster-csi-drivers vmware-vsphere-csi-driver-operator-metrics endpoint, did not see this port on other IAAS(example AWS,OSP)checked on vsphere, curl 8445 without auth returns 500(actually it's forbidden),
{noformat}
$ oc get infrastructure/cluster -o jsonpath={.spec.platformSpec.type}
VSphere
$ oc -n openshift-cluster-csi-drivers get ep
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME ENDPOINTS AGE
vmware-vsphere-csi-driver-controller-metrics 10.129.0.17:9203,10.130.0.21:9203,10.129.0.17:9202 + 9 more... 28m
vmware-vsphere-csi-driver-operator-metrics 10.130.0.17:8445 28m
vmware-vsphere-csi-driver-webhook-svc 10.128.0.45:8443,10.130.0.23:8443 28m
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -s -o /dev/null -w '%{http_code}' https://10.130.0.17:8445/metrics
500
{noformat}
return code is 500, expecting 401 forbidden instead of 500.
{code:java}
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k https://10.130.0.17:8445/metrics
Internal Server Error: "/metrics": subjectaccessreviews.authorization.k8s.io is forbidden: User "system:serviceaccount:openshift-cluster-csi-drivers:vmware-vsphere-csi-driver-operator" cannot create resource "subjectaccessreviews" in API group "authorization.k8s.io" at the cluster scope {code}
curl with auth, no issue
{noformat}
$ token=`oc create token prometheus-k8s -n openshift-monitoring`
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://10.130.0.17:8445/metrics' | head -n3
# HELP aggregator_discovery_aggregation_count_total [ALPHA] Counter of number of times discovery was aggregated
# TYPE aggregator_discovery_aggregation_count_total counter
aggregator_discovery_aggregation_count_total 0 {noformat}
[https://github.com/openshift/origin/pull/30014#issuecomment-3154900379]
When the bug is fixed, please remove it from this code snippet to avoid regression.
[https://github.com/openshift/origin/blob/4f183dd3427cffd8d97b44557caa782d65726416/test/extended/prometheus/prometheus.go#L64-L71]
Status: Verified
| |||
| #OCPBUGS-62010 | issue | 8 days ago | BMC F/W update fails on Dell R740, server stuck in power off ASSIGNED |
Issue 17410716: BMC F/W update fails on Dell R740, server stuck in power off
Description: h4. Description of problem:
Follow on from OCPBUGS-62009.
I deleted the BMH and restarted the metal3 pods to recover.
I then reprovisioned the BMH and waited for inspection to finish.
I then updated the hostfirmwarecomponents to just update the BMH.
It actually did update the firmware on the server but metal3 failed to recover the server.
apiVersion: metal3.io/v1alpha1
kind: HostFirmwareComponents
metadata:
creationTimestamp: "2025-09-21T23:12:18Z"
generation: 2
name: r740xdg1
namespace: r740-pool
ownerReferences:
- apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
name: r740xdg1
uid: 9494c650-ad12-4a75-a304-3e9a91b90a1c
resourceVersion: "18510819"
uid: d1077d36-c7cd-4647-a4d1-8c7eab631e5e
spec:
updates:
- component: bmc
url: http://hv14.telco5gran.eng.rdu2.redhat.com:8888/firmware/r740/iDRAC-with-Lifecycle-Controller_Firmware_XTFXJ_WN64_7.00.00.173_A00.EXE
status:
components:
- component: bios
currentVersion: 2.22.2
initialVersion: 2.22.2
- component: bmc
currentVersion: 7.00.00.181
initialVersion: 7.00.00.181
- component: nic:NIC.Integrated.1
currentVersion: 14.32.20.04
initialVersion: 14.32.20.04
conditions:
- lastTransitionTime: "2025-09-21T23:12:19Z"
message: ""
observedGeneration: 2
reason: OK
status: "True"
type: Valid
- lastTransitionTime: "2025-09-21T23:26:16Z"
message: ""
observedGeneration: 2
reason: OK
status: "False"
type: ChangeDetected
lastUpdated: "2025-09-21T23:26:16Z"
updates:
- component: bmc
url: http://hv14.telco5gran.eng.rdu2.redhat.com:8888/firmware/r740/iDRAC-with-Lifecycle-Controller_Firmware_XTFXJ_WN64_7.00.00.173_A00.EXE
The ser ver is stuck in power off and is powered off again even if I manually power cycle it.
The following was seen in the logs
{"level":"info","ts":1758498122.6741993,"logger":"provisioner.ironic","msg":"current provision state","host":"r740-pool~r740xdg1","lastError":"Node ea1c7224-e159-4201-84b0-c28cad19046b failed step \{'args': {'settings': [{'component': 'bmc', 'url': 'http://hv14.telco5gran.eng.rdu2.redhat.com:8888/firmware/r740/iDRAC-with-Lifecycle-Controller_Firmware_XTFXJ_WN64_7.00.00.173_A00.EXE'}]}, 'interface': 'firmware', 'step': 'update', 'abortable': False, 'priority': 0}: Redfish exception occurred. Error: HTTP POST https://10.6.36.10/redfish/v1/SessionService/Sessions returned code 401. Base.1.12.GeneralError: Unable to complete the operation because an invalid username and/or password is entered, and therefore authentication failed. Extended information: [\{'Message': 'Unable to complete the operation because an invalid username and/or password is entered, and therefore authentication failed.', 'MessageArgs': [], 'MessageArgs@odata.count': 0, 'MessageId': 'IDRAC.2.8.SYS415', 'RelatedProperties': [], 'RelatedProperties@odata.count': 0, 'Resolution': 'Enter valid user name and password and retry the operation.', 'Severity': 'Warning'}]","current":"manageable","target":""}
Credentials are obviously ok as we would have not gotten this far with invalid ones.
BMC logs
2025-09-22 00:06:51 LOG007 The previous log entry was repeated 1 times.
2025-09-21 23:41:18 RAC0720 Unable to locate the ISO or IMG image file or folder in the network share location because the file or folder path or the user credentials entered are incorrect.
2025-09-21 23:41:17 USR0030 Successfully logged in using telemetry, from 127.0.0.1 and REDFISH.
2025-09-21 23:41:15 RAC0717 Remote share unmounted successfully.
2025-09-21 23:40:56 USR0031 Unable to log in for NULL from 10.22.88.128 using eHTML5 Virtual Console.
2025-09-21 23:40:56 LOG007 The previous log entry was repeated 1 times.
2025-09-21 23:40:49 USR0030 Successfully logged in using root, from 10.8.53.44 and REDFISH.
2025-09-21 23:40:37 DIS002 Auto Discovery feature disabled.
2025-09-21 23:40:37 RAC0182 The iDRAC firmware was rebooted with the following reason: user initiated.
2025-09-21 23:40:27 IPA0100 The iDRAC IP Address changed from :: to 2620:52:9:1624:f602:70ff:fee4:f7f4.
2025-09-21 23:40:27 IPA0100 The iDRAC IP Address changed from 0.0.0.0 to 10.6.36.10.
2025-09-21 23:40:07 PR36 Version change detected for Lifecycle Controller firmware. Previous version:7.00.00.181, Current version:7.00.00.173
2025-09-21 23:40:02 THRM0008 The UNC Warning threshold limit of the server board inlet temperature sensor is changed to 38.
2025-09-21 23:39:58 PSU0800 Power Supply 2: Status = 0x1, IOUT = 0x0, VOUT= 0x0, TEMP= 0x0, FAN = 0x0, INPUT= 0x0.
2025-09-21 23:39:58 PSU0800 Power Supply 1: Status = 0x1, IOUT = 0x0, VOUT= 0x0, TEMP= 0x0, FAN = 0x0, INPUT= 0x0.
2025-09-21 23:37:38 USR0032 The session for root from 10.8.53.44 using REDFISH is logged off.
2025-09-21 23:37:36 SYS1001 System is turning off.
2025-09-21 23:37:36 SYS1003 System CPU Resetting.
2025-09-21 23:37:29 JCP037 The (installation or configuration) job JID_585155450820 is successfully completed.
2025-09-21 23:37:29 RED063 The iDRAC firmware updated successfully. Previous version: 7.00.00.181, Current version: 7.00.00.173
2025-09-21 23:37:29 RAC0704 Requested system powerdown.
2025-09-21 23:37:26 SUP1906 Firmware update successful.
2025-09-21 23:36:34 SUP1905 Firmware update programming flash.
2025-09-21 23:36:18 SUP1903 Firmware update verify image headers.
2025-09-21 23:36:18 SUP1904 Firmware update checksumming image.
2025-09-21 23:36:17 SUP1911 Firmware update initialization complete.
2025-09-21 23:36:17 SUP1901 Firmware update initializing.
2025-09-21 23:34:34 USR0032 The session for root from 10.8.53.44 using REDFISH is logged off.
2025-09-21 23:33:23 RED002 Package successfully downloaded.
2025-09-21 23:32:46 RED111 Successfully downloaded the update package details 228.097 MB in 16.9493 secs at 13.4576 MBps (107.661 Mbps) [iDRAC-with-Lifecycle-Controller_Firmware_XTFXJ_WN64_7.00.00.173_A00.EXE].
2025-09-21 23:32:29 RED110 Downloading the iDRAC-with-Lifecycle-Controller_Firmware_XTFXJ_WN64_7.00.00.173_A00.EXE update package.
2025-09-21 23:32:25 JCP027 The (installation or configuration) job JID_585155450820 is successfully created on iDRAC.
2025-09-21 23:29:01 CTL129 The boot media of the Controller RAID Controller in Slot 6 is Disk.Virtual.0:RAID.Slot.6-1.
2025-09-21 23:26:06 SYS1003 System CPU Resetting.
2025-09-21 23:25:43 SYS1000 System is turning on.
2025-09-21 23:25:42 RAC0701 Requested system powerup.
Must Gather: [https://drive.google.com/file/d/1TlEg50A2NqIU8gYCBtwxg_MT1P6Jb8iu/view?usp=drive_link]
if this is the same root cause as OCPBUGS-62009, feel free to mark as a dup.
h4. Version-Release number of selected component (if applicable):
4.20-rc.2
h4. How reproducible:
Have not had a BMC upgrade complete on rc2 yet
h4. Steps to Reproduce:
#
#
# ...
h4. Actual results:
h4. Expected results:
h4. Additional info:
Status: ASSIGNED
| |||
| #OCPBUGS-76988 | issue | 5 days ago | BMH stuck in deleting state while performing IBBF Verified |
Issue 17893050: BMH stuck in deleting state while performing IBBF
Description: This is a clone of issue OCPBUGS-59133. The following is the description of the original issue: --- Description of problem:
{code:none}
While attempting to perform IBBF with the siteconfig operator, the BMH for the spoke is stuck in deleting state for too long due to the old BMH being unreachable. This results in the clusterinstance timing out while waiting for the BMH resource to be removed. We should find a way to mitigate this.
{code}
Version-Release number of selected component (if applicable):
{code:java}
4.18.0-0.nightly-2025-04-02-011053 {code}
How reproducible:
{code:none}
100%{code}
Steps to Reproduce:
{code:none}
1. Install an IBI spoke cluster using siteconfig operator
2. Remove old node and attempt to replace with new node through IBBF
3.
{code}
Actual results:
{code:none}
Reinstallation fails due to timeout{code}
Expected results:
{code:none}
Reinstallation succeeds{code}
Additional info:
{code:none}
$ oc get clusterinstance target-0 -o json | jq -r '.status.reinstall.conditions[0]'
{
"lastTransitionTime": "2025-04-04T17:24:45Z",
"message": "Encountered error executing task: Deleting rendered manifests. Error: deletion timeout exceeded for object (BareMetalHost:target-0/target-0-0): Timed out waiting to delete object (BareMetalHost:target-0/target-0-0)",
"reason": "Failed",
"status": "False",
"type": "ReinstallRequestProcessed"
}
$ oc get bmh target-0-0 -o json | jq -r .status.errorMessage
Failed to get power state for node 7e063860-56c7-45ca-8bfa-d8f63997bd10. Error: Redfish exception occurred. Error: Resource https://[fd2e:6f44:5dd8:4::1]:8000/redfish/v1/Systems/ac8d214a-ae10-46f5-a179-208341bcb688 not found {code}
Status: Verified
| |||
| #OCPBUGS-78114 | issue | 16 hours ago | [openshift-4.12] ose-agent-installer-api-server can't be built hermetically POST |
Issue 17972393: [openshift-4.12] ose-agent-installer-api-server can't be built hermetically
Description: This is a clone of issue OCPBUGS-78113. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78112. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78111. The following is the description of the original issue: --- Description of problem:
ART can't build ose-agent-installer-api-server hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-77028 | issue | 6 days ago | [4.19] Agent service is failing with input/output error CLOSED |
Issue 17900200: [4.19] Agent service is failing with input/output error Description: *Description of the problem:* - Unable to discover hosts using BMC because the agent service is failing to start with this error: Error: netavark: failed to start process: fork/exec /usr/libexec/podman/netavark: input/output error *How reproducible:* Always *Steps to reproduce:* 1. Boot a system using BMC/BMH 2. ssh the node and check the agent service logs *Actual results:* The agent service fails *Expected results:* The Agent service should start and the hosts should get discovered. *Additional Info:* This issue is not faced when: 1. Hosts are discovered using a discovery iso 2. When we disable converged_flow Status: CLOSED | |||
| #OCPBUGS-57118 | issue | 8 days ago | certified-operators are failing regularly due to startup probe timing out frequently and generating alert for KubePodCrashLooping New |
Comment 28008468 by Jian Zhang at 2025-09-08T09:33:59.903+0000
{quote}4h14m Warning Failed pod/redhat-operators-rr59t Error: ImagePullBackOff
2h31m Warning Failed pod/redhat-operators-rr59t Failed to pull image "registry.redhat.io/redhat/redhat-operator-index:v4.16": copying system image from manifest list: reading signatures: reading signature from [https://registry.redhat.io/containers/sigstore/redhat/redhat-operator-index@sha256=XXX/signature-1:] status 503 (Service Unavailable)
| |||
| #OCPBUGS-76929 | issue | 2 days ago | GCP: Provider produced inconsistent result after apply - google_service_account Verified |
Issue 17884788: GCP: Provider produced inconsistent result after apply - google_service_account
Description: Description of problem:
{code:none}
The fix given for issue https://issues.redhat.com/browse/OCPBUGS-42014(pointing to same problem) was at service account key creation. but the actual problem lies at service account creation itself.
We suggest upgrading terraform and provider to the latest version in openshift-installer for 4.12+. The underlying issue is already fixed upstream and released in v6.45.0.
Note: we see this problem more often in lower openshift versions 4.12, 4.13 than in 4.16.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
Intermittent{code}
Steps to Reproduce:
{code:none}
1.Install the UPI openshift cluster in GCP platform
2.
3.
{code}
Actual results:
{code:none}
level=error msg=Error: Provider produced inconsistent result after apply
level=error
level=error msg=When applying changes to
level=error msg=module.master.google_service_account.master-node-sa[0], provider
level=error msg="provider[\"openshift/local/google\"]" produced an unexpected new value: Root
level=error msg=resource was present, but now absent.
level=error
level=error msg=This is a bug in the provider, which should be reported in the provider's own{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: Verified
| |||
| #OCPBUGS-77768 | issue | 6 days ago | [openshift-4.12] hypershift can't be built hermetically New |
Issue 17953646: [openshift-4.12] hypershift can't be built hermetically
Description: Description of problem:
ART can't build hypershift hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir. {code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #TRT-2568 | issue | Serial and techpreview-serial jobs failing with Prometheus auth check on CCO metrics endpoint in 4.22 nightly amd64 payload CLOSED | |
Issue 17966625: Serial and techpreview-serial jobs failing with Prometheus auth check on CCO metrics endpoint in 4.22 nightly amd64 payload
Description: h2. Payload Regression PR https://github.com/openshift/cloud-credential-operator/pull/976 is causing blocking job failures in the nightly amd64 payload. h2. Evidence * Payload: [4.22.0-0.nightly-2026-03-08-004901|https://amd64.ocp.releases.ci.openshift.org/releasestream/4.22.0-0.nightly/release/4.22.0-0.nightly-2026-03-08-004901] * Failing blocking jobs: ** [aws-ovn-serial-1of2|https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ovn-serial-1of2/2030491767962341376] ** [aws-ovn-serial-2of2|https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ovn-serial-2of2/2030494424521248768] ** [aws-ovn-techpreview-serial-1of3|https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn-techpreview-serial-1of3/2030513102306414592] ** [aws-ovn-techpreview-serial-2of3|https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn-techpreview-serial-2of3/2030495813175611392] ** [aws-ovn-techpreview-serial-3of3|https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn-techpreview-serial-3of3/2030495769554849792] ** [overall-analysis-all|https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-nightly-4.22-overall-analysis-all/2030574039981887488] (downstream) * Originating payload: 4.22.0-0.nightly-2026-03-07-161029 (first rejected after last accepted 4.22.0-0.nightly-2026-03-07-091925) * Test failure: {{[sig-instrumentation][Late] Platform Prometheus targets should not be accessible without auth}} * Error: {{failed to ensure scraping target of pod openshift-cloud-credential-operator/cco-metrics requires auth: context deadline exceeded}} * The PR removed kube-rbac-proxy from the CCO metrics endpoint and reimplemented RBAC via controller-runtime. The Prometheus auth conformance test detects that the CCO metrics endpoint no longer requires authentication properly. * Revert PR: https://github.com/openshift/cloud-credential-operator/pull/982 h2. Action Revert PR https://github.com/openshift/cloud-credential-operator/pull/976 to restore payload acceptance.
Status: CLOSED
| |||
| #OCPBUGS-32745 | issue | 2 weeks ago | 2 minutes time out degrades machineconfignode resources when pinning release images New |
Issue 15961872: 2 minutes time out degrades machineconfignode resources when pinning release images
Description: Description of problem:{code:none}
{code}
Version-Release number of selected component (if applicable):{code:none}
{code}
How reproducible:{code:none}
{code}
Steps to Reproduce:{code:none}
1Description of problem:{code:none}
If all the images in the pinnedimageset are not pulled before 2 minutes the machineconfignode is degraded
- lastTransitionTime: "2024-04-23T10:30:46Z"
message: 'Error: requeue: prefetching images incomplete after: 2m0s'
reason: PrefetchFailed
status: "True"
type: PinnedImageSetsDegraded
{code}
Version-Release number of selected component (if applicable):{code:none}
pre-merge https://github.com/openshift/machine-config-operator/pull/4303
{code}
How reproducible:{code:none}
Always
{code}
Steps to Reproduce:{code:none}
1. execute this command to fill in the following pinnedimageset
$ oc adm release info -o pullspec | awk '{print " - name: " $1}'
- name: registry.build03.ci.openshift.org/ci-ln-d30q8sk/stable@sha256:7b1b1255645d6fe7a...
- name: registry.build03.ci.openshift.org/ci-ln-d30q8sk/stable@sha256:42141b60a0ecaaab...
....
apiVersion: machineconfiguration.openshift.io/v1alpha1
kind: PinnedImageSet
metadata:
[?4ms:
machineconfiguration.openshift.io/role: worker
name: my-worker-pinned-images-all
spec:
pinnedImages:
- name: registry.build03.ci.openshift.org/ci-ln-d30q8sk/stable@sha256:7b1b1255645d6fe7a...
- name: registry.build03.ci.openshift.org/ci-ln-d30q8sk/stable@sha256:42141b60a0ecaaab...
......
{code}
Actual results:{code:none}
The pinnedimageset will take longer than 2 minutes and the machineconfignodes will be degraded.
{code}
Expected results:{code:none}
It is highly unlikely that MCO will be able to pull all the images in a relase in less than 2 minutes, so this timeout will always degrade MCO when we try to pin the release images.
We expect MCO to be able to pull all the images in a relase without being degraded.
{code}
Additional info:{code:none}
This is already planned to increase the timeout in follow-up PRs. This issue will not block the merge. The main goal of this ticket is not to forget that change and it's verification.
{code}
2.
3.
{code}
Actual results:{code:none}
{code}
Expected results:{code:none}
{code}
Additional info:{code:none}
{code}
Status: New
| |||
| #OCPBUGS-78013 | issue | 34 hours ago | [openshift-4.13] sriov-cni can't be build hermetically POST |
Issue 17969334: [openshift-4.13] sriov-cni can't be build hermetically
Description: This is a clone of issue OCPBUGS-78011. The following is the description of the original issue: --- Description of problem:
ART can't build sriov-cni hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-72604 | issue | 7 weeks ago | "Observe - Metrics" unable to show metrics results Verified |
Issue 17752877: "Observe - Metrics" unable to show metrics results
Description: Description of problem:
checked with 4.22.0-0.nightly-2026-01-11-222452 on chrome Version 143.0.7499.169 (Official Build) (64-bit), admin user goes to "Observe - Metrics", search metrics cluster_infrastructure_provider first, it can show results, search another metrics, example prometheus_build_info, the result page is empty, see screen recording(sometimes always see the error for any metrics):
[https://drive.google.com/file/d/1LhRIueFTfsI1jG3a7f2ptO-Ap7MAqRN_/view?usp=drive_link]
error is [https://drive.google.com/file/d/1O0RZT5qlwylTQaBkN1O3ylgdH37t1hjZ/view?usp=drive_link]
{code:none}
5643-chunk-6a634255311d2aaad42a.min.js:1 Caught error in a child component: Error: Minified React error #301; visit https://reactjs.org/docs/error-decoder.html?invariant=301 for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
at sa (react-dom.production.min.js:157:181)
at Va (react-dom.production.min.js:180:154)
at Go (react-dom.production.min.js:269:343)
at Ec (react-dom.production.min.js:250:347)
at Sc (react-dom.production.min.js:250:278)
at xc (react-dom.production.min.js:250:138)
at gc (react-dom.production.min.js:243:163)
at react-dom.production.min.js:123:115
at t.unstable_runWithPriority (scheduler.production.min.js:18:343)
at Gi (react-dom.production.min.js:122:325) {componentStack: '\n at Ce (https://console-openshift-console.apps...9a68b1aa4dc782775.min.js:2:29335)\n at Suspense'}{code}
this is a 4.22 regression, no issue with 4.21 and the latest accepted 4.22 build 4.22.0-0.nightly-2026-01-06-164201
Version-Release number of selected component (if applicable):
{code:none}
4.22 only{code}
How reproducible:
{code:none}
always{code}
Steps to Reproduce:
{code:none}
1. see the description{code}
Actual results:
{code:none}
"Observe - Metrics" unable to show metrics results {code}
Expected results:
{code:none}
show results{code}
Status: Verified
| |||
| #OCPBUGS-77992 | issue | 35 hours ago | [openshift-4.16] openshift-enterprise-console-operator can't be built hermetically New |
Issue 17968937: [openshift-4.16] openshift-enterprise-console-operator can't be built hermetically
Description: This is a clone of issue OCPBUGS-77862. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77861. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77860. The following is the description of the original issue: --- Description of problem:
ART can't build openshift-enterprise-console-operator hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-14473 | issue | 5 weeks ago | Cannot read properties of undefined (reading 'description') error when visiting Overview page ASSIGNED |
Issue 15295508: Cannot read properties of undefined (reading 'description') error when visiting Overview page
Description: Description of problem:
{code:none}
visit Home -> Overview page, keep the window open for a long time, then we can see an error page displayed
We have seen this error several times, just not sure what is exactly the trigger{code}
Version-Release number of selected component (if applicable):
{code:none}
4.14.0-0.nightly-2023-05-31-080250{code}
How reproducible:
{code:none}
Sometimes, not sure what triggers the error{code}
Steps to Reproduce:
{code:none}
1. Visit Home -> Overview page and keep the window open for a long time and do nothing
2.
3.
{code}
Actual results:
{code:none}
We can see an error page displayed after sometime
TypeErrorDescription:Cannot read properties of undefined (reading 'description')
Component trace:
at https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1352496
at https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/dashboards~metal3-baremetalhost~namespace~node~node-detail-chunk-92927db820425366ad3d.min.js:1:22265
at p.memo.t.children.e.children.t.reduxes.e.reduxes.t.reduxes.every.reduxID (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:634756)
at S (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:87:86676)
at <anonymous> (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:636622)
at S (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:87:86676)
at <anonymous> (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/dashboards~metal3-baremetalhost~namespace~node~node-detail-chunk-92927db820425366ad3d.min.js:1:588)
at S (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:87:86676)
at div
at u (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:120940)
at div
at p (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1341337)
at article
at d (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:75090)
at C (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/dashboards~metal3-baremetalhost~namespace~node~node-detail-chunk-92927db820425366ad3d.min.js:1:23649)
at div
at u (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:57168)
at div
at u (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:120940)
at div
at u (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:57168)
at div
at u (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:120940)
at div
at t.default (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1343061)
at div
at t.default (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:222744)
at h (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/dashboards-chunk-8ac1ba95f2636c5fc2db.min.js:1:353)
at i (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:367513)
at t (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:21:69403)
at t (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:21:71448)
at Suspense
at div
at https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:772795
at https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/dashboards-chunk-8ac1ba95f2636c5fc2db.min.js:1:15657
at S (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:87:86676)
at s (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:162531)
at t (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:21:69403)
at T
at t (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:21:71448)
at Suspense
at i (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:367513)
at section
at m (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:170067)
at div
at div
at t.a (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1261973)
at div
at div
at c (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:699285)
at d (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:219102)
at div
at d (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:89497)
at c (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1026224)
at https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:374347
at S (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:87:86676)
at main
at div
at v (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:466913)
at div
at div
at c (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:311318)
at div
at div
at c (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:699285)
at d (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:219102)
at div
at d (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendor-patternfly-core-chunk-73fa94b476a411a945fb.min.js:1:89497)
at Jn (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:36:185603)
at t.default (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:781345)
at t.default (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/quick-start-chunk-65e076b79c5e124b90d0.min.js:1:1261)
at s (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:162531)
at t.a (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1340377)
at ee (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1358378)
at St (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:36:142711)
at ee (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1358378)
at ee (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1358378)
at i (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:737017)
at t.a (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1334610)
at t.a (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1334799)
at t.a (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1332284)
at te (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1358509)
at https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1361613
at r (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:36:122340)
at t (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:21:69403)
at t (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:21:71448)
at t (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:21:66008)
at ae (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1361877)
at t.a (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:711369)
at t.a (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:959460)
at s (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:567305)
at t.a (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:135:257438)
at Suspense
Stack trace:
TypeError: Cannot read properties of undefined (reading 'description')
at https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1353099
at https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/main-chunk-d5c8c19ae7b84e8e6960.min.js:1:1353338
at na (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:248:58879)
at za (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:248:68397)
at La (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:248:67731)
at Hs (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:248:114591)
at xl (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:248:98327)
at Cl (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:248:98255)
at _l (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:248:98118)
at pl (https://console-openshift-console.apps.qe-daily1-414-0602.qe.devcluster.openshift.com/static/vendors~main-chunk-24c590a6d12fc8fa6bbf.min.js:248:95105){code}
Expected results:
{code:none}
no errors happen{code}
Additional info:
{code:none}
{code}
Status: ASSIGNED
| |||
| #OCPBUGS-75912 | issue | 5 days ago | [4.20] openshift-cluster-csi-drivers vmware-vsphere-csi-driver-operator-metrics endpoint 8445 port should return 401 instead 500 Verified |
Issue 17839406: [4.20] openshift-cluster-csi-drivers vmware-vsphere-csi-driver-operator-metrics endpoint 8445 port should return 401 instead 500
Description: This is a clone of issue OCPBUGS-74569. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-60159. The following is the description of the original issue: --- Description of problem:
see similar bug OCPBUGS-57585
Stole the testing result from [https://github.com/openshift/origin/pull/30014#issuecomment-3153130188]
8445 port is for openshift-cluster-csi-drivers vmware-vsphere-csi-driver-operator-metrics endpoint, did not see this port on other IAAS(example AWS,OSP)checked on vsphere, curl 8445 without auth returns 500(actually it's forbidden),
{noformat}
$ oc get infrastructure/cluster -o jsonpath={.spec.platformSpec.type}
VSphere
$ oc -n openshift-cluster-csi-drivers get ep
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME ENDPOINTS AGE
vmware-vsphere-csi-driver-controller-metrics 10.129.0.17:9203,10.130.0.21:9203,10.129.0.17:9202 + 9 more... 28m
vmware-vsphere-csi-driver-operator-metrics 10.130.0.17:8445 28m
vmware-vsphere-csi-driver-webhook-svc 10.128.0.45:8443,10.130.0.23:8443 28m
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -s -o /dev/null -w '%{http_code}' https://10.130.0.17:8445/metrics
500
{noformat}
return code is 500, expecting 401 forbidden instead of 500.
{code:java}
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k https://10.130.0.17:8445/metrics
Internal Server Error: "/metrics": subjectaccessreviews.authorization.k8s.io is forbidden: User "system:serviceaccount:openshift-cluster-csi-drivers:vmware-vsphere-csi-driver-operator" cannot create resource "subjectaccessreviews" in API group "authorization.k8s.io" at the cluster scope {code}
curl with auth, no issue
{noformat}
$ token=`oc create token prometheus-k8s -n openshift-monitoring`
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://10.130.0.17:8445/metrics' | head -n3
# HELP aggregator_discovery_aggregation_count_total [ALPHA] Counter of number of times discovery was aggregated
# TYPE aggregator_discovery_aggregation_count_total counter
aggregator_discovery_aggregation_count_total 0 {noformat}
[https://github.com/openshift/origin/pull/30014#issuecomment-3154900379]
When the bug is fixed, please remove it from this code snippet to avoid regression.
[https://github.com/openshift/origin/blob/4f183dd3427cffd8d97b44557caa782d65726416/test/extended/prometheus/prometheus.go#L64-L71]
Status: Verified
| |||
| #OCPBUGS-76596 | issue | 2 weeks ago | Failed to configure pod interface: timed out waiting for OVS port binding New |
Issue 17866577: Failed to configure pod interface: timed out waiting for OVS port binding
Description: Description of problem:
After the OCP cluster upgrade from version 4.17.38 to 4.18.30. We are observing several pods stuck is container creating phase. Following is the sample log message for one of those pods.
{code:java}
Jan 28 10:51:44 helixtest-6w28l-worker-0-q6mwl kubenswrapper[14632]: E0128 10:51:44.775553 14632 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"redis-redis-ha-server-1_helix-itom-test(a82254b6-5c69-4c3b-bb3c-bd6e8aa43c44)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"redis-redis-ha-server-1_helix-itom-test(a82254b6-5c69-4c3b-bb3c-bd6e8aa43c44)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_redis-redis-ha-server-1_helix-itom-test_a82254b6-5c69-4c3b-bb3c-bd6e8aa43c44_0(7b9dfc5ae370376d3e9dff1608bc40a130848aad4c7474c1bad8dfd06aec8b20): error adding pod helix-itom-test_redis-redis-ha-server-1 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"7b9dfc5ae370376d3e9dff1608bc40a130848aad4c7474c1bad8dfd06aec8b20\\\" Netns:\\\"/var/run/netns/453c2a54-81b6-4c59-93ec-f6f613a2bee4\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=helix-itom-test;K8S_POD_NAME=redis-redis-ha-server-1;K8S_POD_INFRA_CONTAINER_ID=7b9dfc5ae370376d3e9dff1608bc40a130848aad4c7474c1bad8dfd06aec8b20;K8S_POD_UID=a82254b6-5c69-4c3b-bb3c-bd6e8aa43c44\\\" Path:\\\"\\\" ERRORED: error configuring pod [helix-itom-test/redis-redis-ha-server-1] networking: [helix-itom-test/redis-redis-ha-server-1/a82254b6-5c69-4c3b-bb3c-bd6e8aa43c44:ovn-kubernetes]: error adding container to network \\\"ovn-kubernetes\\\": CNI request failed with status 400: '[helix-itom-test/redis-redis-ha-server-1 7b9dfc5ae370376d3e9dff1608bc40a130848aad4c7474c1bad8dfd06aec8b20 network default NAD default] [helix-itom-test/redis-redis-ha-server-1 7b9dfc5ae370376d3e9dff1608bc40a130848aad4c7474c1bad8dfd06aec8b20 network default NAD default] failed to configure pod interface: timed out waiting for OVS port binding (ovn-installed) for 0a:58:0a:54:02:3d [10.84.2.61/23]\\n'\\n': StdinData: {\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="helix-itom-test/redis-redis-ha-server-1" podUID="a82254b6-5c69-4c3b-bb3c-bd6e8aa43c44"{code}
After the cluster upgrade, the issue was observed on a few nodes, but not across all nodes. The problem was resolved after restarting the {{ovnkube-node}} pods.
We have observed the same issue on clusters upgraded from version 4.17.38 to 4.18.30.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Upgrade the cluster from 4.17.38 to 4.18.30
2.
3.
Actual results: The pods remain stuck in the creation phase until the {{ovnkube-node}} pods are restarted.
Expected results: The pods should not stuck in creation phase.
Additional info: I'll share the additional details in comments section.
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
# internal CI failure
# customer issue / SD
# internal RedHat testing failure
If it is an internal RedHat testing failure:
* Please share a kubeconfig or creds to a live cluster for the assignee to debug/troubleshoot along with reproducer steps (specially if it's a telco use case like ICNI, secondary bridges or BM+kubevirt).
If it is a CI failure:
* Did it happen in different CI lanes? If so please provide links to multiple failures with the same error instance
* Did it happen in both sdn and ovn jobs? If so please provide links to multiple failures with the same error instance
* Did it happen in other platforms (e.g. aws, azure, gcp, baremetal etc) ? If so please provide links to multiple failures with the same error instance
* When did the failure start happening? Please provide the UTC timestamp of the networking outage window from a sample failure run
* If it's a connectivity issue,
* What is the srcNode, srcIP and srcNamespace and srcPodName?
* What is the dstNode, dstIP and dstNamespace and dstPodName?
* What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
If it is a customer / SD issue:
* Provide enough information in the bug description that Engineering doesn’t need to read the entire case history.
* Don’t presume that Engineering has access to Salesforce.
* Do presume that Engineering will access attachments through supportshell.
* Describe what each relevant attachment is intended to demonstrate (failed pods, log errors, OVS issues, etc).
* Referring to the attached must-gather, sosreport or other attachment, please provide the following details:
** If the issue is in a customer namespace then provide a namespace inspect.
** If it is a connectivity issue:
*** What is the srcNode, srcNamespace, srcPodName and srcPodIP?
*** What is the dstNode, dstNamespace, dstPodName and dstPodIP?
*** What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
*** Please provide the UTC timestamp networking outage window from must-gather
*** Please provide tcpdump pcaps taken during the outage filtered based on the above provided src/dst IPs
** If it is not a connectivity issue:
*** Describe the steps taken so far to analyze the logs from networking components (cluster-network-operator, OVNK, SDN, openvswitch, ovs-configure etc) and the actual component where the issue was seen based on the attached must-gather. Please attach snippets of relevant logs around the window when problem has happened if any.
* When showing the results from commands, include the entire command in the output.
* For OCPBUGS in which the issue has been identified, label with “sbr-triaged”
* For OCPBUGS in which the issue has not been identified and needs Engineering help for root cause, label with “sbr-untriaged”
* Do not set the priority, that is owned by Engineering and will be set when the bug is evaluated
* Note: bugs that do not meet these minimum standards will be closed with label “SDN-Jira-template”
* For guidance on using this template please see
[OCPBUGS Template Training for Networking components|https://source.redhat.com/groups/public/gss-openshift/cee_openshift_wiki/sbr_shift_networking#jive_content_id_Training_Materials]
Status: New
| |||
| #OCPBUGS-78106 | issue | 17 hours ago | [openshift-4.13] cluster-network-operator can't be built hermetically New |
Issue 17972074: [openshift-4.13] cluster-network-operator can't be built hermetically
Description: This is a clone of issue OCPBUGS-78105. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78104. The following is the description of the original issue: --- Description of problem:
ART can't build cluster-network-operator hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
Additional info:
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
# internal CI failure
# customer issue / SD
# internal RedHat testing failure
If it is an internal RedHat testing failure:
* Please share a kubeconfig or creds to a live cluster for the assignee to debug/troubleshoot along with reproducer steps (specially if it's a telco use case like ICNI, secondary bridges or BM+kubevirt).
If it is a CI failure:
* Did it happen in different CI lanes? If so please provide links to multiple failures with the same error instance
* Did it happen in both sdn and ovn jobs? If so please provide links to multiple failures with the same error instance
* Did it happen in other platforms (e.g. aws, azure, gcp, baremetal etc) ? If so please provide links to multiple failures with the same error instance
* When did the failure start happening? Please provide the UTC timestamp of the networking outage window from a sample failure run
* If it's a connectivity issue,
* What is the srcNode, srcIP and srcNamespace and srcPodName?
* What is the dstNode, dstIP and dstNamespace and dstPodName?
* What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
If it is a customer / SD issue:
* Provide enough information in the bug description that Engineering doesn’t need to read the entire case history.
* Don’t presume that Engineering has access to Salesforce.
* Do presume that Engineering will access attachments through supportshell.
* Describe what each relevant attachment is intended to demonstrate (failed pods, log errors, OVS issues, etc).
* Referring to the attached must-gather, sosreport or other attachment, please provide the following details:
** If the issue is in a customer namespace then provide a namespace inspect.
** If it is a connectivity issue:
*** What is the srcNode, srcNamespace, srcPodName and srcPodIP?
*** What is the dstNode, dstNamespace, dstPodName and dstPodIP?
*** What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
*** Please provide the UTC timestamp networking outage window from must-gather
*** Please provide tcpdump pcaps taken during the outage filtered based on the above provided src/dst IPs
** If it is not a connectivity issue:
*** Describe the steps taken so far to analyze the logs from networking components (cluster-network-operator, OVNK, SDN, openvswitch, ovs-configure etc) and the actual component where the issue was seen based on the attached must-gather. Please attach snippets of relevant logs around the window when problem has happened if any.
* When showing the results from commands, include the entire command in the output.
* For OCPBUGS in which the issue has been identified, label with “sbr-triaged”
* For OCPBUGS in which the issue has not been identified and needs Engineering help for root cause, label with “sbr-untriaged”
* Do not set the priority, that is owned by Engineering and will be set when the bug is evaluated
* Note: bugs that do not meet these minimum standards will be closed with label “SDN-Jira-template”
* For guidance on using this template please see
[OCPBUGS Template Training for Networking components|https://source.redhat.com/groups/public/gss-openshift/cee_openshift_wiki/sbr_shift_networking#jive_content_id_Training_Materials]
Status: New
| |||
| #OCPBUGS-73849 | issue | 7 weeks ago | oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9 New |
Issue 17769514: oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9
Description: The oauth-proxy e2e-component tests fail on OCP 4.21 because RHEL 9 enforces fully qualified container image names ("short name mode"). The backend test image {{nginxdemos/nginx-hello:plain-text}} is rejected by Podman since it cannot determine which registry to pull from.
h2. Version-Release number
4.21
h2. How reproducible
Always
h2. Steps to Reproduce
Run the oauth-proxy e2e-component tests on OCP 4.21
The test creates a pod with a backend container using image {{nginxdemos/nginx-hello:plain-text}}
Observe the pod fails to start
h2. Actual results
The backend container fails with:
{code:java}
ImageInspectError: short name mode is enforcing, but image name
nginxdemos/nginx-hello:plain-text returns ambiguous list
{code}
The pod never reaches Running state, and all e2e tests fail.
h2. Expected results
The e2e tests should pass. The backend container should start successfully using a fully qualified image reference.
h2. Additional info
_Root cause:_ RHEL 9 enforces "short name mode" for container images, requiring fully qualified registry paths.
_Fix:_ Replace {{nginxdemos/nginx-hello:plain-text}} with {{{}registry.k8s.io/e2e-test-images/agnhost:2.53{}}}.
_Backport of:_ [OCPBUGS-73848|https://issues.redhat.com/browse/OCPBUGS-73848]
Status: New
| |||
| #OCPBUGS-65559 | issue | 6 days ago | observability tests are failing as loki is no longer a valid exporter for OTEL config file CLOSED |
Issue 17570261: observability tests are failing as loki is no longer a valid exporter for OTEL config file
Description: Description of problem:{code:none}
observability tests are failing as loki is no longer a valid exporter for OTEL config file
{code}
Version-Release number of selected component (if applicable):{code:none}
4.21
{code}
How reproducible:{code:none}
{code}
Steps to Reproduce:{code:none}
1.
2.
3.
{code}
Actual results:{code:none}
Nov 11 16:35:38 el96-src-optional-bootc-host1 opentelemetry-collector[18443]: Error: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):
Nov 11 16:35:38 el96-src-optional-bootc-host1 opentelemetry-collector[18443]: 'exporters' unknown type: "loki" for id: "loki" (valid values: [otlp prometheus prometheusremotewrite kafka awscloudwatchlogs awsemf awsxray loadbalancing debug otlphttp file])
Nov 11 16:35:38 el96-src-optional-bootc-host1 opentelemetry-collector[18443]: 2025/11/11 16:35:38 collector server run finished with error: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):
Nov 11 16:35:38 el96-src-optional-bootc-host1 opentelemetry-collector[18443]: 'exporters' unknown type: "loki" for id: "loki" (valid values: [otlp prometheus prometheusremotewrite kafka awscloudwatchlogs awsemf awsxray loadbalancing debug otlphttp file])
Nov 11 16:35:38 el96-src-optional-bootc-host1 systemd[1]: microshift-observability.service: Main process exited, code=exited, status=1/FAILURE
{code}
Expected results:{code:none}
No failures.
{code}
Additional info:{code:none}
{code}
Status: CLOSED
| |||
| #OCPBUGS-77193 | issue | 2 weeks ago | [openshift-4.19] ose-libvirt-machine-controllers can't be build hermetically New |
Issue 17925857: [openshift-4.19] ose-libvirt-machine-controllers can't be build hermetically
Description: Description of problem:
ART can't build ose-libvirt-machine-controllers hermetically.
The error is the following in the prefetch-dependencies konflux task:
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details. Please try running `go mod vendor` and committing the changes. Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77858 | issue | 2 days ago | [openshift-4.15] oc-mirror can't be built hermetically Verified |
Issue 17959717: [openshift-4.15] oc-mirror can't be built hermetically
Description: This is a clone of issue OCPBUGS-77857. The following is the description of the original issue: --- Description of problem:
ART can't build oc-mirror hermetically
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: Verified
| |||
| #OCPBUGS-75954 | issue | 13 days ago | [4.18] Improper namespace could affect PPC POST |
Issue 17841296: [4.18] Improper namespace could affect PPC
Description: This is a clone of issue OCPBUGS-60218. The following is the description of the original issue: --- Description of problem:
{code:none}
$ podman run --entrypoint performance-profile-creator -v "/root/ppc/must-gather.local.3879427019288622939/":/must-gather:z registry.redhat.io/openshift4/ose-cluster-node-tuning-rhel9-operator:v4.18 info --must-gather-dir-path /must-gather level=info msg="Nodes names targeted by master pool are: " level=info msg="Multiple matches for the specified must gather directory path: /must-gather and suffix: nodes" Error: failed to parse the cluster data: failed to load node's GHW snapshot: can't obtain the node path worker2.ocp4.terryhu.cc: Multiple matches for the specified must gather directory path: /must-gather and suffix: nodes. Expected only one performance-addon-operator-must-gather* directory, please check the must-gather tarball{code}
Version-Release number of selected component (if applicable):
{code:none}
4.18{code}
How reproducible:
{code:none}
100%{code}
Steps to Reproduce:
{code:none}
1. Collect must-gather
2. Create a namespace that has "nodes" as its suffix, such as
$ mkdir /root/ppc/must-gather.local.3879427019288622939/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-a81f2df09a1b9a7ed63a20ced2182b9ba00a4b0ce49a8598d167ba900c87b12a/namespaces/three-nodes
3. Run the PPC info
{code}
Actual results:
{code:none}
Multiple matches for the specified must gather directory path: /must-gather and suffix: nodes. Expected only one performance-addon-operator-must-gather* directory, please check the must-gather tarball{code}
Expected results:
{code:none}
The PPC should only filter the "nodes" under cluster-scoped-resources/core, not cluster-scoped-resources.{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-62309 | issue | 2 weeks ago | [4.22] Unable to provision NVIDIA DGX B200 nodes via ACM, Redfish VirtualMedia InsertMedia request fails with missing UserName/Password parameters Verified |
Issue 17431143: [4.22] Unable to provision NVIDIA DGX B200 nodes via ACM, Redfish VirtualMedia InsertMedia request fails with missing UserName/Password parameters
Description: *Description of problem:*
When attempting to provision NVIDIA DGX B200 nodes through Advanced Cluster Management (ACM), the BareMetalHost (BMH) status hangs at {_}Provisioning{_}.
The Bare Metal Operator (BMO) and Ironic logs show Redfish VirtualMedia mounting failures. The BMC rejects the {{VirtualMedia.InsertMedia}} request with an HTTP 400 error, requiring {{UserName}} and {{Password}} parameters in the request body.
This behavior prevents automated provisioning of DGX B200 nodes.
----
*Version-Release number of selected component (if applicable):*
* OpenShift: 4.18.3 (hub cluster)
* ACM: 2.13
* Ironic/BMO components from OCP 4.18.3 release
----
*How reproducible:*
Always – occurs consistently when attempting to provision NVIDIA DGX B200 nodes.
----
*Steps to Reproduce:*
# Register NVIDIA DGX B200 node in ACM using BareMetalHost (via BMC/Redfish).
# Observe provisioning workflow.
# Check Ironic/BMO logs when provisioning fails.
----
*Actual results:*
* BMH stuck in _Provisioning_ state.
* ISO not mounted via Redfish VirtualMedia.
* BMO/Ironic logs show:
{code:java}
HTTP request: POST https://<bmc>/redfish/v1/Managers/BMC/VirtualMedia/CD1/Actions/VirtualMedia.InsertMedia body: {'Image': 'https://<iso-server>/boot-<uuid>.iso', 'TransferProtocolType': 'HTTPS'}{code}
{{ }}
Response:
{code:java}
status code: 400 error: Base.1.12.GeneralError: The action ... requires the parameter UserName to be present in the request body. MessageId: Base.1.12.ActionParameterMissing{code}
{{ }}
----
*Expected results:*
* Redfish VirtualMedia ISO should mount successfully.
* BMH should proceed past _Provisioning_ and complete host registration.
----
*Additional info:*
* The fix for handling {{UserName}} in Redfish InsertMedia was merged upstream (sushy) and included in OCP 4.18.1 ({{{}Provide vmedia username and password if required{}}}).
* However, in this scenario, the firmware’s response differs: the error lacks a structured {{code}} field, causing credential detection logic to fail.
* Engineering suspects a firmware bug, but BMO/Ironic may require additional handling (e.g., checking {{MessageId=ActionParameterMissing}} or the {{error}} field directly) as a workaround.
Status: Verified
| |||
| #OCPBUGS-77167 | issue | 7 days ago | [openshift-4.22] hermetic build failure - ose-ibm-vpc-block-csi-driver Verified |
Issue 17919616: [openshift-4.22] hermetic build failure - ose-ibm-vpc-block-csi-driver
Description: Description of problem:
{code:none}
IBM VPC Block CSI Driver build fails in hermetic mode due to non-deterministic GOPATH handling in Makefile. The build system rewrites $GOPATH causing the binary output path -o ${GOPATH}/bin/${EXE_DRIVER_NAME} to become invalid, resulting in build failures.{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift 4.19+ (affects hermetic build conversion) {code}
How reproducible:
{code:none}
Always - when building ibm-vpc-block-csi-driver in hermetic mode{code}
Steps to Reproduce:
{code:none}
1. Build ibm-vpc-block-csi-driver container in hermetic mode
2. Makefile uses absolute GOPATH-dependent output path: -o ${GOPATH}/bin/${EXE_DRIVER_NAME}
3. Build system rewrites $GOPATH during hermetic build process {code}
Actual results:
{code:none}
Build fails because binary cannot be found at expected GOPATH location after build system modifies $GOPATH
Error: building at STEP "COPY --from=builder /cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver /bin/ibm-vpc-block-csi-driver": checking on sources under [...]: copier: stat: "/cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver": no such file or directory{code}
Expected results:
{code:none}
Build succeeds using relative path bin/${EXE_DRIVER_NAME} that doesn't depend on GOPATH manipulation, consistent with other CSI drivers like aws-ebs-csi-driver {code}
Additional info:
{code:none}
Fix implemented: Change Makefile to use relative path -o bin/${EXE_DRIVER_NAME} instead of absolute GOPATH-dependent path. This aligns with other CSI drivers and makes builds hermetic-compatible. Comment added to indicate this is a permanent carry patch for OpenShift builds.{code}
Status: Verified
| |||
| #OCPBUGS-77766 | issue | 6 days ago | [openshift-4.19] hermetic build failure - node-feature-discovery Verified |
Issue 17953518: [openshift-4.19] hermetic build failure - node-feature-discovery
Description: This is a clone of issue OCPBUGS-77764. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77763. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77651. The following is the description of the original issue: --- Description of problem:
{code:none}
Node Feature Discovery build fails in hermetic mode due to non-deterministic binary path handling in Dockerfile. The build uses "go install" command which places binaries in unpredictable GOPATH locations, and the Dockerfile copies from multiple inconsistent paths causing binary path conflicts.
{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift Container Platform 4.19+
node-feature-discovery component across all affected versions
{code}
How reproducible:
{code:none}
Always - when building node-feature-discovery in hermetic mode
{code}
Steps to Reproduce:
{code:none}
1. Build node-feature-discovery container in hermetic mode
2. Dockerfile copies binaries from multiple paths: /go/bin/* and /go/node-feature-discovery/bin/*
3. "go install" command places binaries in non-deterministic GOPATH locations during hermetic build process
{code}
Actual results:
{code:none}
Build fails because binaries are placed in unpredictable locations by "go install" and Dockerfile attempts to copy from multiple conflicting paths
Error: Inconsistent binary placement causing copy failures during Docker image build process when switching from "go install" to "go build" in
hermetic mode
{code}
Expected results:
{code:none}
Build succeeds using deterministic binary paths. Should use "go build" with explicit output paths instead of "go install" and copy binaries from single consistent location in Dockerfile.
{code}
Additional info:
{code:none}
Root cause: "go install" command places binaries in GOPATH-dependent locations that are not deterministic in hermetic builds, unlike "go build"
which allows explicit output path control.
Fix implemented: Remove redundant /go/bin/* copy command from Dockerfile and keep only the specific /go/node-feature-discovery/bin/* path. This
ensures consistent binary placement and eliminates path conflicts during hermetic builds.
Reference upstream work: https://github.com/openshift/node-feature-discovery/pull/163/changes
Related hermetic conversion tracking: ART-14586
{code}
Status: Verified
| |||
| #OCPBUGS-77185 | issue | 7 days ago | BMH stuck in deleting state while performing IBBF Verified |
Issue 17925312: BMH stuck in deleting state while performing IBBF
Description: This is a clone of issue OCPBUGS-76988. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-59133. The following is the description of the original issue: --- Description of problem:
{code:none}
While attempting to perform IBBF with the siteconfig operator, the BMH for the spoke is stuck in deleting state for too long due to the old BMH being unreachable. This results in the clusterinstance timing out while waiting for the BMH resource to be removed. We should find a way to mitigate this.
{code}
Version-Release number of selected component (if applicable):
{code:java}
4.18.0-0.nightly-2025-04-02-011053 {code}
How reproducible:
{code:none}
100%{code}
Steps to Reproduce:
{code:none}
1. Install an IBI spoke cluster using siteconfig operator
2. Remove old node and attempt to replace with new node through IBBF
3.
{code}
Actual results:
{code:none}
Reinstallation fails due to timeout{code}
Expected results:
{code:none}
Reinstallation succeeds{code}
Additional info:
{code:none}
$ oc get clusterinstance target-0 -o json | jq -r '.status.reinstall.conditions[0]'
{
"lastTransitionTime": "2025-04-04T17:24:45Z",
"message": "Encountered error executing task: Deleting rendered manifests. Error: deletion timeout exceeded for object (BareMetalHost:target-0/target-0-0): Timed out waiting to delete object (BareMetalHost:target-0/target-0-0)",
"reason": "Failed",
"status": "False",
"type": "ReinstallRequestProcessed"
}
$ oc get bmh target-0-0 -o json | jq -r .status.errorMessage
Failed to get power state for node 7e063860-56c7-45ca-8bfa-d8f63997bd10. Error: Redfish exception occurred. Error: Resource https://[fd2e:6f44:5dd8:4::1]:8000/redfish/v1/Systems/ac8d214a-ae10-46f5-a179-208341bcb688 not found {code}
Status: Verified
| |||
| #OCPBUGS-61601 | issue | 6 days ago | naming and the actual interpretation appear to be misleading for images and image-tags in limitrange New |
Issue 17385394: naming and the actual interpretation appear to be misleading for images and image-tags in limitrange
Description: Description of problem:
{code:none}
naming and the actual interpretation appear to be misleading for images and image-tags in limitrange{code}
Version-Release number of selected component (if applicable):
{code:none}
4.x{code}
How reproducible:
{code:none}
100 %{code}
Steps to Reproduce:
1- create a limitrange using below yaml
{code:java}
apiVersion: v1
kind: LimitRange
metadata:
name: openshift-resource-limits
spec:
limits:
- type: openshift.io/ImageStream
max:
openshift.io/image-tags: 1
openshift.io/images: 2
oc describe limitrange
Name: openshift-resource-limits
Namespace: amuhamme
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
openshift.io/ImageStream openshift.io/image-tags - 1 - - -
openshift.io/ImageStream openshift.io/images - 2 - - -{code}
2- As per above limitrange conf, it should allow maximum one tag's and 2 images per tag
3- Try to create more than one tag and able to create more than one tag
{code:java}
oc get is
NAME IMAGE REPOSITORY TAGS UPDATED
test image-registry.openshift-image-registry.svc:5000/amuhamme/test new,latest 6 seconds ago
test1 image-registry.openshift-image-registry.svc:5000/amuhamme/test1 {code}
3- if i create one more tag (3rd one) its failing with permission denied.
{code:java}
[root@worker-0 ~]# podman push image-registry.openshift-image-registry.svc:5000/amuhamme/test:hi
Getting image source signatures
Copying blob b64aaade7c59 skipped: already exists
Copying blob dc9835b180d8 skipped: already exists
Copying blob ae0badd53767 skipped: already exists
Copying blob 210e6a098c9f skipped: already exists
Copying config a250ef31d2 done |
Writing manifest to image destination
Error: writing manifest: uploading manifest hi to image-registry.openshift-image-registry.svc:5000/amuhamme/test: denied {code}
4- try to push another image (second one) to the tag and see its failing
{code:java}
[root@worker-0 ~]# podman push image-registry.openshift-image-registry.svc:5000/amuhamme/test:latest
Getting image source signatures
Copying blob 210e6a098c9f skipped: already exists
Copying blob ae0badd53767 skipped: already exists
Copying blob c91bbdb1dcb2 done |
Copying blob dc9835b180d8 skipped: already exists
Copying config 6cd179e2e2 done |
Writing manifest to image destination
Error: writing manifest: uploading manifest latest to image-registry.openshift-image-registry.svc:5000/amuhamme/test: denied {code}
It seems that _"image-tags"_ are being treated as the total number of images within a tag, while _"images"_ are being counted as the number of tags per imagestream. This naming and the actual interpretation appear to be misleading.
Actual results:
{code:none}
name and actual considerations are misleading{code}
Expected results:
{code:none}
name and actual considerations should not misleading {code}
Additional info:
{code:none}
{code}
Status: New
Checking if image destination supports signatures
Error: Copying this image would require changing layer representation, which we cannot do: "Would invalidate signatures"
[root@worker-0 ~]# podman push image-registry.openshift-image-registry.svc:5000/amuhamme/test:latest --remove-signatures
Writing manifest to image destination
Error: writing manifest: uploading manifest latest to image-registry.openshift-image-registry.svc:5000/amuhamme/test: denied {code}
Also, seems like this is not respecting by oc image-import command, even manual image push is denied more than one image, i am able to import using oc image-import. see below
| |||
| #OCPBUGS-74407 | issue | 38 hours ago | [4.18] Unable to provision NVIDIA DGX B200 nodes via ACM, Redfish VirtualMedia InsertMedia request fails with missing UserName/Password parameters ASSIGNED |
Issue 17801336: [4.18] Unable to provision NVIDIA DGX B200 nodes via ACM, Redfish VirtualMedia InsertMedia request fails with missing UserName/Password parameters
Description: *Description of problem:*
When attempting to provision NVIDIA DGX B200 nodes through Advanced Cluster Management (ACM), the BareMetalHost (BMH) status hangs at {_}Provisioning{_}.
The Bare Metal Operator (BMO) and Ironic logs show Redfish VirtualMedia mounting failures. The BMC rejects the {{VirtualMedia.InsertMedia}} request with an HTTP 400 error, requiring {{UserName}} and {{Password}} parameters in the request body.
This behavior prevents automated provisioning of DGX B200 nodes.
----
*Version-Release number of selected component (if applicable):*
* OpenShift: 4.18.3 (hub cluster)
* ACM: 2.13
* Ironic/BMO components from OCP 4.18.3 release
----
*How reproducible:*
Always – occurs consistently when attempting to provision NVIDIA DGX B200 nodes.
----
*Steps to Reproduce:*
# Register NVIDIA DGX B200 node in ACM using BareMetalHost (via BMC/Redfish).
# Observe provisioning workflow.
# Check Ironic/BMO logs when provisioning fails.
----
*Actual results:*
* BMH stuck in _Provisioning_ state.
* ISO not mounted via Redfish VirtualMedia.
* BMO/Ironic logs show:
{code:java}
HTTP request: POST https://<bmc>/redfish/v1/Managers/BMC/VirtualMedia/CD1/Actions/VirtualMedia.InsertMedia body: {'Image': 'https://<iso-server>/boot-<uuid>.iso', 'TransferProtocolType': 'HTTPS'}{code}
{{ }}
Response:
{code:java}
status code: 400 error: Base.1.12.GeneralError: The action ... requires the parameter UserName to be present in the request body. MessageId: Base.1.12.ActionParameterMissing{code}
{{ }}
----
*Expected results:*
* Redfish VirtualMedia ISO should mount successfully.
* BMH should proceed past _Provisioning_ and complete host registration.
----
*Additional info:*
* The fix for handling {{UserName}} in Redfish InsertMedia was merged upstream (sushy) and included in OCP 4.18.1 ({{{}Provide vmedia username and password if required{}}}).
* However, in this scenario, the firmware’s response differs: the error lacks a structured {{code}} field, causing credential detection logic to fail.
* Engineering suspects a firmware bug, but BMO/Ironic may require additional handling (e.g., checking {{MessageId=ActionParameterMissing}} or the {{error}} field directly) as a workaround.
Status: ASSIGNED
| |||
| #OCPBUGS-77266 | issue | 13 days ago | dpu-cni, dpu-daemon and dpu-operator can't be built hermetically POST |
Issue 17931079: dpu-cni, dpu-daemon and dpu-operator can't be built hermetically
Description: Description of problem:
ART can't build dpu-operator, dpu-cni and dpu-daemon hermetically. Hermetic builds on Konflux require that go mod vendor does not produce a diff.
{code:none}
diff --git a/vendor/modules.txt b/vendor/modules.txt
index 2950f8c0..c651edc8 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -747,7 +747,6 @@ k8s.io/client-go/util/flowcontrol
k8s.io/client-go/util/homedir
k8s.io/client-go/util/keyutil
k8s.io/client-go/util/retry
-k8s.io/client-go/util/testing
k8s.io/client-go/util/workqueue
# k8s.io/component-base v0.29.0
## explicit; go 1.21 2026-02-24
11:51:31,718 ERROR PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details. Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details. Please try running `go mod vendor` and committing the changes. Note that you may need to `git add --force` ignored files in the vendor/ dir. {code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-78176 | issue | 9 hours ago | [openshift-4.16] ose-azure-cluster-api-controllers can't be built hermetically New |
Issue 17974845: [openshift-4.16] ose-azure-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-77983. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77905. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77904. The following is the description of the original issue: --- Description of problem:
ART can't build ose-azure-cluster-api-controllers
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-71189 | issue | 7 weeks ago | cluster install failed and worker machines stuck in Provisioned on proxy + techpreview cluster Verified |
{code:java}
[~/must-gather.local.7303639993172620028/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-727500d1498ec6cb417e7f7fe80eaaaae551c877c5e456b58c776fb051150431] $ omc -n openshift-machine-config-operator logs machine-config-operator-6f6cb95d74-8blrn -c machine-config-operator 2026-01-07T03:30:39.182782649Z I0107 03:30:39.182742 1 start.go:52] Version: 4.22.0-0.nightly-2026-01-03-160855 (Raw: 6e3697a8, Hash: 9333c5a69d9e8159b98908561d7fb2269b7a9197) 2026-01-07T03:30:39.184776890Z I0107 03:30:39.184732 1 leaderelection.go:121] The leader election gives 4 retries and allows for 30s of clock skew. The kube-apiserver downtime tolerance is 78s. Worst non-graceful lease acquisition is 2m43s. Worst graceful lease acquisition is {26s}. 2026-01-07T03:30:39.185543101Z I0107 03:30:39.185513 1 metrics.go:92] Registering Prometheus metrics 2026-01-07T03:30:39.185599617Z I0107 03:30:39.185581 1 metrics.go:99] Starting metrics listener on 127.0.0.1:8797 2026-01-07T03:30:39.248066417Z I0107 03:30:39.248028 1 leaderelection.go:257] attempting to acquire leader lease openshift-machine-config-operator/machine-config... 2026-01-07T03:30:39.259065477Z I0107 03:30:39.259034 1 leaderelection.go:271] successfully acquired lease openshift-machine-config-operator/machine-config 2026-01-07T03:30:39.290104920Z I0107 03:30:39.290068 1 simple_featuregate_reader.go:171] Starting feature-gate-detector 2026-01-07T03:30:39.298691190Z I0107 03:30:39.298644 1 event.go:377] Event(v1.ObjectReference{Kind:"Node", Namespace:"openshift-machine-config-operator", Name:"ip-10-0-53-145.us-east-2.compute.internal", UID:"4a35db35-5a95-46f2-8ef8-791c8e61f61d", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "AzureWorkloadIdentity", "BootImageSkewEnforcement", "BootcNodeManagement", "BuildCSIVolumes", "CBORServingAndStorage", "CPMSMachineNamePrefix", "CRDCompatibilityRequirementOperator", "ClientsPreferCBOR", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "ConsolePluginContentSecurityPolicy", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "Example", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageModeStatusReporting", "ImageStreamImportMode", "ImageVolume", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "KMSv1", "MachineAPIMigration", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesCPMS", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MetricsCollectionProfiles", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "SELinuxMount", "ServiceAccountTokenNodeBinding", "SignatureStores", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass", "VolumeGroupSnapshot"}, Disabled:[]v1.FeatureGateName{"ClientsAllowCBOR", "ClusterAPIInstall", "ClusterAPIMachineManagementVSphere", "EventedPLEG", "Example2", "ExternalSnapshotMetadata", "HyperShiftOnlyDynamicResourceAllocation", "IngressControllerDynamicConfigurationManager", "MachineAPIOperatorDisableMachineHealthCheckController", "MultiArchInstallAzure", "NetworkConnect", "NewOLMBoxCutterRuntime", "ProvisioningRequestAvailable", "ShortCertRotation"}} 2026-01-07T03:30:39.298746779Z I0107 03:30:39.298690 1 featuregates.go:112] FeatureGates initialized: enabled=[AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk AzureWorkloadIdentity BootImageSkewEnforcement BootcNodeManagement BuildCSIVolumes CBORServingAndStorage CPMSMachineNamePrefix CRDCompatibilityRequirementOperator ClientsPreferCBOR ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterMonitoringConfig ClusterVersionOperatorConfiguration ConsolePluginContentSecurityPolicy DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL Example ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageModeStatusReporting ImageStreamImportMode ImageVolume InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider KMSv1 MachineAPIMigration MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesCPMS ManagedBootImagesvSphere MaxUnavailableStatefulSet MetricsCollectionProfiles MinimumKubeletVersion MixedCPUsAllocation MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate SELinuxMount ServiceAccountTokenNodeBinding SignatureStores SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass VolumeGroupSnapshot], disabled=[ClientsAllowCBOR ClusterAPIInstall ClusterAPIMachineManagementVSphere EventedPLEG Example2 ExternalSnapshotMetadata HyperShiftOnlyDynamicResourceAllocation IngressControllerDynamicConfigurationManager MachineAPIOperatorDisableMachineHealthCheckController MultiArchInstallAzure NetworkConnect NewOLMBoxCutterRuntime ProvisioningRequestAvailable ShortCertRotation] 2026-01-07T03:30:39.768680812Z I0107 03:30:39.768598 1 operator.go:421] Starting MachineConfigOperator 2026-01-07T03:30:39.769692277Z I0107 03:30:39.769670 1 osimagestream_ocp.go:43] Starting building of the OSImageStream instance 2026-01-07T03:30:39.769922691Z I0107 03:30:39.769903 1 event.go:377] Event(v1.ObjectReference{Kind:"ClusterOperator", Namespace:"openshift-machine-config-operator", Name:"machine-config", UID:"17c41ffe-dda1-4394-9647-e05eb8076050", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorVersionChanged' clusteroperator/machine-config started a version change from [] to [{operator 4.22.0-0.nightly-2026-01-03-160855} {operator-image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:114347e569e8c93398dd56ec97f1a4a6be310296e889e2f8e9d4e2c8aa00bfea}] 2026-01-07T03:31:09.781088597Z time="2026-01-07T03:31:09Z" level=warning msg="Failed, retrying in 1s ... (1/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io: i/o timeout" 2026-01-07T03:31:09.781088597Z time="2026-01-07T03:31:09Z" level=warning msg="Failed, retrying in 1s ... (1/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io: i/o timeout" 2026-01-07T03:31:29.784262097Z time="2026-01-07T03:31:29Z" level=warning msg="Failed, retrying in 2s ... (2/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 172.30.0.10:53: read udp 10.130.0.25:38704->172.30.0.10:53: i/o timeout" 2026-01-07T03:31:29.784293967Z time="2026-01-07T03:31:29Z" level=warning msg="Failed, retrying in 2s ... (2/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 172.30.0.10:53: read udp 10.130.0.25:38704->172.30.0.10:53: i/o timeout" 2026-01-07T03:31:41.855214478Z time="2026-01-07T03:31:41Z" level=warning msg="Failed, retrying in 4s ... (3/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 172.30.0.10:53: read udp 10.130.0.25:58883->172.30.0.10:53: read: connection refused" 2026-01-07T03:31:41.855393194Z time="2026-01-07T03:31:41Z" level=warning msg="Failed, retrying in 4s ... (3/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 172.30.0.10:53: read udp 10.130.0.25:58883->172.30.0.10:53: read: connection refused" 2026-01-07T03:31:45.909631456Z time="2026-01-07T03:31:45Z" level=warning msg="Failed, retrying in 8s ... (4/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 172.30.0.10:53: read udp 10.130.0.25:42137->172.30.0.10:53: read: connection refused" 2026-01-07T03:31:45.909661238Z time="2026-01-07T03:31:45Z" level=warning msg="Failed, retrying in 8s ... (4/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp: lookup quay.io on 172.30.0.10:53: read udp 10.130.0.25:42137->172.30.0.10:53: read: connection refused" 2026-01-07T03:32:23.911790239Z time="2026-01-07T03:32:23Z" level=warning msg="Failed, retrying in 16s ... (5/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 34.239.22.218:443: i/o timeout" 2026-01-07T03:32:23.911893557Z time="2026-01-07T03:32:23Z" level=warning msg="Failed, retrying in 16s ... (5/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 34.239.22.218:443: i/o timeout" 2026-01-07T03:33:09.913513194Z time="2026-01-07T03:33:09Z" level=warning msg="Failed, retrying in 32s ... (6/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 54.225.186.3:443: i/o timeout" 2026-01-07T03:33:09.913575620Z time="2026-01-07T03:33:09Z" level=warning msg="Failed, retrying in 32s ... (6/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 54.225.186.3:443: i/o timeout" 2026-01-07T03:34:11.915140223Z time="2026-01-07T03:34:11Z" level=warning msg="Failed, retrying in 1m4s ... (7/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.88.205.176:443: i/o timeout" 2026-01-07T03:34:11.917330967Z time="2026-01-07T03:34:11Z" level=warning msg="Failed, retrying in 1m4s ... (7/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.88.205.176:443: i/o timeout" 2026-01-07T03:34:13.746831592Z I0107 03:34:13.746788 1 operator.go:441] Change observed to kube-apiserver-server-ca 2026-01-07T03:35:39.771942499Z time="2026-01-07T03:35:39Z" level=warning msg="Failed, retrying in 2m8s ... (8/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": context deadline exceeded" 2026-01-07T03:35:39.772002892Z time="2026-01-07T03:35:39Z" level=warning msg="Failed, retrying in 2m8s ... (8/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": context deadline exceeded" 2026-01-07T03:35:39.772002892Z W0107 03:35:39.771978 1 osimagestream.go:170] error fetching OSImageStreams from a source: error inspecting OS image error getting image source for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4446453d8ae3c37706f8982e65b03c661cb6aa664fa2c93e280b6fefee05a00f: error getting image source for //quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4446453d8ae3c37706f8982e65b03c661cb6aa664fa2c93e280b6fefee05a00f: pinging container registry quay.io: Get "https://quay.io/v2/": context deadline exceeded 2026-01-07T03:35:39.772385463Z W0107 03:35:39.772370 1 osimagestream.go:170] error fetching OSImageStreams from a source: pinging container registry registry.ci.openshift.org: Get "https://registry.ci.openshift.org/v2/": context deadline exceeded 2026-01-07T03:35:39.772606165Z I0107 03:35:39.772581 1 event.go:377] Event(v1.ObjectReference{Kind:"ClusterOperator", Namespace:"openshift-machine-config-operator", Name:"machine-config", UID:"17c41ffe-dda1-4394-9647-e05eb8076050", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'OperatorDegraded: OSImageStreamFailed' Unable to apply 4.22.0-0.nightly-2026-01-03-160855: error building the OSImageStream: unable to retrieve any OSImageStream from the configured sources 2026-01-07T03:35:39.780735277Z I0107 03:35:39.780635 1 osimagestream_ocp.go:43] Starting building of the OSImageStream instance 2026-01-07T03:35:39.780926375Z I0107 03:35:39.780899 1 event.go:377] Event(v1.ObjectReference{Kind:"ClusterOperator", Namespace:"openshift-machine-config-operator", Name:"machine-config", UID:"17c41ffe-dda1-4394-9647-e05eb8076050", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorVersionChanged' clusteroperator/machine-config started a version change from [] to [{operator 4.22.0-0.nightly-2026-01-03-160855} {operator-image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:114347e569e8c93398dd56ec97f1a4a6be310296e889e2f8e9d4e2c8aa00bfea}] 2026-01-07T03:36:09.788441114Z time="2026-01-07T03:36:09Z" level=warning msg="Failed, retrying in 1s ... (1/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.89.121.154:443: i/o timeout" 2026-01-07T03:36:09.788441114Z time="2026-01-07T03:36:09Z" level=warning msg="Failed, retrying in 1s ... (1/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.89.121.154:443: i/o timeout" 2026-01-07T03:36:40.790501478Z time="2026-01-07T03:36:40Z" level=warning msg="Failed, retrying in 2s ... (2/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.89.121.154:443: i/o timeout" 2026-01-07T03:36:40.790656389Z time="2026-01-07T03:36:40Z" level=warning msg="Failed, retrying in 2s ... (2/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.89.121.154:443: i/o timeout" 2026-01-07T03:37:12.792224170Z time="2026-01-07T03:37:12Z" level=warning msg="Failed, retrying in 4s ... (3/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 54.225.186.3:443: i/o timeout" 2026-01-07T03:37:12.792224170Z time="2026-01-07T03:37:12Z" level=warning msg="Failed, retrying in 4s ... (3/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 54.225.186.3:443: i/o timeout" 2026-01-07T03:37:46.794170911Z time="2026-01-07T03:37:46Z" level=warning msg="Failed, retrying in 8s ... (4/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.89.119.17:443: i/o timeout" 2026-01-07T03:37:46.794170911Z time="2026-01-07T03:37:46Z" level=warning msg="Failed, retrying in 8s ... (4/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.89.119.17:443: i/o timeout" 2026-01-07T03:38:24.795203614Z time="2026-01-07T03:38:24Z" level=warning msg="Failed, retrying in 16s ... (5/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 54.225.186.3:443: i/o timeout" 2026-01-07T03:38:24.795203614Z time="2026-01-07T03:38:24Z" level=warning msg="Failed, retrying in 16s ... (5/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 54.225.186.3:443: i/o timeout" 2026-01-07T03:39:01.469345834Z E0107 03:39:01.469301 1 leaderelection.go:441] Failed to update lock optimistically: Timeout: request did not complete within requested timeout - context deadline exceeded, falling back to slow path 2026-01-07T03:39:10.796420874Z time="2026-01-07T03:39:10Z" level=warning msg="Failed, retrying in 32s ... (6/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 54.225.186.3:443: i/o timeout" 2026-01-07T03:39:10.796420874Z time="2026-01-07T03:39:10Z" level=warning msg="Failed, retrying in 32s ... (6/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 54.225.186.3:443: i/o timeout" 2026-01-07T03:40:01.470755282Z E0107 03:40:01.470714 1 leaderelection.go:448] error retrieving resource lock openshift-machine-config-operator/machine-config: the server was unable to return a response in the time allotted, but may still be processing the request (get leases.coordination.k8s.io machine-config) 2026-01-07T03:40:12.798360642Z time="2026-01-07T03:40:12Z" level=warning msg="Failed, retrying in 1m4s ... (7/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 44.219.13.75:443: i/o timeout" 2026-01-07T03:40:12.798360642Z time="2026-01-07T03:40:12Z" level=warning msg="Failed, retrying in 1m4s ... (7/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 44.219.13.75:443: i/o timeout" 2026-01-07T03:40:39.787068848Z W0107 03:40:39.787016 1 osimagestream.go:170] error fetching OSImageStreams from a source: error inspecting OS image error getting image source for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4446453d8ae3c37706f8982e65b03c661cb6aa664fa2c93e280b6fefee05a00f: error getting image source for //quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4446453d8ae3c37706f8982e65b03c661cb6aa664fa2c93e280b6fefee05a00f: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 44.219.13.75:443: i/o timeout 2026-01-07T03:40:39.787522935Z W0107 03:40:39.787485 1 osimagestream.go:170] error fetching OSImageStreams from a source: pinging container registry registry.ci.openshift.org: Get "https://registry.ci.openshift.org/v2/": context deadline exceeded 2026-01-07T03:40:39.787734479Z I0107 03:40:39.787700 1 event.go:377] Event(v1.ObjectReference{Kind:"ClusterOperator", Namespace:"openshift-machine-config-operator", Name:"machine-config", UID:"17c41ffe-dda1-4394-9647-e05eb8076050", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'OperatorDegraded: OSImageStreamFailed' Unable to apply 4.22.0-0.nightly-2026-01-03-160855: error building the OSImageStream: unable to retrieve any OSImageStream from the configured sources 2026-01-07T03:40:39.846570394Z I0107 03:40:39.846530 1 osimagestream_ocp.go:43] Starting building of the OSImageStream instance 2026-01-07T03:41:09.853142075Z time="2026-01-07T03:41:09Z" level=warning msg="Failed, retrying in 1s ... (1/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.89.119.17:443: i/o timeout" 2026-01-07T03:41:09.853142075Z time="2026-01-07T03:41:09Z" level=warning msg="Failed, retrying in 1s ... (1/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.89.119.17:443: i/o timeout" 2026-01-07T03:41:40.855420954Z time="2026-01-07T03:41:40Z" level=warning msg="Failed, retrying in 2s ... (2/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 44.219.13.75:443: i/o timeout" 2026-01-07T03:41:40.855420954Z time="2026-01-07T03:41:40Z" level=warning msg="Failed, retrying in 2s ... (2/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 44.219.13.75:443: i/o timeout" 2026-01-07T03:42:12.857492517Z time="2026-01-07T03:42:12Z" level=warning msg="Failed, retrying in 4s ... (3/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 44.219.13.75:443: i/o timeout" 2026-01-07T03:42:12.857492517Z time="2026-01-07T03:42:12Z" level=warning msg="Failed, retrying in 4s ... (3/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 44.219.13.75:443: i/o timeout" 2026-01-07T03:42:46.860653030Z time="2026-01-07T03:42:46Z" level=warning msg="Failed, retrying in 8s ... (4/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.89.121.154:443: i/o timeout" 2026-01-07T03:42:46.860858926Z time="2026-01-07T03:42:46Z" level=warning msg="Failed, retrying in 8s ... (4/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.89.121.154:443: i/o timeout" 2026-01-07T03:43:24.862495147Z time="2026-01-07T03:43:24Z" level=warning msg="Failed, retrying in 16s ... (5/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.89.121.154:443: i/o timeout" 2026-01-07T03:43:24.862495147Z time="2026-01-07T03:43:24Z" level=warning msg="Failed, retrying in 16s ... (5/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.89.121.154:443: i/o timeout" 2026-01-07T03:44:10.864415197Z time="2026-01-07T03:44:10Z" level=warning msg="Failed, retrying in 32s ... (6/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.91.37.92:443: i/o timeout" 2026-01-07T03:44:10.864415197Z time="2026-01-07T03:44:10Z" level=warning msg="Failed, retrying in 32s ... (6/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.91.37.92:443: i/o timeout" 2026-01-07T03:45:12.867431892Z time="2026-01-07T03:45:12Z" level=warning msg="Failed, retrying in 1m4s ... (7/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.208.138.95:443: i/o timeout" 2026-01-07T03:45:12.867431892Z time="2026-01-07T03:45:12Z" level=warning msg="Failed, retrying in 1m4s ... (7/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.208.138.95:443: i/o timeout" 2026-01-07T03:45:39.851781849Z W0107 03:45:39.851735 1 osimagestream.go:170] error fetching OSImageStreams from a source: error inspecting OS image error getting image source for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4446453d8ae3c37706f8982e65b03c661cb6aa664fa2c93e280b6fefee05a00f: error getting image source for //quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4446453d8ae3c37706f8982e65b03c661cb6aa664fa2c93e280b6fefee05a00f: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 3.208.138.95:443: i/o timeout 2026-01-07T03:45:39.852161055Z W0107 03:45:39.852146 1 osimagestream.go:170] error fetching OSImageStreams from a source: pinging container registry registry.ci.openshift.org: Get "https://registry.ci.openshift.org/v2/": context deadline exceeded 2026-01-07T03:45:39.852454419Z I0107 03:45:39.852431 1 osimagestream_ocp.go:43] Starting building of the OSImageStream instance 2026-01-07T03:46:09.856170272Z time="2026-01-07T03:46:09Z" level=warning msg="Failed, retrying in 1s ... (1/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 44.219.13.75:443: i/o timeout" 2026-01-07T03:46:09.856266863Z time="2026-01-07T03:46:09Z" level=warning msg="Failed, retrying in 1s ... (1/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 44.219.13.75:443: i/o timeout" 2026-01-07T03:46:40.858003632Z time="2026-01-07T03:46:40Z" level=warning msg="Failed, retrying in 2s ... (2/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 44.219.13.75:443: i/o timeout" 2026-01-07T03:46:40.858038529Z time="2026-01-07T03:46:40Z" level=warning msg="Failed, retrying in 2s ... (2/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 44.219.13.75:443: i/o timeout" 2026-01-07T03:47:12.859537756Z time="2026-01-07T03:47:12Z" level=warning msg="Failed, retrying in 4s ... (3/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.208.138.95:443: i/o timeout" 2026-01-07T03:47:12.859537756Z time="2026-01-07T03:47:12Z" level=warning msg="Failed, retrying in 4s ... (3/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.208.138.95:443: i/o timeout" 2026-01-07T03:47:46.862650766Z time="2026-01-07T03:47:46Z" level=warning msg="Failed, retrying in 8s ... (4/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.208.138.95:443: i/o timeout" 2026-01-07T03:47:46.862650766Z time="2026-01-07T03:47:46Z" level=warning msg="Failed, retrying in 8s ... (4/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.208.138.95:443: i/o timeout" 2026-01-07T03:48:24.863428543Z time="2026-01-07T03:48:24Z" level=warning msg="Failed, retrying in 16s ... (5/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.91.37.92:443: i/o timeout" 2026-01-07T03:48:24.863428543Z time="2026-01-07T03:48:24Z" level=warning msg="Failed, retrying in 16s ... (5/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.91.37.92:443: i/o timeout" 2026-01-07T03:49:10.865865310Z time="2026-01-07T03:49:10Z" level=warning msg="Failed, retrying in 32s ... (6/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.91.37.92:443: i/o timeout" 2026-01-07T03:49:10.865865310Z time="2026-01-07T03:49:10Z" level=warning msg="Failed, retrying in 32s ... (6/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.91.37.92:443: i/o timeout" 2026-01-07T03:50:12.869451737Z time="2026-01-07T03:50:12Z" level=warning msg="Failed, retrying in 1m4s ... (7/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.208.138.95:443: i/o timeout" 2026-01-07T03:50:12.869451737Z time="2026-01-07T03:50:12Z" level=warning msg="Failed, retrying in 1m4s ... (7/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 3.208.138.95:443: i/o timeout" 2026-01-07T03:50:39.856418285Z W0107 03:50:39.856376 1 osimagestream.go:170] error fetching OSImageStreams from a source: error inspecting OS image error getting image source for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4446453d8ae3c37706f8982e65b03c661cb6aa664fa2c93e280b6fefee05a00f: error getting image source for //quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4446453d8ae3c37706f8982e65b03c661cb6aa664fa2c93e280b6fefee05a00f: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 3.208.138.95:443: i/o timeout 2026-01-07T03:50:39.856923760Z W0107 03:50:39.856890 1 osimagestream.go:170] error fetching OSImageStreams from a source: pinging container registry registry.ci.openshift.org: Get "https://registry.ci.openshift.org/v2/": context deadline exceeded 2026-01-07T03:50:39.859295024Z I0107 03:50:39.857256 1 osimagestream_ocp.go:43] Starting building of the OSImageStream instance 2026-01-07T03:51:09.864502095Z time="2026-01-07T03:51:09Z" level=warning msg="Failed, retrying in 1s ... (1/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.88.205.176:443: i/o timeout" 2026-01-07T03:51:09.864502095Z time="2026-01-07T03:51:09Z" level=warning msg="Failed, retrying in 1s ... (1/50). Error: pinging container registry quay.io: Get \"https://quay.io/v2/\": dial tcp 98.88.205.176:443: i/o timeout" ...
{code}
| |||
| #OCPBUGS-64665 | issue | 3 months ago | kube-descheduler-operator e2e tests failure on s390x CLOSED |
Issue 17541721: kube-descheduler-operator e2e tests failure on s390x
Description: Description of problem:
{code:none}
Observing TestRemoveDuplicates, TestLiveMigrationInBackground, TestFailedPods, TestLowNodeUtilizationKubernetesMetrics, TestTooManyRestarts, TestTopologySpreadConstraint tests failed in e2e tests.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
Every time{code}
Steps to Reproduce:
{code:none}
1. Install descheduler operator.
2. Clone https://github.ibm.com/OpenShift-on-Z/ocp_test_automation -b madeel-pipes
branch
3. Run the descheduler e2e test suit
{code}
Actual results:
{code:none}
TestRemoveDuplicates, TestLiveMigrationInBackground, TestFailedPods, TestLowNodeUtilizationKubernetesMetrics, TestTooManyRestarts, TestTopologySpreadConstraint failed{code}
Expected results:
{code:none}
All the tests in e2e suit should pass.{code}
Additional info:
{code:none}
1. TestRemoveDuplicates - exec container process `/pause`: Exec format error
2. TestLiveMigrationInBackground - Unable to get kubevirt/kubevirt: the server could not find the requested resource (get kubevirts.kubevirt.io kubevirt)
3. TestFailedPods - Error from server (BadRequest): container "pause" in pod "test-failed-pods-default-args-5ll2b" is waiting to start: CreateContainerError - Error: container create failed: executable file `/bin/false` not found: No such file or directory
4. TestLowNodeUtilizationKubernetesMetrics - Error from server (BadRequest): container "pause" in pod "lownodeutilization-kubernetes-metrics-pod-86cc8687d5-8fzkv" is waiting to start: ImageInspectError - Failed to inspect image "": rpc error: code = Unknown desc = short name mode is enforcing, but image name narmidm/k8s-pod-cpu-stressor:latest returns ambiguous list
5. TestTooManyRestarts - Error from server (BadRequest): container "pause" in pod "restart-pod-5f974fbc98-l8kp8" is waiting to start: CreateContainerError - Error: container create failed: executable file `/bin/false` not found: No such file or directory
6. TestTopologySpreadConstraint - exec container process `/pause`: Exec format error{code}
Status: CLOSED
| |||
| #OCPBUGS-74405 | issue | 38 hours ago | [4.20] Unable to provision NVIDIA DGX B200 nodes via ACM, Redfish VirtualMedia InsertMedia request fails with missing UserName/Password parameters Verified |
Issue 17801333: [4.20] Unable to provision NVIDIA DGX B200 nodes via ACM, Redfish VirtualMedia InsertMedia request fails with missing UserName/Password parameters
Description: *Description of problem:*
When attempting to provision NVIDIA DGX B200 nodes through Advanced Cluster Management (ACM), the BareMetalHost (BMH) status hangs at {_}Provisioning{_}.
The Bare Metal Operator (BMO) and Ironic logs show Redfish VirtualMedia mounting failures. The BMC rejects the {{VirtualMedia.InsertMedia}} request with an HTTP 400 error, requiring {{UserName}} and {{Password}} parameters in the request body.
This behavior prevents automated provisioning of DGX B200 nodes.
----
*Version-Release number of selected component (if applicable):*
* OpenShift: 4.18.3 (hub cluster)
* ACM: 2.13
* Ironic/BMO components from OCP 4.18.3 release
----
*How reproducible:*
Always – occurs consistently when attempting to provision NVIDIA DGX B200 nodes.
----
*Steps to Reproduce:*
# Register NVIDIA DGX B200 node in ACM using BareMetalHost (via BMC/Redfish).
# Observe provisioning workflow.
# Check Ironic/BMO logs when provisioning fails.
----
*Actual results:*
* BMH stuck in _Provisioning_ state.
* ISO not mounted via Redfish VirtualMedia.
* BMO/Ironic logs show:
{code:java}
HTTP request: POST https://<bmc>/redfish/v1/Managers/BMC/VirtualMedia/CD1/Actions/VirtualMedia.InsertMedia body: {'Image': 'https://<iso-server>/boot-<uuid>.iso', 'TransferProtocolType': 'HTTPS'}{code}
{{ }}
Response:
{code:java}
status code: 400 error: Base.1.12.GeneralError: The action ... requires the parameter UserName to be present in the request body. MessageId: Base.1.12.ActionParameterMissing{code}
{{ }}
----
*Expected results:*
* Redfish VirtualMedia ISO should mount successfully.
* BMH should proceed past _Provisioning_ and complete host registration.
----
*Additional info:*
* The fix for handling {{UserName}} in Redfish InsertMedia was merged upstream (sushy) and included in OCP 4.18.1 ({{{}Provide vmedia username and password if required{}}}).
* However, in this scenario, the firmware’s response differs: the error lacks a structured {{code}} field, causing credential detection logic to fail.
* Engineering suspects a firmware bug, but BMO/Ironic may require additional handling (e.g., checking {{MessageId=ActionParameterMissing}} or the {{error}} field directly) as a workaround.
Status: Verified
| |||
| #OCPBUGS-77853 | issue | 5 days ago | [openshift-4.18] multus-cni can't be built hermetically POST |
Issue 17959374: [openshift-4.18] multus-cni can't be built hermetically
Description: Description of problem:
ART can't build multus-cni hermetically. Same applies for multus-cni-container-microshift
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir. {code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-61298 | issue | 8 hours ago | [ACM] ACM cannot access ironic-python-agent on dual-stack environment but only IPv6 is unreachable ON_QA |
Issue 17365535: [ACM] ACM cannot access ironic-python-agent on dual-stack environment but only IPv6 is unreachable
Description: Description of problem:
{code:none}
Baremetal host is not detected randomly when we meet the following conditions:
- ACM Hub Cluster is IPv4/IPv6 dual-stack
- Managed cluster and NMStateConfig is IPv4/IPv6 dual-stack
- Managed nodes are added by BMH resource, not discovery ISO
- IPv4 is reachable between ACM and managed nodes
- IPv6 is unreachable between ACM and managed nodes
In this case, when we create a BMH resource, the managed node is not detected well randomly.
ironic-python-agent has some issues, agent.service doesn't start, and Agent resource is not created.
ironic-python-agent sometimes advertises an IPv6 URL to ACM:
~~~
ironic-agent[XXX]: 2025-01-01 00:00:00.000 1 DEBUG ironic_python_agent.ironic_api_client [-] Heartbeat: announcing callback URL https://[2001:db8::1]:9999, API version is 1.68 heartbeat /usr/lib/python3.9/site-packages/ironic_python_agent/ironic_api_client.py:186
~~~
But as IPv6 is unreachable, ACM metal3-ironic shows the following error:
~~~
2025-01-01 00:00:00.000 1 ERROR ironic.drivers.modules.agent_client [-] Failed to connect to the agent running on node XXXXXXXX to collect commands status. Error: HTTPSConnectionPool(host='2001:db8::1', port=9999): Max retries exceeded with url: /v1/commands/ (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0xXXXXXX>, 'Connection to 2001:db8::1 timed out. (connect timeout=60)')): requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='2001:db8::1', port=9999): Max retries exceeded with url: /v1/commands/ (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0xXXXXXX>, 'Connection to 2001:db8::1 timed out. (connect timeout=60)'))
~~~
Then agent.service doesn't start on the managed node and the node is not detected by ACM.
But sometimes ironic-python-agent advertises an IPv4 URL to ACM.
In this case, everything works well.
The issue occurrence is random.
~~~
ironic-agent[XXX]: 2025-01-01 00:00:00.000 1 DEBUG ironic_python_agent.ironic_api_client [-] Heartbeat: announcing callback URL https://192.0.2.1:9999, API version is 1.68 heartbeat /usr/lib/python3.9/site-packages/ironic_python_agent/ironic_api_client.py:186
~~~{code}
Version-Release number of selected component (if applicable):
{code:none}
OCP 4.18
ACM 2.13{code}
How reproducible:
Steps to Reproduce:
{code:none}
1. Prepare ACM hub cluster and managed nodes. IPv4 is reachable between ACM and managed nodes, but IPv6 is not reachable.
2. Create an ACM Hub Cluster with IPv4/IPv6 dual-stack 3. Create NMStateConfig resource with IPv4/IPv6 dual-stack setting for managed nodes
4. Create InfraEnv resource
5. Create a BMH resoruce
6. The managed node starts automatically, but the node is not detected by the above errors randomly{code}
Actual results:
{code:none}
ACM metal3-ironic cannot connect to the managed nodes{code}
Expected results:
{code:none}
It works with IPv4 even if IPv6 is not reachable{code}
Workaround:
{code:none}
We can specify the advertised IP address using advertise_host setting.
The issue can be solved by manually setting it on the managed node.
~~~
# vi /etc/ironic-python-agent.conf
:
[DEFAULT]
advertise_host = 192.0.2.1
:
# systemctl restart ironic-agent.service
~~~
However, we need to make this config change manually on each managed node after the issue occurs.{code}
Additional info:
{code:none}
I checked the source code.
The advertised IP address is determined by the following code:
https://github.com/openstack/ironic-python-agent/blob/master/ironic_python_agent/agent.py#L344
https://github.com/openstack/ironic-python-agent/blob/master/ironic_python_agent/agent.py#L305-L324
~~~
def _find_routable_addr(self):
ips = set()
for api_url in self.api_urls:
ironic_host = urlparse.urlparse(api_url).hostname
# Try resolving it in case it's not an IP address
try:
addrs = socket.getaddrinfo(ironic_host, 0)
except socket.gaierror:
LOG.debug('Could not resolve %s, maybe no DNS', ironic_host)
ips.add(ironic_host)
continue
ips.update(addr for _, _, _, _, (addr, *_) in addrs)
for attempt in range(self.ip_lookup_attempts):
for ironic_host in ips:
found_ip = self._get_route_source(ironic_host)
if found_ip:
return found_ip
time.sleep(self.ip_lookup_sleep)
~~~
api_urls has both IPv4 address and IPv6 address in dual-stack environment
~~~
# cat /etc/ironic-python-agent.conf
[DEFAULT]
api_url = https://192.0.2.2:6385,https://[2001:db8::2]:6385
:
~~~
The above code determines the advertised IP address using "ip route" command.
It doesn't check the actual connectivity.
That's why unreachable IPv6 address can be selected.
Additionally, it has "ips = set()".
As set() doesn't keep the order of its elements, it randomly returns IPv4 address or IPv6 address in the subsequent "for" loop.
If it returns IPv4 first, IPv4 URL is advertised and it works.
If it returns IPv6 first, IPv6 URL is advertised and it doesn't work.
That's why this issue occurrence is random.{code}
Fix ideas:
{code:none}
I came up with some ideas to fix this issue.
What do you think?
A: In the _find_routable_addr() function, check the actual network connectivity, not only checking the routing table.
If it's not reachable, use another IP address.
I think this is the easiest and best fix.
B: Add some parameters to InfraEnv CR to specify which network is used for the provisioning.
If we can specify the reachable network address in the CR, it will avoid the issue.
But I understand that this will be an RFE.
C: Add some parameters to add KernelArguments in BMH resource.
Ironic-python-agent can get setting parameters from kernel arguments, and currently InfraEnv can add KernelArguments to DisocveryISO.
But to avoid the issue, we need to set different values to ipa-advertise-host per BMH.
InfraEnv cannot set different settings per node.
If BMH can have a parameter to set KernelArgs, we will be able to set different ipa-advertise-host per host, and we can avoid the issue.
But I understand that this is also an RFE.{code}
Status: ON_QA
| |||
| #OCPBUGS-77988 | issue | 31 hours ago | Component Readiness: failed to pull test image for DRA test New |
Issue 17968761: Component Readiness: failed to pull test image for DRA test
Description: I0309 06:23:43.998820 3396 dump.go:53] At 2026-03-09 06:13:10 +0000 UTC - event for sample-device-plugin-beta: {daemonset-controller } SuccessfulCreate: Created pod: sample-device-plugin-beta-v7dgf I0309 06:23:43.998837 3396 dump.go:53] At 2026-03-09 06:13:12 +0000 UTC - event for sample-device-plugin-beta-v7dgf: {kubelet worker-0.ostest.test.metalkube.org} Failed: Failed to pull image "virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-registry-k8s-io-e2e-test-images-sample-device-plugin-1-7-ULwza-sZKxhdAQs1": unable to pull image or OCI artifact: pull image err: initializing source docker://virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-registry-k8s-io-e2e-test-images-sample-device-plugin-1-7-ULwza-sZKxhdAQs1: reading manifest e2e-registry-k8s-io-e2e-test-images-sample-device-plugin-1-7-ULwza-sZKxhdAQs1 in virthost.ostest.test.metalkube.org:5000/localimages/local-test-image: manifest unknown; artifact err: get manifest: build image source: reading manifest e2e-registry-k8s-io-e2e-test-images-sample-device-plugin-1-7-ULwza-sZKxhdAQs1 in virthost.ostest.test.metalkube.org:5000/localimages/local-test-image: manifest unknown I0309 06:23:43.998855 3396 dump.go:53] At 2026-03-09 06:13:12 +0000 UTC - event for sample-device-plugin-beta-v7dgf: {kubelet worker-0.ostest.test.metalkube.org} Pulling: Pulling image "virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-registry-k8s-io-e2e-test-images-sample-device-plugin-1-7-ULwza-sZKxhdAQs1" I0309 06:23:43.998873 3396 dump.go:53] At 2026-03-09 06:13:12 +0000 UTC - event for sample-device-plugin-beta-v7dgf: {kubelet worker-0.ostest.test.metalkube.org} Failed: Error: ErrImagePull I0309 06:23:43.998890 3396 dump.go:53] At 2026-03-09 06:13:12 +0000 UTC - event for sample-device-plugin-beta-v7dgf: {multus } AddedInterface: Add eth0 [10.128.2.46/23 fd01:0:0:5::2e/64] from ovn-kubernetes I0309 06:23:43.998908 3396 dump.go:53] At 2026-03-09 06:13:13 +0000 UTC - event for sample-device-plugin-beta-v7dgf: {kubelet worker-0.ostest.test.metalkube.org} BackOff: Back-off pulling image "virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-registry-k8s-io-e2e-test-images-sample-device-plugin-1-7-ULwza-sZKxhdAQs1" I0309 06:23:43.998923 3396 dump.go:53] At 2026-03-09 06:13:13 +0000 UTC - event for sample-device-plugin-beta-v7dgf: {kubelet worker-0.ostest.test.metalkube.org} Failed: Error: ImagePullBackOff I0309 06:23:43.998942 3396 dump.go:53] At 2026-03-09 06:23:12 +0000 UTC - event for dra-test-driver: {replicaset-controller } SuccessfulDelete: Deleted pod: dra-test-driver-hnqkt I0309 06:23:43.998957 3396 dump.go:53] At 2026-03-09 06:23:12 +0000 UTC - event for dra-test-driver-hnqkt: {kubelet worker-0.ostest.test.metalkube.org} Killing: Stopping container pause (_Feel free to update this bug's summary to be more specific._) Component Readiness has found a potential regression in the following test: {code:none}[sig-node] [DRA] kubelet [Feature:DynamicResourceAllocation] on single node supports extended resources together with ResourceClaim [Serial]{code} Test has a 0.00% pass rate, but 95.00% is required. Sample (being evaluated) Release: 4.22 Start Time: 2026-03-02T00:00:00Z End Time: 2026-03-09T12:00:00Z Success Rate: 0.00% Successes: 0 Failures: 9 Flakes: 0 Base (historical) Release: 4.21 Start Time: 2026-01-04T00:00:00Z End Time: 2026-02-03T23:59:59Z Success Rate: 0.00% Successes: 0 Failures: 0 Flakes: 0 View the [test details report|https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/test_details?Architecture=amd64&FeatureSet=techpreview&Installer=ipi&LayeredProduct=none&Network=ovn&Platform=metal&Suite=serial&Topology=ha&Upgrade=none&baseEndTime=2026-02-03T23%3A59%3A59Z&baseRelease=4.21&baseStartTime=2026-01-04T00%3A00%3A00Z&capability=DynamicResourceAllocation&columnGroupBy=Network%2CPlatform%2CTopology&component=Node+%2F+Kubelet&confidence=95&dbGroupBy=Architecture%2CFeatureSet%2CInstaller%2CLayeredProduct%2CNetwork%2CPlatform%2CSuite%2CTopology%2CUpgrade&environment=Architecture%3Aamd64+FeatureSet%3Atechpreview+Installer%3Aipi+LayeredProduct%3Anone+Network%3Aovn+Platform%3Ametal+Suite%3Aserial+Topology%3Aha+Upgrade%3Anone&flakeAsFailure=false&ignoreDisruption=true&ignoreMissing=false&includeMultiReleaseAnalysis=true&includeVariant=Architecture%3Aamd64&includeVariant=Architecture%3Aarm64&includeVariant=Architecture%3Amulti&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Acrun&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=FeatureSet%3Atechpreview&includeVariant=Installer%3Ahypershift&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=JobTier%3Ablocking&includeVariant=JobTier%3Ainforming&includeVariant=JobTier%3Astandard&includeVariant=LayeredProduct%3Anone&includeVariant=LayeredProduct%3Avirt&includeVariant=Network%3Aovn&includeVariant=OS%3Arhcos9&includeVariant=Owner%3Aeng&includeVariant=Owner%3Aservice-delivery&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Arosa&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aexternal&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&includeVariant=Upgrade%3Amicro&includeVariant=Upgrade%3Aminor&includeVariant=Upgrade%3Anone&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2026-03-09T12%3A00%3A00Z&sampleRelease=4.22&sampleStartTime=2026-03-02T00%3A00%3A00Z&testId=openshift-tests%3A822a302164b8b59711e2e93707ba2cf8] for additional context. Filed by: [~cmeadors@redhat.com]
Status: ASSIGNED
| |||
| #OCPBUGS-76318 | issue | 5 days ago | Confusing prompt "groups.user.openshift.io are not enabled on this cluster" Verified |
Issue 17846259: Confusing prompt "groups.user.openshift.io are not enabled on this cluster"
Description: Description of problem:
{code:none}
Confusing prompt "groups.user.openshift.io are not enabled on this cluster"
{code}
Version-Release number of selected component (if applicable):
{code:none}
$ oc version
Client Version: 4.22.0-0.nightly-2026-02-06-032407
Kustomize Version: v5.7.1
Server Version: 4.22.0-0.nightly-2026-02-03-002928
Kubernetes Version: ...
{code}
How reproducible:
{code:none}
Always
{code}
Steps to Reproduce:
{code:none}
1. Launch a 4.22 fresh env
2. $ oc adm groups new testgroup
group.user.openshift.io/testgroup created
3. $ oc adm groups add-users testgruop testuser-1
error: Error: groups.user.openshift.io are not enabled on this cluster.
Is the embeedded OAuth server disabled?.
Original error:
%!w(<nil>)
{code}
Actual results:
{code:none}
Confusing prompt. The Groups are indeed enabled.
{code}
Expected results:
{code:none}
Message should not be incorrect and confusing
{code}
Additional info:
{code:none}
{code}
Status: Verified
| |||
| #OCPBUGS-52324 | issue | 8 days ago | Inspection error when installing baremetal cluster on HP ProLiant DL360 Gen10 hardware running iLO5 v3.08 CLOSED |
Issue 16713141: Inspection error when installing baremetal cluster on HP ProLiant DL360 Gen10 hardware running iLO5 v3.08
Description: Description of problem:
{code:none}
Customer attempted to install baremetal cluster on HP ProLiant DL360 Gen10 hardware running iLO5 v3.08. The cluster install status was hanging on baremetalhost:cp0: inspecting.
The status of the baremetal resource was: 'Failed to inspect hardware. Reason: unable to start inspection: Redfish exception occurred. Error: Setting boot mode to uefi failed for node 813b617e-42f9-48db-99e9-1b2f3a35f94b. Error: HTTP PATCH https://10.64.143.135/redfish/v1/Systems/1 returned code 400. iLO.0.10.ExtendedInfo: See @Message.ExtendedInfo for more information. Extended information: [{''MessageId'': ''iLO.2.24.UnsupportedOperation''}]'
They tried the workaround to manually set the BIOS mode to UEFI first on each node before running the installer (here’s a Bugzilla bug for it https://bugzilla.redhat.com/show_bug.cgi?id=1820337) which says it is resolved but they would like to check if it was tested using a hardware that already had the mode set to UEFI.
HPE support advised customer that ilo API endpoint is READONLY and does not work in any version of ILO5 because ILO 5 implements `BootSourceOverrideMode` as a read-only item that reflects the current boot mode.
Reference: https://servermanagementportal.ext.hpe.com/docs/redfishservices/ilos/ilo5/ilo5_adaptation/#boot-source-override
Slack discussion: https://redhat-internal.slack.com/archives/CFP6ST0A3/p1734579771973629{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: CLOSED
| |||
| #OCPBUGS-77444 | issue | 8 days ago | Topology page loading with error after installed Virtualization operator ASSIGNED |
Issue 17940088: Topology page loading with error after installed Virtualization operator
Description: Description of problem:
{code:none}
Topology page loading with error after installed Virtualization operator{code}
Version-Release number of selected component (if applicable):
{code:none}
4.21.0-0.nightly-2026-02-26-211439 {code}
How reproducible:
{code:none}
Always{code}
Steps to Reproduce:
{code:none}
Make sure Virtualization operator has been installed on the cluster
1. Navigate to Workloads -> Topology page
2. Select a Project (eg: openshift-cnv, or others)
3. Choose list view, Click the resources that list on the page
{code}
Actual results:
{code:none}
Error: Something wrong happened(0 , r.getModifyApplicationAction) is not a function
TypeErrorDescription:(0 , r.getModifyApplicationAction) is not a functionComponent trace:at o (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:186784)
at v (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:187346)
at S (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:184138)
at f (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:1596403)
at div
at o (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3216981)
at div
at o (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3216626)
at div
at o (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3666721)
at div
at o (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3666277)
at div
at l (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3662004)
at div
at l (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3660900)
at section
at v (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3439814)
at g (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3199640)
at div
at p (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:342999)
at div
at d (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/15646-chunk-d51f2bcad5f4e16934fd.min.js:1:57545)
at div
at a (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/15646-chunk-d51f2bcad5f4e16934fd.min.js:1:54932)
at c (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/15646-chunk-d51f2bcad5f4e16934fd.min.js:1:52460)
at div
at div
at l (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:52:160317)
at div
at div
at a (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3609865)
at v (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3284786)
at d (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/15646-chunk-d51f2bcad5f4e16934fd.min.js:1:54256)
at div
at o (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3283992)
at h (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3282768)
at div
at d (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3281064)
at div
at div
at o (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3667597)
at div
at o (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3667206)
at div
at https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/15646-chunk-d51f2bcad5f4e16934fd.min.js:1:44828
at S (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:18753)
at I (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:2414861)
at https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/15646-chunk-d51f2bcad5f4e16934fd.min.js:1:38140
at https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:2392338
at div
at A (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:471365)
at j (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:471805)
at f (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:1596403)
at https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/15646-chunk-d51f2bcad5f4e16934fd.min.js:1:41548
at https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:1175966
at div
at div
at i (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/15646-chunk-d51f2bcad5f4e16934fd.min.js:1:5900)
at f (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/15646-chunk-d51f2bcad5f4e16934fd.min.js:1:65177)
at d (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:854733)
at https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/15646-chunk-d51f2bcad5f4e16934fd.min.js:1:41875
at i (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:277483)
at withFallback()
at r (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/dev-console/code-refs/topologyPage-chunk-e08f224937cee6e37e3e.min.js:1:2180)
at b (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:75336)
at f (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:75681)
at S (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:75855)
at _ (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:51105)
at ne (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:56537)
at Suspense
at i (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:277483)
at m
at section
at v (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3439814)
at W (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:933627)
at main
at div
at div
at div
at o (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3283992)
at h (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3282768)
at div
at d (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3281064)
at div
at div
at w (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3431896)
at div
at l (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3660900)
at c (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:875511)
at div
at o (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3283313)
at div
at div
at o (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3283992)
at h (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3282768)
at div
at d (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3281064)
at Sn (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3115697)
at i (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:54467)
at a (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:54036)
at fe (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:946219)
at tt (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:3074630)
at fe (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:946219)
at fe (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:946219)
at u (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/api/plugins/kubevirt-plugin/exposed-modalProvider-chunk-5f1caebfdb41b6e7076c.min.js:1:423)
at fe (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:946219)
at fe (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:946219)
at Suspense
at f (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:112989)
at b (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:111873)
at a (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:23099)
at u (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:25343)
at ye (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:946335)
at Se (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:949610)
at _ (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:51105)
at ne (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:56537)
at _ (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:51105)
at ne (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:56537)
at te (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:55950)
at G (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:43270)
at t (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:65457)
at xe (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:949902)
at g (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:478028)
at c (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:86080)
at e (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:2594154)
at v (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/main-bundle-6e23ac1e136e272a2f6a.min.js:1:926590)
at c (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:16683)
at SuspenseStack trace:TypeError: (0 , r.getModifyApplicationAction) is not a function
at https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/api/plugins/kubevirt-plugin/exposed-topology-chunk-04d21a7cb3ef7251ab91.min.js:1:65957
at Object.useMemo (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/vendors~main-chunk-37548-cefc1afc0027db257b90.min.js:2:2517670)
at t.useMemo (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:78415)
at m (https://console-openshift-console.apps.qe-uidaily-0227.qe.azure.devcluster.openshift.com/api/plugins/kubevirt-plugin/exposed-{code}
Expected results:
{code:none}
No errors, resources can loaded normally{code}
Additional info:
{code:none}
{code}
Status: ASSIGNED
| |||
| #OCPBUGS-78162 | issue | 9 hours ago | [openshift-4.13] ose-metallb-operator can't be built hermetically POST |
Issue 17974682: [openshift-4.13] ose-metallb-operator can't be built hermetically
Description: This is a clone of issue OCPBUGS-78161. The following is the description of the original issue: --- Description of problem:
ART can't build ose-metallb-operator hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-77214 | issue | 12 days ago | [openshift-4.12] hermetic build failure - ose-ibm-vpc-block-csi-driver ASSIGNED |
Issue 17926364: [openshift-4.12] hermetic build failure - ose-ibm-vpc-block-csi-driver
Description: This is a clone of issue OCPBUGS-77213. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77212. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77211. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77210. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77209. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77207. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77206. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77205. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77203. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77167. The following is the description of the original issue: --- Description of problem:
{code:none}
IBM VPC Block CSI Driver build fails in hermetic mode due to non-deterministic GOPATH handling in Makefile. The build system rewrites $GOPATH causing the binary output path -o ${GOPATH}/bin/${EXE_DRIVER_NAME} to become invalid, resulting in build failures.{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift 4.19+ (affects hermetic build conversion) {code}
How reproducible:
{code:none}
Always - when building ibm-vpc-block-csi-driver in hermetic mode{code}
Steps to Reproduce:
{code:none}
1. Build ibm-vpc-block-csi-driver container in hermetic mode
2. Makefile uses absolute GOPATH-dependent output path: -o ${GOPATH}/bin/${EXE_DRIVER_NAME}
3. Build system rewrites $GOPATH during hermetic build process {code}
Actual results:
{code:none}
Build fails because binary cannot be found at expected GOPATH location after build system modifies $GOPATH
Error: building at STEP "COPY --from=builder /cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver /bin/ibm-vpc-block-csi-driver": checking on sources under [...]: copier: stat: "/cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver": no such file or directory{code}
Expected results:
{code:none}
Build succeeds using relative path bin/${EXE_DRIVER_NAME} that doesn't depend on GOPATH manipulation, consistent with other CSI drivers like aws-ebs-csi-driver {code}
Additional info:
{code:none}
Fix implemented: Change Makefile to use relative path -o bin/${EXE_DRIVER_NAME} instead of absolute GOPATH-dependent path. This aligns with other CSI drivers and makes builds hermetic-compatible. Comment added to indicate this is a permanent carry patch for OpenShift builds.{code}
Status: ASSIGNED
| |||
| #OCPBUGS-39360 | issue | 8 days ago | Many tests failing on DualStack IPv6 primary POST |
{ fail [k8s.io/kubernetes/test/e2e/network/endpointslice.go:611]: EndpointSlice.discovery.k8s.io "e2e-custom-slicel8znj" is invalid: endpoints[0].addresses[0]: Invalid value: "fd01:0:0:4::951": must be a valid IPv4 address
Error: exit with code 1 {code}
We seem to not be informing the tests that v6 is primary.
| |||
| #OCPBUGS-76625 | issue | 11 days ago | Microshift pods fails to start with error container create failed: systemd failed to install eBPF device filter on cgroup New |
Issue 17870004: Microshift pods fails to start with error container create failed: systemd failed to install eBPF device filter on cgroup
Description: Description of problem:{code:none}
When running iso installs and ostree z or y upgrades i see microshift pods goes into createContainerError state and doing a describe on the pods shows error below
openshift-ovn-kubernetes 3h20m Warning Failed pod/ovnkube-node-gscpl (combined from similar events): Error: container create failed: systemd failed to install eBPF device filter on cgroup `/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2535c48_e30b_4620_93c2_afc19d1685c1.slice/crio-34e6ac00324e13e50481a795f54b5a1ace01bf2e8c186a75b407700d0ca66e5b.scope`
{code}
Version-Release number of selected component (if applicable):{code:none}
4.18.33
microshift-4.18.33-202602090801.p0.ge2f5f13.assembly.4.18.33.el9.x86_64
{code}
How reproducible:{code:none}
always
{code}
Steps to Reproduce:{code:none}
1. Trigger iso installs and ostree z and y upgrades with microshift release testing pipeline.
2.
3.
{code}
Actual results:{code:none}
Microshift pods fails to start with error as described in the description
{code}
Expected results:{code:none}
Microshift pods should start fine.
{code}
Additional info:{code:none}
https://redhat-internal.slack.com/archives/C03CJTNLKAT/p1770818291495579
{code}
Status: ASSIGNED
| |||
| #OCPBUGS-74573 | issue | 5 hours ago | Machine creation fails on ASH when using AzureManaged boot diagnostics without proper validation Verified |
Issue 17811325: Machine creation fails on ASH when using AzureManaged boot diagnostics without proper validation
Description: Description of problem:
{code:none}
Before the fix https://github.com/openshift/machine-api-provider-azure/pull/182
When creating a Machine/MachineSet on Azure Stack Hub with diagnostics.boot.storageAccountType: AzureManaged, the Machine creation succeed, but in fact, it doesn't take affect.
After the fix https://github.com/openshift/machine-api-provider-azure/pull/182 When creating a Machine/MachineSet on Azure Stack Hub with diagnostics.boot.storageAccountType: AzureManaged, the Machine creation fails with a 400 error from Azure API, but there is no validation at the Machine API level to prevent this invalid configuration.{code}
Version-Release number of selected component (if applicable):
{code:none}
4.22{code}
How reproducible:
{code:none}
Always {code}
Steps to Reproduce:
{code:none}
1. On an Azure Stack Hub cluster, create a MachineSet with the following configuration:
diagnostics:
boot:
storageAccountType: AzureManaged
2. Check machine status
3.
{code}
Actual results:
{code:none}
Machine create failed with error:
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidParameter
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidParameter",
"message": "Required parameter 'bootDiagnostics.storageAccountUri' is missing (null).",
"target": "bootDiagnostics.storageAccountUri"
}
}{code}
Expected results:
{code:none}
Azure Stack Hub does not support Azure-managed boot diagnostics storage.
So Machine/MachineSet creation should fail with validation error like "Error: Invalid Machine configuration: AzureManaged boot diagnostics is not supported on Azure Stack Hub"{code}
Additional info:
{code:none}
{code}
Status: Verified
| |||
| #OCPBUGS-78041 | issue | 34 hours ago | [openshift-4.14] ose-powervs-machine-controllers can't be built hermetically New |
Issue 17969587: [openshift-4.13] ose-powervs-machine-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78039. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78038. The following is the description of the original issue: --- Description of problem:
ART can't build ose-powervs-machine-controllers hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-78026 | issue | 34 hours ago | [openshift-4.12] ose-vsphere-cluster-api-controllers can't be built hermetically New |
Issue 17969534: [openshift-4.12] ose-vsphere-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78024. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78023. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78021. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78020. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78019. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78018. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78017. The following is the description of the original issue: --- Description of problem:
ART can't build ose-vsphere-cluster-api-controllers hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-17155 | issue | 2 weeks ago | Console: Minified React error #306 after cluster upgrade CLOSED |
Issue 15400366: Console: Minified React error #306 after cluster upgrade
Description: Description of problem:
Admin Pesrpective of the cluster console -> Observe -> Metrics
https://console.build02.ci.openshift.org/monitoring/query-browser
The page does not work (see the screenshots for the error) after upgrading the cluster {{build02}} to 4.14.0-ec.4 with MacOS Chrome.
It works with Firefox on the same laptop.
The same step works for build01 with {{4.13.7}}.
Client and Server version are listed below and the screenshots.
{noformat}
oc --context build02 version
Client Version: 4.13.4
Kustomize Version: v4.5.7
Server Version: 4.14.0-ec.4
Kubernetes Version: v1.27.3+4aaeaec
~ oc --context build01 version
Client Version: 4.13.4
Kustomize Version: v4.5.7
Server Version: 4.13.7
Kubernetes Version: v1.26.6+73ac561
{noformat}
{noformat}
Error
Description:4566Minified React error #306; visit https://reactjs.org/docs/error-decoder.html?invariant=306&args[]=null&args[]= for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
Component trace:
at t (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:21:71524)
at I (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:1269039)
at t (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:21:73605)
at Suspense
at i (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:313033)
at section
at m (https://console.build02.ci.openshift.org/static/vendor-patternfly-core-chunk-31bef589fa2b3b6b8d69.min.js:1:170065)
at div
at div
at t.a (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:1269268)
at div
at div
at c (https://console.build02.ci.openshift.org/static/vendor-patternfly-core-chunk-31bef589fa2b3b6b8d69.min.js:1:699584)
at d (https://console.build02.ci.openshift.org/static/vendor-patternfly-core-chunk-31bef589fa2b3b6b8d69.min.js:1:217958)
at div
at d (https://console.build02.ci.openshift.org/static/vendor-patternfly-core-chunk-31bef589fa2b3b6b8d69.min.js:1:89498)
at c (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:1015663)
at https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:318678
at x (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:120:86684)
at main
at div
at v (https://console.build02.ci.openshift.org/static/vendor-patternfly-core-chunk-31bef589fa2b3b6b8d69.min.js:1:466913)
at div
at div
at c (https://console.build02.ci.openshift.org/static/vendor-patternfly-core-chunk-31bef589fa2b3b6b8d69.min.js:1:313082)
at div
at div
at c (https://console.build02.ci.openshift.org/static/vendor-patternfly-core-chunk-31bef589fa2b3b6b8d69.min.js:1:699584)
at d (https://console.build02.ci.openshift.org/static/vendor-patternfly-core-chunk-31bef589fa2b3b6b8d69.min.js:1:217958)
at div
at d (https://console.build02.ci.openshift.org/static/vendor-patternfly-core-chunk-31bef589fa2b3b6b8d69.min.js:1:89498)
at Jn (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:58:195416)
at t.default (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:782052)
at t.default (https://console.build02.ci.openshift.org/static/quick-start-chunk-807ebd742dc381028a39.min.js:1:1260)
at s (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:149799)
at t.a (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:1345565)
at te (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:1363570)
at xt (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:58:152524)
at te (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:1363570)
at te (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:1363570)
at o (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:725597)
at t.a (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:1339708)
at t.a (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:1339897)
at t.a (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:1337382)
at ne (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:1363701)
at https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:1366805
at r (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:58:206190)
at t (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:21:71524)
at t (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:21:73605)
at P (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:58:4488)
at ne (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:58:9932)
at te (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:58:9341)
at Y (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:115:11631)
at t (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:21:67954)
at re (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:1367069)
at t.a (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:699943)
at t.a (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:949236)
at s (https://console.build02.ci.openshift.org/static/main-chunk-db2d63d44701b9895477.min.js:1:511869)
at t.a (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:168:261025)
at Suspense
Stack trace:
Error: Minified React error #306; visit https://reactjs.org/docs/error-decoder.html?invariant=306&args[]=null&args[]= for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
at Hs (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:281:112221)
at Sl (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:281:98327)
at Cl (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:281:98255)
at _l (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:281:98118)
at pl (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:281:95105)
at https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:281:44774
at t.unstable_runWithPriority (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:289:3768)
at Uo (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:281:44551)
at Wo (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:281:44719)
at N (https://console.build02.ci.openshift.org/static/vendors~main-chunk-f91c32b721335a248446.min.js:289:2828)
{noformat}
Status: CLOSED
| |||
| #OCPBUGS-73918 | issue | 6 weeks ago | [4.21][Hypershift] Ingress Operator Degraded due to 'ManagementCPUsOverride' admission rejection during initial bootstrap (0 worker nodes) ASSIGNED |
- Similarity: Same admission plugin (autoscaling.openshift.io/ManagementCPUsOverride) rejecting pod creation - Error: Error creating: pods "perf-node-gather-daemonset-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the pod namespace does not allow the workload type management - Context: SNO with Telco DU profile during must-gather operations | |||
| #OCPBUGS-44026 | issue | 13 days ago | BaremetalHosts stuck in 'provisioning' state despite deploy_step success New |
Issue 16426241: BaremetalHosts stuck in 'provisioning' state despite deploy_step success
Description: h3. Description of problem:
During a ZTP install (by [~rhn-gps-rspazzol]), the BMHs remain stuck in the "provisioning" state while the Agent CRs show "Pending user interaction".
It seems possible that the cause is a restart of the metal3 Pod with a fresh ironic database during deployment.
h3. Version-Release number of selected component (if applicable):
Likely affects all extant versions of metal platform when used with the converged flow of ZTP.
h3. How reproducible:
Reportedly happens regularly; haven't done repeated tests with the reproduction steps below but likely to be 100%.
h3. Steps to Reproduce:
# Start provisioning a cluster with ZTP
# While one or more BMHs are in the "provisioning" state, delete the metal3 pod
# Observe the BMHs stuck in provisioning forever
h3. Actual results:
The assisted-installer completes writing CoreOS to disk successfully and stops the agent service ready to restart the host:
{noformat}
Oct 30 14:20:37 localhost.localdomain assisted-installer[45058]: time="2024-10-30T14:20:37Z" level=info msg="Updating node installation stage: Rebooting - Ironic will reboot the node shortly" request_id=9d003e7b-b2f1-49b2-8c72-f122c7b69a68
Oct 30 14:20:37 localhost.localdomain assisted-installer[45058]: time="2024-10-30T14:20:37Z" level=info msg="Running systemctl stop [agent.service]"
Oct 30 14:20:37 localhost.localdomain systemd[1]: Stopping agent.service...
{noformat}
However, by this point in time, IPA heartbeats have been failing for some time with the error:
{noformat}
Oct 30 14:19:21 localhost.localdomain podman[3035]: 2024-10-30 14:19:21.355 1 ERROR ironic_python_agent.agent [-] error sending heartbeat to ['https://10.9.49.125:6385']: ironic_python_agent.errors.HeartbeatError: Error heartbeating to agent API: Error 404: Node 6f7546a2-f49e-4d8d-88f6-a462d53868b6 could not be found.
Oct 30 14:19:21 localhost.localdomain podman[3035]: 2024-10-30 14:19:21.355 1 ERROR ironic_python_agent.agent Traceback (most recent call last):
Oct 30 14:19:21 localhost.localdomain podman[3035]: 2024-10-30 14:19:21.355 1 ERROR ironic_python_agent.agent File "/usr/lib/python3.9/site-packages/ironic_python_agent/agent.py", line 148, in do_heartbeat
Oct 30 14:19:21 localhost.localdomain podman[3035]: 2024-10-30 14:19:21.355 1 ERROR ironic_python_agent.agent self.api.heartbeat(
Oct 30 14:19:21 localhost.localdomain podman[3035]: 2024-10-30 14:19:21.355 1 ERROR ironic_python_agent.agent File "/usr/lib/python3.9/site-packages/ironic_python_agent/ironic_api_client.py", line 200, in heartbeat
Oct 30 14:19:21 localhost.localdomain podman[3035]: 2024-10-30 14:19:21.355 1 ERROR ironic_python_agent.agent raise errors.HeartbeatError(error)
Oct 30 14:19:21 localhost.localdomain podman[3035]: 2024-10-30 14:19:21.355 1 ERROR ironic_python_agent.agent ironic_python_agent.errors.HeartbeatError: Error heartbeating to agent API: Error 404: Node 6f7546a2-f49e-4d8d-88f6-a462d53868b6 could not be found.
Oct 30 14:19:21 localhost.localdomain podman[3035]: 2024-10-30 14:19:21.355 1 ERROR ironic_python_agent.agent
{noformat}
Checking the BMH object, it has _not_ been disconnected from ironic, and it shows a _different_ provisioning ID from the node ID that IPA is looking up in ironic (6f7546a2-f49e-4d8d-88f6-a462d53868b6). This suggests that ironic has been restarted with a fresh database and a new node created with a fresh ID.
Just after the installation succeeds, we also get some heartbeat errors of the form:
{noformat}
Oct 30 14:20:53 localhost.localdomain podman[3035]: 2024-10-30 14:20:53.707 1 ERROR ironic_python_agent.agent [-] error sending heartbeat to ['https://10.9.49.125:6385']: ironic_python_agent.errors.HeartbeatConnectionError: Error attempting to heartbeat - Possible transitory network failure or
blocking port may be present.: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
{noformat}
This continues to happen intermittently, but mostly the logs are full of the "node could not be found" errors.
h3. Expected results:
After the custom deploy method successfully completes, IPA should inform ironic of this:
{noformat}
Oct 30 14:20:48 localhost.localdomain podman[3035]: 2024-10-30 14:20:47.939 1 DEBUG ironic_coreos_install [-] Assisted Agent is in state inactive (result success) _is_assisted_running /usr/lib/python3.9/site-packages/ironic_coreos_install.py:136
Oct 30 14:20:48 localhost.localdomain podman[3035]: 2024-10-30 14:20:47.947 1 INFO ironic_coreos_install [-] Succesfully installed using the assisted agent
Oct 30 14:20:48 localhost.localdomain podman[3035]: 2024-10-30 14:20:47.951 1 INFO root [-] Deploy step completed: {'args': {}, 'interface': 'deploy', 'priority': 80, 'step': 'start_assisted_install'}, result: None
Oct 30 14:20:48 localhost.localdomain podman[3035]: 2024-10-30 14:20:47.951 1 INFO root [-] Asynchronous command execute_deploy_step completed: {'deploy_result': None, 'deploy_step': {'args': {}, 'interface': 'deploy', 'priority': 80, 'step': 'start_assisted_install'}}
{noformat}
Once ironic knows that deployment is complete, it will reboot the host, which will come up as a Node and join the cluster, and the BMH provisioning state will change to {{{}provisioned{}}}.
This would presumably work if we either restarted IPA or looked up the node ID again when heartbeating fails. (We may want to limit this behaviour to Metal³, since changes of node ID are not expected in regular OpenStack deployments.)
h3. Additional info:
Note that when an ironic node is in the {{deploying}} state, Metal³ cannot even attempt to deprovision the host (which is also required for deleting the BMH); it [must wait|https://github.com/metal3-io/baremetal-operator/blob/v0.8.0/pkg/provisioner/ironic/ironic.go#L1697-L1698] for the state to change to {{active}} (which won't happen when this bug is triggered) or {{deploy-wait}} (which doesn't seem to happen either). So this bug is a possible trigger of other reported bugs where deprovisioning never completes, such as OCPBUGS-38253.
It seems likely that the same issue would affect regular (non-assisted) baremetal deployments using the {{install_coreos}} custom deploy method also.
Status: New
2024-10-30T14:19:30.936 creating PXE enabled ironic port for node {host: 'etl7~dl380g9-10', NodeUUID: '26d24895-bc25-4f33-8db6-06e35caa9fee', MAC: '00:11:0a:68:06:4c'}
2024-10-30T14:19:31.020 current provision state {host: 'etl7~dl380g9-10', lastError: '', current: 'enroll', target: ''}
2024-10-30T14:19:31.020 changing provisioning state {host: 'etl7~dl380g9-10', current: 'enroll', existing target: '', new target: 'manage'}
2024-10-30T14:19:41.425 pending PreprovisioningImage not ready {baremetalhost: {'name': 'dl380g9-10', 'namespace': 'etl7'}, provisioningState: 'provisioning'}
2024-10-30T14:19:41.461 current provision state {host: 'etl7~dl380g9-10', lastError: '', current: 'manageable', target: ''}
2024-10-30T14:19:41.461 updating option data {host: 'etl7~dl380g9-10', option: 'instance_uuid', value: '122f0196-c2af-4343-a459-5f3142811de4', oldValue: ''}
2024-10-30T14:24:41.670 pending PreprovisioningImage not ready {baremetalhost: {'name': 'dl380g9-10', 'namespace': 'etl7'}, provisioningState: 'provisioning'}
2024-10-30T14:24:41.719 current provision state {host: 'etl7~dl380g9-10', lastError: '', current: 'manageable', target: ''}
2024-10-30T14:24:41.719 done {baremetalhost: {'name': 'dl380g9-10', 'namespace': 'etl7'}, provisioningState: 'provisioning', requeue: True, after: 300}
ster-engine.apps.hub2.ocp.rht-labs.com/boot-artifacts/kernel?arch=x86_64&version=4.16', 'ExtraKernelParams': 'coreos.live.rootfs_url=https://assisted-image-service-multicluster-engine.apps.hub2.ocp.rht-labs.com/boot-artifacts/rootfs?arch=x86_64&version=4.16 rd.bootif=0', 'Format': 'initrd'}}
2024-10-30T14:04:38.045 current provision state {host: 'etl7~dl380g9-10', lastError: '', current: 'wait call-back', target: 'active'}
2024-10-30T14:04:38.045 provisioning {baremetalhost: {'name': 'dl380g9-10', 'namespace': 'etl7'}, provisioningState: 'provisioning'}
2024-10-30T14:04:44.729 pending PreprovisioningImage not ready {baremetalhost: {'name': 'dl380g9-10', 'namespace': 'etl7'}, provisioningState: 'provisioning'}
2024-10-30T14:04:44.776 current provision state {host: 'etl7~dl380g9-10', lastError: '', current: 'wait call-back', target: 'active'}
2024-10-30T14:04:44.777 provisioning {baremetalhost: {'name': 'dl380g9-10', 'namespace': 'etl7'}, provisioningState: 'provisioning'}
| |||
| #OCPBUGS-77996 | issue | 35 hours ago | [openshift-4.12] openshift-enterprise-console-operator can't be built hermetically New |
Issue 17969152: [openshift-4.12] openshift-enterprise-console-operator can't be built hermetically
Description: This is a clone of issue OCPBUGS-77995. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77994. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77993. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77992. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77862. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77861. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77860. The following is the description of the original issue: --- Description of problem:
ART can't build openshift-enterprise-console-operator hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-78157 | issue | 9 hours ago | [openshift-4.13] ose-nutanix-machine-controllers can't be built hermetically POST |
Issue 17974657: [openshift-4.13] ose-nutanix-machine-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78156. The following is the description of the original issue: --- Description of problem:
ART can't build ose-nutanix-machine-controllers hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-74406 | issue | 38 hours ago | [4.19] Unable to provision NVIDIA DGX B200 nodes via ACM, Redfish VirtualMedia InsertMedia request fails with missing UserName/Password parameters POST |
Issue 17801335: [4.19] Unable to provision NVIDIA DGX B200 nodes via ACM, Redfish VirtualMedia InsertMedia request fails with missing UserName/Password parameters
Description: *Description of problem:*
When attempting to provision NVIDIA DGX B200 nodes through Advanced Cluster Management (ACM), the BareMetalHost (BMH) status hangs at {_}Provisioning{_}.
The Bare Metal Operator (BMO) and Ironic logs show Redfish VirtualMedia mounting failures. The BMC rejects the {{VirtualMedia.InsertMedia}} request with an HTTP 400 error, requiring {{UserName}} and {{Password}} parameters in the request body.
This behavior prevents automated provisioning of DGX B200 nodes.
----
*Version-Release number of selected component (if applicable):*
* OpenShift: 4.18.3 (hub cluster)
* ACM: 2.13
* Ironic/BMO components from OCP 4.18.3 release
----
*How reproducible:*
Always – occurs consistently when attempting to provision NVIDIA DGX B200 nodes.
----
*Steps to Reproduce:*
# Register NVIDIA DGX B200 node in ACM using BareMetalHost (via BMC/Redfish).
# Observe provisioning workflow.
# Check Ironic/BMO logs when provisioning fails.
----
*Actual results:*
* BMH stuck in _Provisioning_ state.
* ISO not mounted via Redfish VirtualMedia.
* BMO/Ironic logs show:
{code:java}
HTTP request: POST https://<bmc>/redfish/v1/Managers/BMC/VirtualMedia/CD1/Actions/VirtualMedia.InsertMedia body: {'Image': 'https://<iso-server>/boot-<uuid>.iso', 'TransferProtocolType': 'HTTPS'}{code}
{{ }}
Response:
{code:java}
status code: 400 error: Base.1.12.GeneralError: The action ... requires the parameter UserName to be present in the request body. MessageId: Base.1.12.ActionParameterMissing{code}
{{ }}
----
*Expected results:*
* Redfish VirtualMedia ISO should mount successfully.
* BMH should proceed past _Provisioning_ and complete host registration.
----
*Additional info:*
* The fix for handling {{UserName}} in Redfish InsertMedia was merged upstream (sushy) and included in OCP 4.18.1 ({{{}Provide vmedia username and password if required{}}}).
* However, in this scenario, the firmware’s response differs: the error lacks a structured {{code}} field, causing credential detection logic to fail.
* Engineering suspects a firmware bug, but BMO/Ironic may require additional handling (e.g., checking {{MessageId=ActionParameterMissing}} or the {{error}} field directly) as a workaround.
Status: POST
| |||
| #OCPBUGS-77207 | issue | 5 days ago | [openshift-4.18] hermetic build failure - ose-ibm-vpc-block-csi-driver Verified |
Issue 17926425: [openshift-4.18] hermetic build failure - ose-ibm-vpc-block-csi-driver
Description: This is a clone of issue OCPBUGS-77206. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77205. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77203. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77167. The following is the description of the original issue: --- Description of problem:
{code:none}
IBM VPC Block CSI Driver build fails in hermetic mode due to non-deterministic GOPATH handling in Makefile. The build system rewrites $GOPATH causing the binary output path -o ${GOPATH}/bin/${EXE_DRIVER_NAME} to become invalid, resulting in build failures.{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift 4.19+ (affects hermetic build conversion) {code}
How reproducible:
{code:none}
Always - when building ibm-vpc-block-csi-driver in hermetic mode{code}
Steps to Reproduce:
{code:none}
1. Build ibm-vpc-block-csi-driver container in hermetic mode
2. Makefile uses absolute GOPATH-dependent output path: -o ${GOPATH}/bin/${EXE_DRIVER_NAME}
3. Build system rewrites $GOPATH during hermetic build process {code}
Actual results:
{code:none}
Build fails because binary cannot be found at expected GOPATH location after build system modifies $GOPATH
Error: building at STEP "COPY --from=builder /cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver /bin/ibm-vpc-block-csi-driver": checking on sources under [...]: copier: stat: "/cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver": no such file or directory{code}
Expected results:
{code:none}
Build succeeds using relative path bin/${EXE_DRIVER_NAME} that doesn't depend on GOPATH manipulation, consistent with other CSI drivers like aws-ebs-csi-driver {code}
Additional info:
{code:none}
Fix implemented: Change Makefile to use relative path -o bin/${EXE_DRIVER_NAME} instead of absolute GOPATH-dependent path. This aligns with other CSI drivers and makes builds hermetic-compatible. Comment added to indicate this is a permanent carry patch for OpenShift builds.{code}
Status: Verified
| |||
| #OCPBUGS-77756 | issue | 6 days ago | [openshift-4.13] hypershift can't be built hermetically POST |
Issue 17952763: [openshift-4.13] hypershift can't be built hermetically
Description: Description of problem:
ART can't build hypershift hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir. {code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-59320 | issue | 2 months ago | Some failed bundle unpack pods remain in Error state and are not automatically cleaned up. New |
Issue 17145530: Some failed bundle unpack pods remain in Error state and are not automatically cleaned up.
Description: Description of problem:
{code:none}
In hypershift hosted cluster, some failed bundle unpack pods remain in Error state and are not automatically cleaned up.{code}
Version-Release number of selected component (if applicable):
{code:none}
Cluster version is 4.20.0-0.nightly-2025-07-15-031201{code}
How reproducible:
{code:none}
always{code}
Steps to Reproduce:
{code:none}
1. create hypershift hosted cluster and create some subs
2. check pod in openshift-marketplace
xzha@xzha1-mac 420_upgrade_test % oc get pod -n openshift-marketplace
NAME READY STATUS RESTARTS AGE
bf4ed2748e7e3aa99a4089248dd10559a123af0a4556d4231c44159a9dc2pfc 0/1 Error 0 125m
bf4ed2748e7e3aa99a4089248dd10559a123af0a4556d4231c44159a9dd5w6b 0/1 Completed 0 124m
fef64e67f7ed7822d1a21a290838f02d042715e25164247c3fafdbd62ccxc6l 0/1 Completed 0 3m11s
fef64e67f7ed7822d1a21a290838f02d042715e25164247c3fafdbd62cfwnvz 0/1 Error 0 4m12s
fef64e67f7ed7822d1a21a290838f02d042715e25164247c3fafdbd62cw84qm 0/1 Error 0 3m46s
there are some failed pods remain in Error state and are not automatically cleaned up.
state:
terminated:
containerID: cri-o://f84d39209f18ff35913b26c8775bc04b3c743b3628cc313b44df9a23269affc6
exitCode: 1
finishedAt: "2025-07-15T12:01:48Z"
message: |+
time="2025-07-15T12:01:38Z" level=info msg="Using in-cluster kube client config"
Error: error loading manifests from directory: Get "https://172.31.0.1:443/api/v1/namespaces/openshift-marketplace/configmaps/bf4ed2748e7e3aa99a4089248dd10559a123af0a4556d4231c44159a9d3c60a": net/http: TLS handshake timeout
Usage:{code}
Actual results:
{code:none}
some failed pods remain in Error state and are not automatically cleaned up. {code}
Expected results:
{code:none}
those failed pods will be cleaned automatically{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-78109 | issue | 16 hours ago | [openshift-4.13] openshift-enterprise-haproxy-router can't be built hermetically POST |
Issue 17972290: [openshift-4.13] openshift-enterprise-haproxy-router can't be built hermetically
Description: This is a clone of issue OCPBUGS-78108. The following is the description of the original issue: --- Description of problem:
ART can't build openshift-enterprise-haproxy-router & ose-haproxy-router-base hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-72507 | issue | 4 weeks ago | [JobSet] - It's not possible to create a JobSet using default options Verified |
Unable to parse ALM expamples
SyntaxError: JSON.parse: expected double-quoted property name at line 11 column 5 of the JSON data {code}
I think it was the newlines being included. YAML's | vs |-
| |||
| #OCPBUGS-77435 | issue | 4 days ago | "Something wrong happened" n is not iterable showcasing in Openshift web console during NNCP creation New |
Issue 17939175: "Something wrong happened" n is not iterable showcasing in Openshift web console during NNCP creation
Description: Description of problem:
{code:none}
If we start the NNCP wizard to create a new bridge. As soon as we click next on the policy name, the page "something wrong happened" appears with the message "n is not iterable".
{code}
Version-Release number of selected component (if applicable):
{code:none}
kubernetes-nmstate-operator.4.21.0-202602180447
Openshift version 4.21.2{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
Component Trace:
at hn (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/exposed-Topology-chunk-e190b1827ce330a09353.min.js:1:52049)
at div
at div
at K (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/5662-chunk-e3a18368582cf532b0da.min.js:1:10277)
at div
at div
at J (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/5662-chunk-e3a18368582cf532b0da.min.js:1:11289)
at ee (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/5662-chunk-e3a18368582cf532b0da.min.js:1:19753)
at div
at L (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/5662-chunk-e3a18368582cf532b0da.min.js:1:7793)
at Z (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/5662-chunk-e3a18368582cf532b0da.min.js:1:17418)
at bn (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/exposed-Topology-chunk-e190b1827ce330a09353.min.js:1:54251)
at wn (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/exposed-Topology-chunk-e190b1827ce330a09353.min.js:1:55731)
at Hn (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/exposed-Topology-chunk-e190b1827ce330a09353.min.js:1:65714)
at div
at div
at l (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:52:160317)
at Xn (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/exposed-Topology-chunk-e190b1827ce330a09353.min.js:1:66964)
at div
at o (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3688430)
at div
at o (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3688039)
at w (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:52:161445)
at a (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:52:163267)
at Sa (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/exposed-Topology-chunk-e190b1827ce330a09353.min.js:1:73585)
at b (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:75336)
at f (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:75681)
at S (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:75855)
at _ (https://console-openshift-console.apps.gsphub.gsplab.local/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:51105)
at ne (https://console-openshift-console.apps.gsphub.gsplab.local/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:56537)
at Suspense
at i (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:277483)
at m
at section
at v (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3460647)
at W (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:933627)
at main
at div
at div
at div
at o (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3304825)
at h (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3303601)
at div
at d (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3301897)
at div
at div
at w (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3452729)
at div
at l (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3681733)
at c (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:875511)
at div
at o (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3304146)
at div
at div
at o (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3304825)
at h (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3303601)
at div
at d (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3301897)
at Sn (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3136530)
at i (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:54467)
at a (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:54036)
at fe (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:946219)
at tt (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:3095463)
at fe (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:946219)
at fe (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:946219)
at t.QueryClientProvider (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/mce/8168.24f24aef.js:205:1396530)
at Zn (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/mce/8168.24f24aef.js:205:4728798)
at RecoilRoot (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/mce/8168.24f24aef.js:205:4730771)
at t.PluginDataContextProvider (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/mce/exposed-contextProvider.e725d21f.js:1:10344)
at fe (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:946219)
at u (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/kubevirt-plugin/exposed-modalProvider-chunk-5f1caebfdb41b6e7076c.min.js:1:423)
at fe (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:946219)
at fe (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:946219)
at Suspense
at f (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:112989)
at b (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:111873)
at a (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:23099)
at u (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:25343)
at ye (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:946335)
at Se (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:949610)
at _ (https://console-openshift-console.apps.gsphub.gsplab.local/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:51105)
at ne (https://console-openshift-console.apps.gsphub.gsplab.local/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:56537)
at _ (https://console-openshift-console.apps.gsphub.gsplab.local/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:51105)
at ne (https://console-openshift-console.apps.gsphub.gsplab.local/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:56537)
at te (https://console-openshift-console.apps.gsphub.gsplab.local/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:55950)
at G (https://console-openshift-console.apps.gsphub.gsplab.local/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:43270)
at t (https://console-openshift-console.apps.gsphub.gsplab.local/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:65457)
at xe (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:949902)
at g (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:478028)
at c (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:86080)
at e (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:2689533)
at v (https://console-openshift-console.apps.gsphub.gsplab.local/static/main-bundle-1476fa5274b883cc891a.min.js:1:926590)
at c (https://console-openshift-console.apps.gsphub.gsplab.local/static/72133-bundle-9dbb1ef6b0b5f1bf2c3e.min.js:2:16683)
at Suspense
Stack Trace:
TypeError: n is not iterable
at https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/exposed-Topology-chunk-e190b1827ce330a09353.min.js:1:28106
at Array.reduce (<anonymous>)
at https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/exposed-Topology-chunk-e190b1827ce330a09353.min.js:1:28011
at https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/exposed-Topology-chunk-e190b1827ce330a09353.min.js:1:28124
at https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/exposed-Topology-chunk-e190b1827ce330a09353.min.js:1:28265
at Array.map (<anonymous>)
at https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/exposed-Topology-chunk-e190b1827ce330a09353.min.js:1:27974
at Pe (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/exposed-Topology-chunk-e190b1827ce330a09353.min.js:1:28311)
at hn (https://console-openshift-console.apps.gsphub.gsplab.local/api/plugins/nmstate-console-plugin/exposed-Topology-chunk-e190b1827ce330a09353.min.js:1:52087)
at sa (https://console-openshift-console.apps.gsphub.gsplab.local/static/vendors~main-chunk-37548-c2cda3d4584ab055c403.min.js:2:2607126)
{code}
Status: New
| |||
| #OCPBUGS-78179 | issue | 9 hours ago | [openshift-4.14] ose-azure-cluster-api-controllers can't be built hermetically New |
Issue 17974788: [openshift-4.13] ose-azure-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78178. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78177. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78176. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77983. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77905. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77904. The following is the description of the original issue: --- Description of problem:
ART can't build ose-azure-cluster-api-controllers
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-62000 | issue | 5 days ago | Security requirements do not allow UserInfo mapping resulting in unsupported config of auth ASSIGNED |
Issue 17408594: Security requirements do not allow UserInfo mapping resulting in unsupported config of auth
Description: Chase Morgan has a security requirement to disable UserInfo in their ADFS server.
curls to the ADFS to obtain a token will fail with
~~~
Error: non-200 response from UserInfo: 401, WWW-Authenticate=Bearer error="invalid_token", error_description="MSIS9921: Received invalid UserInfo request. Audience 'https://oauth-openshift.apps.wh-ngcptnt1.svr.us.jpmchase.net/oauth2callback/adfs' in the access token is not same as the identifier of the UserInfo relying party trust 'urn:microsoft:userinfo'."r
~~~
That is with the below authconfig
~~~
urls:
authorize: https://mysso.example.com/auth/realms/master/protocol/openid-connect/auth
token: https://mysso.example.com/auth/realms/master/protocol/openid-connect/token
userInfo: https://mysso.example.com/auth/realms/master/protocol/openid-connect/userinfo
~~~
Removing the userinfo from the authconfig will result in the curls working but results in an unsupported state.
https://access.redhat.com/solutions/6960548
https://access.redhat.com/articles/7064122
They have implemented the fixes from Microsoft that was mentioned in https://issues.redhat.com/browse/RFE-3596 and https://issues.redhat.com/browse/RFE-2770
How can JPMC implement the authconfig to not use userinfo with adfs and not end up unsupported?
Status: ASSIGNED
| |||
| #OCPBUGS-65897 | issue | 6 weeks ago | [assisted-service] go mod tidy fix ASSIGNED |
Issue 17601884: [assisted-service] go mod tidy fix
Description: ART builds failing with incorrect go mod dependencies. Prefetch task failing on konflux: [https://konflux-ui.apps.kflux-ocp-p01.7ayg.p1.openshiftapps.com/ns/ocp-art-tenant/applications/openshift-4-16/pipelineruns/ose-4-16-ose-agent-installer-api-server-zwhp6/logs] (ART internal)
{code:java}
go: downloading golang.org/x/mod v0.17.0
go: updates to go.mod needed; to update it:
go mod tidy
2025-11-22 04:18:10,928 ERROR Failed to fetch gomod dependencies
2025-11-22 04:18:14,114 ERROR PackageManagerError: Go execution failed: `/usr/local/bin/go mod vendor` failed with rc=1
Error: PackageManagerError: Go execution failed: `/usr/local/bin/go mod vendor` failed with rc=1
The cause of the failure could be:
- something is broken in hermeto
- something is wrong with your repository
- communication with an external service failed (please try again)
The output of the failing command should provide more details, please check the logs. {code}
Builds will be paused until issue is resolved. Please feel free to close the bug as soon as the PR is merged
Status: ASSIGNED
| |||
| #OCPBUGS-77206 | issue | 8 days ago | [openshift-4.19] hermetic build failure - ose-ibm-vpc-block-csi-driver Verified |
Issue 17926170: [openshift-4.19] hermetic build failure - ose-ibm-vpc-block-csi-driver
Description: This is a clone of issue OCPBUGS-77205. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77203. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77167. The following is the description of the original issue: --- Description of problem:
{code:none}
IBM VPC Block CSI Driver build fails in hermetic mode due to non-deterministic GOPATH handling in Makefile. The build system rewrites $GOPATH causing the binary output path -o ${GOPATH}/bin/${EXE_DRIVER_NAME} to become invalid, resulting in build failures.{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift 4.19+ (affects hermetic build conversion) {code}
How reproducible:
{code:none}
Always - when building ibm-vpc-block-csi-driver in hermetic mode{code}
Steps to Reproduce:
{code:none}
1. Build ibm-vpc-block-csi-driver container in hermetic mode
2. Makefile uses absolute GOPATH-dependent output path: -o ${GOPATH}/bin/${EXE_DRIVER_NAME}
3. Build system rewrites $GOPATH during hermetic build process {code}
Actual results:
{code:none}
Build fails because binary cannot be found at expected GOPATH location after build system modifies $GOPATH
Error: building at STEP "COPY --from=builder /cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver /bin/ibm-vpc-block-csi-driver": checking on sources under [...]: copier: stat: "/cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver": no such file or directory{code}
Expected results:
{code:none}
Build succeeds using relative path bin/${EXE_DRIVER_NAME} that doesn't depend on GOPATH manipulation, consistent with other CSI drivers like aws-ebs-csi-driver {code}
Additional info:
{code:none}
Fix implemented: Change Makefile to use relative path -o bin/${EXE_DRIVER_NAME} instead of absolute GOPATH-dependent path. This aligns with other CSI drivers and makes builds hermetic-compatible. Comment added to indicate this is a permanent carry patch for OpenShift builds.{code}
Status: Verified
| |||
| #OCPBUGS-77468 | issue | 11 days ago | [openshift-4.13] openshift-enterprise-hyperkube can't be built hermetically New |
Issue 17940936: [openshift-4.13] openshift-enterprise-hyperkube can't be built hermetically
Description: Description of problem:
ART can't build openshift-enterprise-hyperkube hermetically. Same applies for ose-installer-kube-apiserver-artifacts
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.
{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-66250 | issue | 2 months ago | GCP: Install failures due to inconsistent iam apply Verified |
Issue 17631044: GCP: Install failures due to inconsistent iam apply
Description: *Description of problem*. In Advanced Cluster Security we rely for OCP creation in our CI and recently observed [an increase of cluster creation failures|https://docs.google.com/document/d/1vE-kLPWooO75BXxGub6YZpM4NZolKBcOrvJULZ5NBw0/edit]. While we've been [advised to retry the failures|https://redhat-internal.slack.com/archives/C68TNFWA2/p1722873956362609] (and we do so now, see [ROX-25416|https://issues.redhat.com/browse/ROX-25416]), I'm afraid our use case is not so unique and others are affected as well.
We suggest upgrading terraform and provider to the latest version (possible before license changes) in openshift-installer for 4.12+. The underlying issue is [probably already fixed upstream|https://github.com/hashicorp/terraform-provider-google/issues/17332] and released in [v5.37.0|https://github.com/hashicorp/terraform-provider-google/releases/tag/v5.37.0].
Version-Release number of selected component (if applicable): TBD
How reproducible: TBD
Steps to Reproduce: TBD
Actual results: TBD
Expected results: TBD
h3. Additional info.
The most common error we see in our [JIRA issues|https://issues.redhat.com/browse/ROX-25724?jql=text%20~%20%22module.master.google_service_account.master-node-sa%22] is and that is something we could find similar issues with AWS provider too eg. OCPBUGS-4213.
{code}
level=error msg=Error: Provider produced inconsistent result after apply .... resource was present, but now absent
{code}
Summary of errors from:
{code}
3 failed to create cluster: failed to apply Terraform: error(GCPComputeBackendTimeout) from Infrastructure Provider: GCP is experiencing backend service interuptions, the compute instance failed to create in reasonable time."
3 Provider produced inconsistent result after apply\n\nWhen applying changes to\nmodule.master.google_service_account.master-node-sa[0], provider\n\"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new value: Root\nresource was present, but now absent.\n\n
6 Error waiting to create Network: Error waiting for Creating Network: timeout while waiting for state to become 'DONE' (last state: 'RUNNING', timeout: 4m0s)\n\n with module.network.google_compute_network.cluster_network[0],\n on network/network.tf line 1, in resource \"google_compute_network\" \"cluster_network\":\n 1: resource \"google_compute_network\" \"cluster_network\" {\n\n"
9 error applying Terraform configs: failed to apply Terraform: error(GCPComputeBackendTimeout) from Infrastructure Provider: GCP is experiencing backend service interuptions, the compute instance failed to create in reasonable time."
14 Provider produced inconsistent result after apply\n\nWhen applying changes to module.master.google_service_account.master-node-sa,\nprovider \"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new\nvalue: Root resource was present, but now absent.
16 Provider produced inconsistent result after apply\n\nWhen applying changes to google_service_account_key.bootstrap, provider\n\"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new value: Root\nresource was present, but now absent.
18 Provider produced inconsistent result after apply\n\nWhen applying changes to module.iam.google_service_account.worker-node-sa,\nprovider \"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new\nvalue: Root resource was present, but now absent.
34 Error creating service account key: googleapi: Error 404: Service account projects/acs-san-stackroxci/serviceAccounts/XXX@acs-san-stackroxci.iam.gserviceaccount.com does not exist., notFound\n\n with google_service_account_key.bootstrap,\n on main.tf line 38, in resource \"google_service_account_key\" \"bootstrap\":\n 38: resource \"google_service_account_key\" \"bootstrap\" {\n\n"
45 error applying Terraform configs: failed to apply Terraform: exit status 1\n\nError: Provider produced inconsistent result after apply\n\nWhen applying changes to\nmodule.master.google_service_account.master-node-sa[0], provider\n\"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new value: Root\nresource was present, but now absent.
59 error applying Terraform configs: failed to apply Terraform: exit status 1\n\nError: Provider produced inconsistent result after apply\n\nWhen applying changes to module.iam.google_service_account.worker-node-sa,\nprovider \"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new\nvalue: Root resource was present, but now absent.
100 Provider produced inconsistent result after apply\n\nWhen applying changes to google_service_account.bootstrap-node-sa, provider\n\"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new value: Root\nresource was present, but now absent.
103 Provider produced inconsistent result after apply\n\nWhen applying changes to module.iam.google_service_account.worker-node-sa[0],\nprovider \"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new\nvalue: Root resource was present, but now absent.
116 Provider produced inconsistent result after apply\n\nWhen applying changes to\nmodule.master.google_service_account.master-node-sa[0], provider\n\"provider[\\\"openshift/local/google\\\"]\" produced an unexpected new value: Root\nresource was present, but now absent.
{code}
The openshift installer contains a bundled terraform and google-provider
* 4.12
** Terraform 1.0.11 (November 10, 2021)
** https://github.com/openshift/installer/blob/release-4.12/terraform/terraform/go.mod#L5
** Provider: v4.5.0 (Dec 20, 2021)
** https://github.com/openshift/installer/blob/release-4.12/terraform/providers/google/go.mod#L5
* 4.14+
** Terraform: 1.3.7 (January 04, 2023)
** https://github.com/openshift/installer/blob/release-4.14/terraform/terraform/go.mod#L5
** Provider: v4.74.0 (Jul 18, 2023)
** https://github.com/openshift/installer/blob/release-4.14/terraform/providers/google/go.mod#L5
* 4.18+
** Does not contain google provider at all: https://github.com/openshift/installer/pull/8723
Status: Verified
| |||
| #OCPBUGS-78023 | issue | 34 hours ago | [openshift-4.14] ose-vsphere-cluster-api-controllers can't be built hermetically New |
Issue 17969432: [openshift-4.14] ose-vsphere-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78021. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78020. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78019. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78018. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78017. The following is the description of the original issue: --- Description of problem:
ART can't build ose-vsphere-cluster-api-controllers hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-76586 | issue | 12 hours ago | GCP: Provider produced inconsistent result after apply - google_service_account Verified |
Issue 17866072: GCP: Provider produced inconsistent result after apply - google_service_account
Description: Description of problem:
{code:none}
The fix given for issue https://issues.redhat.com/browse/OCPBUGS-42014(pointing to same problem) was at service account key creation. but the actual problem lies at service account creation itself.
We suggest upgrading terraform and provider to the latest version in openshift-installer for 4.12+. The underlying issue is already fixed upstream and released in v6.45.0.
Note: we see this problem more often in lower openshift versions 4.12, 4.13 than in 4.16.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
Intermittent{code}
Steps to Reproduce:
{code:none}
1.Install the UPI openshift cluster in GCP platform
2.
3.
{code}
Actual results:
{code:none}
level=error msg=Error: Provider produced inconsistent result after apply
level=error
level=error msg=When applying changes to
level=error msg=module.master.google_service_account.master-node-sa[0], provider
level=error msg="provider[\"openshift/local/google\"]" produced an unexpected new value: Root
level=error msg=resource was present, but now absent.
level=error
level=error msg=This is a bug in the provider, which should be reported in the provider's own{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: Verified
| |||
| #OCPBUGS-76299 | issue | 56 minutes ago | SNO ZTP deployment fails with Dell GNR-D iDRAC Firmware Version: 1.30.10.50 ASSIGNED |
Issue 17846211: SNO ZTP deployment fails with Dell GNR-D iDRAC Firmware Version: 1.30.10.50
Description: Description of problem:
ZTP deployment failure
Server: Dell GRNRD
iDRAC IP: 192.168.224.190
iDRAC Firmware Version: 1.30.10.50
OpenShift Version: 4.21.0
Deployment Method: ZTP via ACM/Assisted Installer
Version-Release number of selected component (if applicable):
4.21
How reproducible:
Every time SNO deploy using ZTP
Steps to Reproduce:{
1. Dell gnrd Perla with latest IDRAC version : 1.30.10.50
2. using ZTP deploy SNO
3.The Malformed Redfish Attribute Error
Node 8c798425-3ae4-463d-a4cd-539c1611924d failed verify step clear_job_queue with unexpected error:
The attribute MemorySummary/TotalSystemMemoryGiB is malformed in the resource /redfish/v1/Systems/System.Embedded.1:
invalid literal for int() with base 10: '256 GB'
Actual results:
Root Cause: iDRAC firmware returns "256 GB" (string with unit) instead of 256 (integer) for TotalSystemMemoryGiB
Expected results:
virtual media mount successful iso
Additional info:
Error Message (with idrac-redfish-virtualmedia://):
The Value is not valid BMC address, supported protocols are redfish-virtualmedia or idrac-virtualmedia.
When Ironic's idrac driver parses this response, it attempts to convert TotalSystemMemoryGiB to an integer:
# Ironic code
memory_gib = int(system_data['MemorySummary']['TotalSystemMemoryGiB'])
# Raises: ValueError: invalid literal for int() with base 10: '256 GB'
#
curl -k -s -u user:passowrd \
> https://192.168.224.190/redfish/v1/Systems/System.Embedded.1 | jq '.MemorySummary'
{
"Status": {
"Health": "OK",
"HealthRollup": "OK",
"State": "Enabled"
},
"Status@Redfish.Deprecated": "Please migrate to use Status in the individual Memory resources",
"TotalSystemMemoryGiB": 256,
"MemoryMirroring": "System"
}
Impact on Ironic/Metal3
# Ironic sushy library code (simplified)
memory_gib = int(system_data['MemorySummary']['TotalSystemMemoryGiB'])
# Raises: ValueError: invalid literal for int() with base 10: '256 GB'
Status: ASSIGNED
| |||
| #OCPBUGS-73920 | issue | 6 days ago | Redfish vMedia mount failure on Cisco B-Series blades during ACM provisioning New |
Issue 17773649: Redfish vMedia mount failure on Cisco B-Series blades during ACM provisioning
Description: Description of problem: {code:none} Redfish Virtual Media (vMedia) mounting fails consistently on Cisco B-Series (Blade) servers when provisioned via ACM / SiteConfig Operator, whereas Cisco C-Series (Rack) servers in the same environment work successfully.
When ACM attempts to map the discovery ISO, the CIMC reports a "Mount failed" error and the resource state moves to "UnavailableOffline." Logs indicate that the Redfish 'PATCH' method is not supported for this resource, and the subsequent 'InsertMedia' POST action returns a 400 Bad Request. Internal discussions suggest a potential communication gap between the CIMC and the ACM Hub through the Fabric Interconnects, but this has not been confirmed via packet capture. {code}
Version-Release number of selected component (if applicable): {code:none} ACM: 2.x (via SiteConfig Operator) OCP: 4.20 (Target) Hardware: Cisco B200 M5 Firmware: 4.3(6c) {code}
How reproducible: {code:none} 100% reproducible on Cisco B-Series blades managed by Fabric Interconnects. {code}
Steps to Reproduce: {code:none}
Define a Bare Metal cluster using ACM SiteConfig/ClusterInstance.
Configure bmcAddress using 'redfish-virtualmedia+https' pointing to a Cisco B-Series CIMC.
Apply the configuration and monitor the node provisioning process.
Check Virtual Media status via Redfish API: /redfish/v1/Managers/CIMC/VirtualMedia/3 {code}
Actual results: {code:none} CIMC reports a critical health status with "Mount failed." Sushy/Ironic logs show: "HTTP PATCH... status code: 400, error: Base.1.13.0.GeneralError: The action 'Patch Update For Resource' is not supported by the resource." followed by a failed POST to InsertMedia. {code}
Expected results: {code:none} The discovery ISO should mount successfully as a Virtual DVD, allowing the blade to boot into the discovery environment. {code}
Additional info: {code:none}
Manual mounting via the CIMC Web UI works, indicating the ISO itself is valid.
The environment uses Fabric Interconnects for B-Series management traffic, which is the primary architectural difference from the successful C-Series deployments.
{code}
Status: New
| |||
| #OCPBUGS-74983 | issue | 37 hours ago | Redfish PATCH failure: BootSourceOverrideMode unsupported on Cisco CIMC hardware blocks discovery ASSIGNED |
Issue 17829950: Redfish PATCH failure: BootSourceOverrideMode unsupported on Cisco CIMC hardware blocks discovery
Description: Description of problem: {code:none} During baremetal host provisioning via ACM 2.14 / Metal3 on Cisco hardware, the ironic-conductor (sushy) fails to set the boot mode. The Redfish API returns a 400 Bad Request because the property "BootSourceOverrideMode" is either read-only or unsupported by the Cisco CIMC Redfish implementation.
Even after manually setting the hardware to UEFI and Secure Boot, Ironic attempts to patch the "BootSourceOverrideMode" to "UEFI", resulting in a fatal error that prevents the host from booting the discovery ISO.
Log snippet: 2026-02-03 07:08:13.542 1 ERROR ironic.drivers.modules.redfish.management [None req-f5103489-5928-4fce-9975-360e1dede01d] Setting boot mode to uefi failed for node bec24335-d444-4460-9d62-652897ca16c9. Error: HTTP PATCH https://<BMC_IP>/redfish/v1/Systems/WZP29020ZJN returned code 400. Base.1.13.0.GeneralError: Following properties in the request body cannot be updated because they are either read only, unknown or unsupported : "BootSourceOverrideMode" {code}
Version-Release number of selected component (if applicable): {code:none} OCP Hub: 4.18.14 ACM: 2.14 Hardware: Cisco CIMC {code}
How reproducible: {code:none} Always reproducible on this specific Cisco hardware/firmware combination when using the Redfish driver. {code}
Steps to Reproduce: {code:none}
Define a BareMetalHost (BMH) with bootMode: UEFI.
Attempt to provision the host using ZTP/ACM 2.14.
Observe the metal3-ironic pod logs during the management/provisioning phase. {code}
Actual results: {code:none} Ironic attempts to send a PATCH request with {'Boot': {'BootSourceOverrideMode': 'UEFI'}}. The BMC rejects this with a 400 GeneralError, causing Ironic to fail the operation and stop the provisioning flow. {code}
Expected results: {code:none} Ironic/Sushy should verify if "BootSourceOverrideMode" is a supported/writable property before attempting a PATCH. If the host is already in the desired boot mode (manually configured), the failure to PATCH should not be fatal, or the driver should use an alternative method compatible with Cisco Redfish. {code}
Additional info: {code:none}
Manual verification via curl shows that UEFI is enabled on the host, but the "BootSourceOverrideMode" parameter is missing from the Redfish tree or marked as non-updatable.
Must-gather attached to the support case. {code}
Status: ASSIGNED
| |||
| #OCPBUGS-45804 | issue | 10 hours ago | Installer fails GetMarketplaceImage with context deadline exceeds New |
Issue 16509003: Installer fails GetMarketplaceImage with context deadline exceeds
Description: Description of problem:
In openshift/installer, when running GetMarketplaceImage, we get
{code:none}
encountered error: failed to fetch marketplace image: could not get marketplace image: compute.VirtualMachineImagesClient#Get: Failure sending request: StatusCode=0 – Original Error: context deadline exceeded{code}
This error comes from [https://github.com/openshift/installer/blob/2b773d8339292cb3f2ba294e3b8f186da9cf7395/pkg/asset/installconfig/azure/client.go#L367-L378]
However, it doesn't have to timeout in 30 secs here, and considering the time required to the whole install, we should retry the request when it fails decent number of times.
Also it's still using the deprecated SDK, which lacks retry mechanism.
With the new SDK, there's a rich retry mechanism and that should retry when it gets certain errors.
[https://learn.microsoft.com/en-us/azure/storage/blobs/storage-retry-policy-go]
[https://azure.github.io/azure-sdk/releases/deprecated/go.html]
[https://github.com/Azure/azure-sdk-for-go/blob/69f43460d64e153c91e6fe52113ad31ce6f91a06/sdk/azcore/runtime/policy_retry.go#L106]
Version-Release number of selected component (if applicable):
{code:none}
all versions implementing GetMarketplaceImage{code}
How reproducible:
{code:none}
when there's a network issue.{code}
Steps to Reproduce:
It relates to the network issue. not reproducible always.
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
It terminates immediately.{code}
Expected results:
{code:none}
It should retry some times.{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77912 | issue | 36 hours ago | OpenShift Lightspeed "Import to console" action fails on 4.20+ ASSIGNED |
Issue 17963284: OpenShift Lightspeed "Import to console" action fails on 4.20+
Description: Description of problem:
{code:none}
When clicking the "Import to console" action on an OpenShift Lightspeed YAML code block (generated as part of the AI response), it correctly redirects to the YAML import page, but the editor is not pre-populated with the YAML {code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
Always{code}
Steps to Reproduce:
{code:none}
1. Install OpenShift Lightspeed operator
2. Prompt the AI with something that will cause it to respond with some YAML such as "Show me some example deployment YAML"
3. Click the "Import to console" action on the resulting YAML code block{code}
Actual results:
{code:none}
YAML not injected into editor and JavaScript error thrown{code}
Expected results:
{code:none}
YAML editor should be pre-populated with the YAML code {code}
Additional info:
{code:none}
Error message is: "TypeError: Cannot use 'in' operator to search for 'editor' in undefined"{code}
Status: POST
| |||
| #OCPBUGS-66294 | issue | 2 months ago | LCA seedgen container fails when TechPreviewNoUpgrade is enabled on seed cluster New |
Issue 17633189: LCA seedgen container fails when TechPreviewNoUpgrade is enabled on seed cluster
Description: Description of problem:{code:none}
In preperation to sanity check LCA operator deployed via OLMv1 (which currently requires the TechPreviewNoUpgrade feature gate enabled).
First was to enable the feature gate, but install LCA the traditional way first in order to find if the feature gate affects LCA seedgen.
The result is seed gen fails due to container error on spoke cluster
[core@helix80 ~]$ sudo podman logs lca_image_builder | grep -C5 -i error
time="2025-12-02 22:47:08" level=info msg="Unauthenticated etcd server for recert is up and running"
time="2025-12-02 22:47:08" level=info msg="Start running recert"
time="2025-12-02 22:47:08" level=info msg="Executing nsenter with args [--target 1 --cgroup --mount --ipc --pid -- podman run --rm --network=host --privileged --replace --name recert -v /etc:/host-etc -v /etc/ssh:/ssh -v /etc/kubernetes:/kubernetes -v /var/lib/kubelet:/kubelet -v /var/tmp:/var/tmp -v /etc/machine-config-daemon:/machine-config-daemon -v /etc/pki:/pki -e RECERT_CONFIG=/var/tmp/backup/recert_config.json --authfile /var/lib/lca/ibu-seedgen-orch/auth.json registry.redhat.io/openshift4/recert-rhel9@sha256:73c87fcc1a20bfec8aef234bb2d12bd69ba24072256548f003f245e2f560618f]"
time="2025-12-02 22:47:10" level=info msg="Killing the unauthenticated etcd server"
time="2025-12-02 22:47:10" level=info msg="Executing nsenter with args [--target 1 --cgroup --mount --ipc --pid -- podman stop recert_etcd]"
time="2025-12-02 22:47:10" level=error msg="failed to create seed image: failed to run once recert: failed to run recert tool container: 2025-12-02 22:47:09 - INFO - src/cluster_crypto/crypto_utils.rs:272: using openssl: OpenSSL 3.5.1 1 Jul 2025 (Library: OpenSSL 3.5.1 1 Jul 2025)\n\n2025-12-02 22:47:09 - INFO - src/cluster_crypto/crypto_utils.rs:300: OpenSSL version is compatible\nError: etcd to utf-8\n\nCaused by:\n invalid utf-8 sequence of 1 bytes from index 0\n: exit status 1"
time="2025-12-02 22:47:10" level=info msg="Cleaning up seed cluster"
time="2025-12-02 22:47:10" level=info msg="Removing seed image"
time="2025-12-02 22:47:10" level=info msg="Executing nsenter with args [--target 1 --cgroup --mount --ipc --pid -- podman rmi registry.kni-qe-84.telcoqe.eng.rdu2.dc.redhat.com:5000/ibu/seed:4.21.0-0.nightly-2025-11-22-193140]"
time="2025-12-02 22:47:10" level=error msg="failed to remove seed image: failed to run \"podman\" in host namespace with args [rmi registry.kni-qe-84.telcoqe.eng.rdu2.dc.redhat.com:5000/ibu/seed:4.21.0-0.nightly-2025-11-22-193140]: Error: registry.kni-qe-84.telcoqe.eng.rdu2.dc.redhat.com:5000/ibu/seed:4.21.0-0.nightly-2025-11-22-193140: image not known\n: exit status 1"
time="2025-12-02 22:47:10" level=info msg="Cleaning up systemd service units"
time="2025-12-02 22:47:10" level=info msg="Disabling service unit installation-configuration.service"
time="2025-12-02 22:47:10" level=info msg="Running systemctl disable [installation-configuration.service --now]"
time="2025-12-02 22:47:10" level=info msg="Executing nsenter with args [--target 1 --cgroup --mount --ipc --pid -- systemctl disable installation-configuration.service --now]"
time="2025-12-02 22:47:11" level=info msg="Removing service unit installation-configuration.service"
--
time="2025-12-02 22:47:11" level=info msg="Removing /etc/cni/multus/certs folder"
time="2025-12-02 22:47:11" level=info msg="Removing /var/seed_data folder"
time="2025-12-02 22:47:11" level=info msg="Restoring cluster services (i.e. kubelet.service unit)"
time="2025-12-02 22:47:11" level=info msg="Running systemctl enable [kubelet.service --now]"
time="2025-12-02 22:47:11" level=info msg="Executing nsenter with args [--target 1 --cgroup --mount --ipc --pid -- systemctl enable kubelet.service --now]"
time="2025-12-02 22:47:13" level=fatal msg="Failed to restore seed cluster: encountered 1 errors during restoration"
{code}
Version-Release number of selected component (if applicable):{code:none}
OCP 4.21.0-0.nightly-2025-11-22-193140
LCA
{code}
How reproducible:{code:none}
Always
{code}
Steps to Reproduce:{code:none}
1. Deploy SNO OCP 4.21 spoke for seed gen via ZTP applying RAN DU profile from RDS with the feature gate enabled with:
installConfigOverrides: |
{
"featureSet": "TechPreviewNoUpgrade"
}
2. Run seed gen job after seed deployment and monitor progress of the seedgen CR
3.
{code}
Actual results:{code:none}
seedgen fails due to container exit 1 status
{code}
Expected results:{code:none}
seedgen completes successfully.
{code}
Additional info:{code:none}
This issue does not happen if the TPNU feature gate is not enabled with deployment config the same otherwise.
{code}
Status: New
| |||
| #OCPBUGS-78010 | issue | 35 hours ago | [openshift-4.13] sriov-network-* images can't be built hermetically New |
Issue 17969096: [openshift-4.12] sriov-network-* images can't be built hermetically
Description: This is a clone of issue OCPBUGS-78009. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78007. The following is the description of the original issue: --- Description of problem:
ART can't build sriov-network-* images hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-74938 | issue | 30 hours ago | Kubelet and NetworkManager do not start automatically on any node after reboot, leaving nodes stuck in NotReady CLOSED |
{code:java}
Jan 30 09:12:11 ocp-w1.ocp-prd-lv.afcucorp.local bash[3148]: Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2c3ae2f63061eb25dab528c0b904994ed08cbb76debc19fcb4604c2e57e6e785: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io on [::1]:53: read udp [::1]:48765->[::1]:53: read: connection refused{code}
From the logs, the runtime is failing with a “connection refused” error while trying to resolve quay.io using DNS at [::1]:53 (IPv6 loopback).
| |||
| #OCPBUGS-70326 | issue | 8 days ago | Inspection failure due to unreachable image shows another confusing error message ASSIGNED |
Issue 17727752: Inspection failure due to unreachable image shows another confusing error message
Description: Description of problem:
While inspecting a Dell machine (PowerEdge R640, iDRAC9) from a dev-scripts based BMO setup, inspection fails with this error message:
{code:java}
2026-01-02 14:24:24.256 1 ERROR ironic.conductor.task_manager [None req-1b5a3a1b-c989-4e24-9462-3a3f531eba68 - - - - - -] Node b26bc87d-cc12-46db-85b4-98d24b16de03 moved to provision state "inspect failed" from state "inspecting"; target provision state is "manageable": ironic.common.exception.HardwareInspectionFailure: Failed to inspect hardware. Reason: unable to start inspection: HTTP POST https://IP:443/redfish/v1/Systems/System.Embedded.1/VirtualMedia/1/Actions/VirtualMedia.InsertMedia returned code 500. Base.1.12.GeneralError: A general error has occurred. See ExtendedInfo for more information Extended information: [{'Message': 'Virtual Media is detached or Virtual Media devices are already in use.', 'MessageArgs': [], 'MessageArgs@odata.count': 0, 'MessageId': 'IDRAC.2.8.VRM0021', 'RelatedProperties': [], 'RelatedProperties@odata.count': 0, 'Resolution': 'Change the Virtual Media Attach Mode to Attach or Auto-Attach.Stop existing Virtual Media sessions and retry the operation.', 'Severity': 'Informational'}] {code}
There are no virtual media devices attached. the HTTP POST request to insert virtual media is as follows:
{code:java}
2026-01-02 14:24:21.939 1 DEBUG sushy.connector [None req-1b5a3a1b-c989-4e24-9462-3a3f531eba68 - - - - - -] HTTP request: POST https://<IP>:443/redfish/v1/Systems/System.Embedded.1/VirtualMedia/1/Actions/VirtualMedia.InsertMedia; headers: {'Content-Type': 'application/json', 'OData-Version': '4.0', 'Accept-Encoding': 'identity'}; body: {'Image': 'https://192.168.111.22:6183/redfish/boot-b26bc87d-cc12-46db-85b4 98d24b16de03.iso'}; blocking: False; timeout: 60; session arguments: {}; _op /usr/lib/python3.12/site-packages/sushy/connector.py:167 {code}
The image URL in the request body needs to be fixed, and the error message from ironic is misleading.
iDRAC firmware version was upgraded from 6.10.80.00 to 7.00.00.183, and the BIOS version was upgraded from 2.18.1 to 2.25.0, and the same issue was observed with both versions of both.
Version-Release number of selected component (if applicable):
{code:none}
4.20{code}
How reproducible:
{code:none}
Always{code}
Steps to Reproduce:
{code:none}
1. Inspect a (Dell?) BMH using a dev-scripts-based BMO and ironic
2.
3.{code}
Actual results:
{code:none}
Misleading error message{code}
Expected results:
{code:none}
Better error message about image being unreachable{code}
Additional info:
{code:none}
{code}
Status: ASSIGNED
| |||
| #OCPBUGS-76644 | issue | 3 weeks ago | ABI OVE: Initial virtual device mount fails New |
Issue 17872088: ABI OVE: Initial virtual device mount fails
Description: Description of problem:
{code:none}
There is an issue during the initial virtual mounting of the ISO. The manifest file inside the virtual mounting folder appears to be corrupt, which causes the local registry service to throw an error while attempting to pull images. Instead of displaying the appropriate failure message, the system proceeds with the next steps. {code}
Version-Release number of selected component (if applicable):
{code:none}
4.22{code}
How reproducible:
{code:none}
Intermittent {code}
Steps to Reproduce:
{code:none}
1. Boot the OVE ISO
2. Observe the start-local-registry.service
{code}
Actual results:
{code:none}
Feb 12 10:27:42 localhost systemd[1]: Starting Local Registry (OCP)...
Feb 12 10:27:51 localhost.localdomain podman[2252]: 2026-02-12 10:27:51.328585781 +0000 UTC m=+3.114220808 system refresh
Feb 12 10:27:51 localhost.localdomain setup-local-registry.sh[2252]: Error: read /mnt/agentdata/images/registry/manifest.json: input/output error
Feb 12 10:27:51 localhost.localdomain podman[2252]: 2026-02-12 10:27:51.353089288 +0000 UTC m=+3.138724318 image pull-error dir:/mnt/agentdata/images/registry read /mnt/agentdata/images/registry/manifest.json: input/output error
Feb 12 10:27:51 localhost.localdomain setup-local-registry.sh[2104]: Warning: Could not find image with digest sha256:0e889a92bb1795b586c5fb2531e7f64d8e33f98b3720b9408ece1feee1135ef3
Feb 12 10:27:51 localhost.localdomain setup-local-registry.sh[2104]: Generating self-signed certificate for local registry
Feb 12 10:27:51 localhost.localdomain setup-local-registry.sh[2362]: {code}
Expected results:
{code:none}
The ISO should mount successfully, and the start-local-registry.service should pull images without manifest-related errors.{code}
Additional info:
{code:none}
[core@master-01 ~]$ dmesg | tail -50
[ 2085.858677] loop262: detected capacity change from 0 to 6291456
[ 2085.861062] loop263: detected capacity change from 0 to 6291456
[ 2085.863354] loop264: detected capacity change from 0 to 6291456
[ 2085.865628] loop265: detected capacity change from 0 to 6291456
[ 2085.867838] loop266: detected capacity change from 0 to 6291456
[ 2085.870319] loop267: detected capacity change from 0 to 6291456
[ 2085.872614] loop268: detected capacity change from 0 to 6291456
[ 2085.875191] loop269: detected capacity change from 0 to 6291456
[ 2085.877599] loop270: detected capacity change from 0 to 6291456
[ 2085.879915] loop271: detected capacity change from 0 to 6291456
[ 2085.882252] loop272: detected capacity change from 0 to 6291456
[ 2085.884609] loop273: detected capacity change from 0 to 5096940
[ 2085.925774] podman: attempt to access beyond end of device
dm-0: rw=0, sector=103829356, nr_sectors = 4 limit=99468780{code}
The root cause is that when we replace the ISO in virtual media, Dell iDRAC sometimes retains the old session with the previous ISO, which causes the installation to fail. This can be resolved by restarting iDRAC.
Status: New
| |||
| #OCPBUGS-77165 | issue | 13 hours ago | Image not known errors causing mass failures across CI since cri-o 1.35.0 ASSIGNED |
Issue 17919457: Image not known errors causing mass failures across CI since cri-o 1.35.0
Description: Initially this was found as a install failure where the network operator complains that multus is still awaiting 1-2 nodes on gcp. However the situation has now evolved to where we can see the underlying issue on many tests, jobs, and platforms:
{code:java}
Error: reading image "92c3c9025fe657a9160372a805bfd1624fb087262f2e3db326f937c298d47d70": locating image with ID "92c3c9025fe657a9160372a805bfd1624fb087262f2e3db326f937c298d47d70": image not known
{code}
Comments below show job runs where this is being seen.
Scanning bigquery for tests reporting this in their output:
{code:java}
SELECT modified_time, prowjob_build_id, test_name, success, test_id, branch, prowjob_name, failure_content
FROM `openshift-gce-devel.ci_analysis_us.junit`
WHERE success = falseAND modified_time BETWEEN DATETIME("2026-02-10") AND DATETIME("2026-02-25")
AND failure_content like '%image not known%' order by modified_time asc
LIMIT 1000 {code}
Shows again, explosion of this problem on Feb 19. It is happening only in 4.22 jobs and a few presubmits, ruling out infrastructure or registries. (I think?) I will attach the results of the above query in json format.
Sample job runs:
[https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn-techpreview/2025590450445881344]
[https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-main-ci-4.22-e2e-azure-ovn-techpreview/2025540689521020928]
Or see the test details report below.
[Search.ci|https://search.dptools.openshift.org/?search=image+not+known&maxAge=48h&context=1&type=junit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job] can be used to find job runs impacted. It doesn't always load, but if you wait 5-10 minutes and try again it usually will.
h2. Original Report
Component Readiness has found a potential regression in the following test:
{code:none}
install should succeed: overall{code}
Extreme regression detected.
Fishers Exact probability of a regression: 100.00%.
Test pass rate dropped from 98.81% to 69.93%.
Sample (being evaluated) Release: 4.22
Start Time: 2026-02-16T00:00:00Z
End Time: 2026-02-23T12:00:00Z
Success Rate: 69.93%
Successes: 200
Failures: 86
Flakes: 0
Base (historical) Release: 4.20
Start Time: 2025-09-21T00:00:00Z
End Time: 2025-10-21T00:00:00Z
Success Rate: 98.81%
Successes: 1000
Failures: 12
Flakes: 0
View the [test details report|https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/test_details?Architecture=amd64&FeatureSet=default&Installer=ipi&LayeredProduct=none&Network=ovn&Platform=gcp&Suite=unknown&Topology=ha&Upgrade=micro&baseEndTime=2026-02-03T23%3A59%3A59Z&baseRelease=4.21&baseStartTime=2026-01-04T00%3A00%3A00Z&capability=Other&columnGroupBy=Network%2CPlatform%2CTopology&component=Installer+%2F+openshift-installer&confidence=95&dbGroupBy=Architecture%2CFeatureSet%2CInstaller%2CLayeredProduct%2CNetwork%2CPlatform%2CSuite%2CTopology%2CUpgrade&environment=Architecture%3Aamd64+FeatureSet%3Adefault+Installer%3Aipi+LayeredProduct%3Anone+Network%3Aovn+Platform%3Agcp+Suite%3Aunknown+Topology%3Aha+Upgrade%3Amicro&flakeAsFailure=false&ignoreDisruption=true&ignoreMissing=false&includeMultiReleaseAnalysis=true&includeVariant=Architecture%3Aamd64&includeVariant=Architecture%3Aarm64&includeVariant=Architecture%3Amulti&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Acrun&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=FeatureSet%3Atechpreview&includeVariant=Installer%3Ahypershift&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=JobTier%3Ablocking&includeVariant=JobTier%3Ainforming&includeVariant=JobTier%3Astandard&includeVariant=LayeredProduct%3Anone&includeVariant=LayeredProduct%3Avirt&includeVariant=Network%3Aovn&includeVariant=OS%3Arhcos9&includeVariant=Owner%3Aeng&includeVariant=Owner%3Aservice-delivery&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Arosa&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aexternal&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&includeVariant=Upgrade%3Amicro&includeVariant=Upgrade%3Aminor&includeVariant=Upgrade%3Anone&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2026-02-23T12%3A00%3A00Z&sampleRelease=4.22&sampleStartTime=2026-02-16T00%3A00%3A00Z&testBasisRelease=4.20&testId=cluster+install%3A0cb1bb27e418491b1ffdacab58c5c8c0] for additional context.
Several other gcp install variant combos are showing problems, this is just the most visible one. See the triage record link that will be added to this card in a comment shortly for the full list of all regressions, but there should be more than enough job runs to investigate in this report linked above.
I think I see multiple causes here, unclear if there's one unifying underlying cause. I see problems with storage volume mounting, network operator awaiting one node, possibly more.
Consider this card a bug to get gcp install rates back up from their current 70% success rate.
Filed by: [~dgoodwin@redhat.com]
From what I can see, almost all of these runs are also showing a failure for:
{code:java}
verify operator conditions network
{ Operator Progressing=True (Deploying): DaemonSet "/openshift-multus/multus-additional-cni-plugins" is not available (awaiting 2 nodes)}
{code}
Sometimes it's just one node. Yet all nodes seem to be fine.
The test seems to be showing regressed in other areas too, here's a [metal regression that looks similar|https://sippy-auth.dptools.openshift.org/sippy-ng/component_readiness/test_details?Architecture=amd64&FeatureSet=default&Installer=ipi&LayeredProduct=none&Network=ovn&Platform=metal&Suite=serial&Topology=ha&Upgrade=none&baseEndTime=2026-02-03T23%3A59%3A59Z&baseRelease=4.21&baseStartTime=2026-01-04T00%3A00%3A00Z&capability=operator-conditions&columnGroupBy=Network%2CPlatform%2CTopology&component=Networking+%2F+cluster-network-operator&confidence=95&dbGroupBy=Architecture%2CFeatureSet%2CInstaller%2CLayeredProduct%2CNetwork%2CPlatform%2CSuite%2CTopology%2CUpgrade&environment=Architecture%3Aamd64+FeatureSet%3Adefault+Installer%3Aipi+LayeredProduct%3Anone+Network%3Aovn+Platform%3Ametal+Suite%3Aserial+Topology%3Aha+Upgrade%3Anone&flakeAsFailure=false&ignoreDisruption=true&ignoreMissing=false&includeMultiReleaseAnalysis=true&includeVariant=Architecture%3Aamd64&includeVariant=Architecture%3Aarm64&includeVariant=Architecture%3Amulti&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Acrun&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=FeatureSet%3Atechpreview&includeVariant=Installer%3Ahypershift&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=JobTier%3Ablocking&includeVariant=JobTier%3Ainforming&includeVariant=JobTier%3Astandard&includeVariant=LayeredProduct%3Anone&includeVariant=LayeredProduct%3Avirt&includeVariant=Network%3Aovn&includeVariant=OS%3Arhcos9&includeVariant=Owner%3Aeng&includeVariant=Owner%3Aservice-delivery&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Arosa&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aexternal&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&includeVariant=Upgrade%3Amicro&includeVariant=Upgrade%3Aminor&includeVariant=Upgrade%3Anone&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2026-02-23T12%3A00%3A00Z&sampleRelease=4.22&sampleStartTime=2026-02-16T00%3A00%3A00Z&testId=openshift-tests%3A25ad49f2396feeec12df2da38f57bc3f].
[Global test analysis|https://sippy-auth.dptools.openshift.org/sippy-ng/tests/4.22/analysis?test=verify%20operator%20conditions%20network&filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22verify%20operator%20conditions%20network%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D] shows a sharp decline probably around the 19th.
Mostly hitting gcp but some metal in the mix as well.
Status: ASSIGNED
{code}
event happened 25 times, something is wrong: namespace/openshift-ingress node/ci-op-mw99dt7j-fb783-4fdxg-worker-eastus1-jjtbt pod/gateway-cl995-openshift-default-79dd68f786-pkzcq hmsg/f04e4a5657 - reason/Failed Error: reading image "92c3c9025fe657a9160372a805bfd1624fb087262f2e3db326f937c298d47d70": locating image with ID "92c3c9025fe657a9160372a805bfd1624fb087262f2e3db326f937c298d47d70": image not known (14:27:15Z) result=reject }
{code}
{code}
console-7b4b4768bb-66s4w hmsg/efa454141e - reason/Failed Error: reading image "a81a5024f77f881f002b28dc2d8be8adfbc4c63e3ee7981c0c169bc6f03cf46e": locating image with ID "a81a5024f77f881f002b28dc2d8be8adfbc4c63e3ee7981c0c169bc6f03cf46e": image not known (17:54:22Z) result=reject event happened 557 times, something is wrong: namespace/openshift-console node/ip-10-0-17-146.ec2.internal pod/console-7b4b4768bb-66s4w hmsg/6cba065bf0 - reason/Pulled Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:923d6a13397a5a3b506536c6bdce57b37aaa0bf5b95c952f59d49f76b0a00e8f" already present on machine (17:54:22Z) result=reject }
{code}
| |||
| #OCPBUGS-78024 | issue | 34 hours ago | [openshift-4.13] ose-vsphere-cluster-api-controllers can't be built hermetically New |
Issue 17969358: [openshift-4.13] ose-vsphere-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78023. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78021. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78020. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78019. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78018. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78017. The following is the description of the original issue: --- Description of problem:
ART can't build ose-vsphere-cluster-api-controllers hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77627 | issue | 7 days ago | [4.18] openshift-cluster-csi-drivers vmware-vsphere-csi-driver-operator-metrics endpoint 8445 port should return 401 instead 500 ASSIGNED |
Issue 17948243: [4.18] openshift-cluster-csi-drivers vmware-vsphere-csi-driver-operator-metrics endpoint 8445 port should return 401 instead 500
Description: This is a clone of issue OCPBUGS-77007. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-75912. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-74569. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-60159. The following is the description of the original issue: --- Description of problem:
see similar bug OCPBUGS-57585
Stole the testing result from [https://github.com/openshift/origin/pull/30014#issuecomment-3153130188]
8445 port is for openshift-cluster-csi-drivers vmware-vsphere-csi-driver-operator-metrics endpoint, did not see this port on other IAAS(example AWS,OSP)checked on vsphere, curl 8445 without auth returns 500(actually it's forbidden),
{noformat}
$ oc get infrastructure/cluster -o jsonpath={.spec.platformSpec.type}
VSphere
$ oc -n openshift-cluster-csi-drivers get ep
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME ENDPOINTS AGE
vmware-vsphere-csi-driver-controller-metrics 10.129.0.17:9203,10.130.0.21:9203,10.129.0.17:9202 + 9 more... 28m
vmware-vsphere-csi-driver-operator-metrics 10.130.0.17:8445 28m
vmware-vsphere-csi-driver-webhook-svc 10.128.0.45:8443,10.130.0.23:8443 28m
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -s -o /dev/null -w '%{http_code}' https://10.130.0.17:8445/metrics
500
{noformat}
return code is 500, expecting 401 forbidden instead of 500.
{code:java}
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k https://10.130.0.17:8445/metrics
Internal Server Error: "/metrics": subjectaccessreviews.authorization.k8s.io is forbidden: User "system:serviceaccount:openshift-cluster-csi-drivers:vmware-vsphere-csi-driver-operator" cannot create resource "subjectaccessreviews" in API group "authorization.k8s.io" at the cluster scope {code}
curl with auth, no issue
{noformat}
$ token=`oc create token prometheus-k8s -n openshift-monitoring`
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://10.130.0.17:8445/metrics' | head -n3
# HELP aggregator_discovery_aggregation_count_total [ALPHA] Counter of number of times discovery was aggregated
# TYPE aggregator_discovery_aggregation_count_total counter
aggregator_discovery_aggregation_count_total 0 {noformat}
[https://github.com/openshift/origin/pull/30014#issuecomment-3154900379]
When the bug is fixed, please remove it from this code snippet to avoid regression.
[https://github.com/openshift/origin/blob/4f183dd3427cffd8d97b44557caa782d65726416/test/extended/prometheus/prometheus.go#L64-L71]
Status: ASSIGNED
| |||
| #OCPBUGS-78174 | issue | 9 hours ago | [openshift-4.13] ose-machine-api-provider-openstack can't be built hermetically New |
Issue 17974704: [openshift-4.13] ose-machine-api-provider-openstack can't be built hermetically
Description: This is a clone of issue OCPBUGS-78172. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78171. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78170. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78169. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78168. The following is the description of the original issue: --- Description of problem:
ART can't build ose-machine-api-provider-openstack hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77278 | issue | 7 days ago | BMH stuck in deleting state while performing IBBF Verified |
Issue 17931739: BMH stuck in deleting state while performing IBBF
Description: This is a clone of issue OCPBUGS-76988. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-59133. The following is the description of the original issue: --- Description of problem:
{code:none}
While attempting to perform IBBF with the siteconfig operator, the BMH for the spoke is stuck in deleting state for too long due to the old BMH being unreachable. This results in the clusterinstance timing out while waiting for the BMH resource to be removed. We should find a way to mitigate this.
{code}
Version-Release number of selected component (if applicable):
{code:java}
4.18.0-0.nightly-2025-04-02-011053 {code}
How reproducible:
{code:none}
100%{code}
Steps to Reproduce:
{code:none}
1. Install an IBI spoke cluster using siteconfig operator
2. Remove old node and attempt to replace with new node through IBBF
3.
{code}
Actual results:
{code:none}
Reinstallation fails due to timeout{code}
Expected results:
{code:none}
Reinstallation succeeds{code}
Additional info:
{code:none}
$ oc get clusterinstance target-0 -o json | jq -r '.status.reinstall.conditions[0]'
{
"lastTransitionTime": "2025-04-04T17:24:45Z",
"message": "Encountered error executing task: Deleting rendered manifests. Error: deletion timeout exceeded for object (BareMetalHost:target-0/target-0-0): Timed out waiting to delete object (BareMetalHost:target-0/target-0-0)",
"reason": "Failed",
"status": "False",
"type": "ReinstallRequestProcessed"
}
$ oc get bmh target-0-0 -o json | jq -r .status.errorMessage
Failed to get power state for node 7e063860-56c7-45ca-8bfa-d8f63997bd10. Error: Redfish exception occurred. Error: Resource https://[fd2e:6f44:5dd8:4::1]:8000/redfish/v1/Systems/ac8d214a-ae10-46f5-a179-208341bcb688 not found {code}
Status: Verified
| |||
| #OCPBUGS-77192 | issue | 2 weeks ago | [openshift-4.20] ose-libvirt-machine-controllers can't be build hermetically New |
Issue 17925856: [openshift-4.20] ose-libvirt-machine-controllers can't be build hermetically
Description: Description of problem:
ART can't build ose-libvirt-machine-controllers hermetically.
The error is the following in the prefetch-dependencies konflux task:
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details. Please try running `go mod vendor` and committing the changes. Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-76610 | issue | 6 hours ago | MCD Validation Discrepancy during Interrupted rpm-ostree Finalization New |
Issue 17867884: MCD Validation Discrepancy during Interrupted rpm-ostree Finalization Description: *Summary:* MCD enters Degraded loop when `rpm-ostree` finalization fails due to discrepancy between on-disk `currentconfig` and Node annotation Issue Type: Bug Priority: High (Severity 2) Component: Machine Config Operator (MCO) , RHCOS Labels: shift_sno, mco, rpm-ostree *Description :* In OpenShift 4.18/4.19, the Machine Config Daemon (MCD) can enter a persistent "Degraded" state if a node reboot is interrupted or if `ostree-finalize-staged.service` fails to complete (e.g., due to a timeout). During this failure mode, the MCD writes the "new" configuration to the on-disk `currentconfig` file before the kernel has successfully pivoted into it. Upon reboot, the MCD prioritizes the on-disk file over the Node's `currentConfig` annotation, leading to a validation failure because the actual running state (kernel arguments, etc.) matches the old configuration, not the "new" one recorded on disk. *Steps to Reproduce :* 1. Trigger a Configuration Change: Apply a `MachineConfig` that requires a reboot and additional kernel arguments (e.g., `audit=1`, `page_poison=1`). 2. Simulate/Induce Finalization Failure: Interrupt the `ostree-finalize-staged.service` during the shutdown/reboot process so it exits with a 'timeout' or failure result. 3. Boot into Old Deployment: Allow the node to reboot. Because finalization failed, the node remains on the previous RHCOS deployment/kernel version. 4. Observe MCD Logs: Check the logs of the `machine-config-daemon` pod. *Actual Results :* * Validation Error: MCD detects that the on-disk file overrides the node annotation: `"Disk currentConfig \"rendered-master-NEW\" overrides node's currentConfig annotation \"rendered-master-OLD\""`. * State Mismatch: MCD validates the system against the NEW config but finds the OLD state, resulting in a Degraded status: `"unexpected on-disk state validating against rendered-master-NEW: missing expected kernel arguments: [...]"`. * Persistent Loop: The node remains Degraded and fails to reconcile because it is stuck validating against a configuration it never successfully booted. *Expected Results :* * The MCD should verify that the booted deployment matches the configuration defined in the on-disk `currentconfig` before using it for validation. * If a discrepancy exists (i.e., the pivot failed), the MCD should fallback to the Node's `currentConfig` annotation or automatically clean up the stale on-disk file to allow for a clean reconciliation. *Environment Information :* * OCP Version: 4.18 / 4.19 * Platform: Single Node OpenShift (SNO), Disconnected, Bare Metal * OS: RHCOS * Disconnected Air Gapped and Secured Environment *Workaround :* 1. Access the node via SSH. 2. Backup and delete the stale on-disk config: `rm /etc/machine-config-daemon/currentconfig`. 3. Reboot the node: `systemctl reboot`. 4. After reboot, the MCD will correctly reconcile against the Node annotation. Status: New | |||
| #OCPBUGS-77212 | issue | 3 days ago | [openshift-4.14] hermetic build failure - ose-ibm-vpc-block-csi-driver Verified |
Issue 17926182: [openshift-4.14] hermetic build failure - ose-ibm-vpc-block-csi-driver
Description: This is a clone of issue OCPBUGS-77211. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77210. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77209. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77207. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77206. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77205. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77203. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77167. The following is the description of the original issue: --- Description of problem:
{code:none}
IBM VPC Block CSI Driver build fails in hermetic mode due to non-deterministic GOPATH handling in Makefile. The build system rewrites $GOPATH causing the binary output path -o ${GOPATH}/bin/${EXE_DRIVER_NAME} to become invalid, resulting in build failures.{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift 4.19+ (affects hermetic build conversion) {code}
How reproducible:
{code:none}
Always - when building ibm-vpc-block-csi-driver in hermetic mode{code}
Steps to Reproduce:
{code:none}
1. Build ibm-vpc-block-csi-driver container in hermetic mode
2. Makefile uses absolute GOPATH-dependent output path: -o ${GOPATH}/bin/${EXE_DRIVER_NAME}
3. Build system rewrites $GOPATH during hermetic build process {code}
Actual results:
{code:none}
Build fails because binary cannot be found at expected GOPATH location after build system modifies $GOPATH
Error: building at STEP "COPY --from=builder /cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver /bin/ibm-vpc-block-csi-driver": checking on sources under [...]: copier: stat: "/cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver": no such file or directory{code}
Expected results:
{code:none}
Build succeeds using relative path bin/${EXE_DRIVER_NAME} that doesn't depend on GOPATH manipulation, consistent with other CSI drivers like aws-ebs-csi-driver {code}
Additional info:
{code:none}
Fix implemented: Change Makefile to use relative path -o bin/${EXE_DRIVER_NAME} instead of absolute GOPATH-dependent path. This aligns with other CSI drivers and makes builds hermetic-compatible. Comment added to indicate this is a permanent carry patch for OpenShift builds.{code}
Status: Verified
| |||
| #OCPBUGS-55106 | issue | 5 weeks ago | RHCOS 4.19 live iso x86_64 contains restrictive policy.json Verified |
Issue 16833641: RHCOS 4.19 live iso x86_64 contains restrictive policy.json
Description: Description of problem:
{code:none}
Starting April 10th, RHCOS 4.19 live iso (from 4.19 ocp installer reference) contains a restrictive */etc/containers/policy.json* that causes OCP clusters deployed by assisted service to fail in certain scenarios.
{code}
Version-Release number of selected component (if applicable):
{code:none}
Trying to deploy 4.19.0-ec4
Booted image information:
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="9.6.20250321-0 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.6"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 9.6.20250321-0 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.6"
OSTREE_VERSION='9.6.20250321-0'
VARIANT=CoreOS
VARIANT_ID=coreos
Red Hat Enterprise Linux release 9.6 (Plow)
Red Hat Enterprise Linux release 9.6 (Plow) {code}
How reproducible:
{code:none}
100%
{code}
Steps to Reproduce:
{code:none}
1. Run coreos image
2. cat /etc/containers/policy.json
{code}
Actual results:
{code:none}
[core@sno ~]$ cat /etc/containers/policy.json
{
"default": [
{
"type": "insecureAcceptAnything"
}
],
"transports": {
"docker": {
"registry.access.redhat.com": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPaths": ["/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release", "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta"]
}
],
"registry.redhat.io": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPaths": ["/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release", "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta"]
}
]
},
"docker-daemon": {
"": [
{
"type": "insecureAcceptAnything"
}
]
}
}
} {code}
Expected results:
{code:none}
{
"default": [
{
"type": "insecureAcceptAnything"
}
],
"transports": {
"docker-daemon": {
"": [
{
"type": "insecureAcceptAnything"
}
]
}
}
}
{code}
{code:java}
There was no change to the configuration and the deployment the day before (On April 9th completed successfully). The issue persists since then on every deployment attempt.{code}
The issue itself is that with those records, agent image cannot be downloaded:
{code:java}
Apr 17 12:05:33 sno.xr11.kni-qe-9.lab.eng.rdu2.redhat.com systemd[1]: Starting agent.service...
Apr 17 12:05:34 sno.xr11.kni-qe-9.lab.eng.rdu2.redhat.com podman[971303]: Trying to pull registry.redhat.io/multicluster-engine/assisted-installer-agent-rhel9@sha256:49f93c9d88aac3a1a4bb9df2f46d52ea2581c05095a42b01993ea15594b21aa9...
Apr 17 12:05:34 sno.xr11.kni-qe-9.lab.eng.rdu2.redhat.com podman[971303]: Error: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists
Apr 17 12:05:34 sno.xr11.kni-qe-9.lab.eng.rdu2.redhat.com podman[971303]: 2025-04-17 12:05:34.17935456 +0000 UTC m=+0.165353038 image pull-error registry.redhat.io/multicluster-engine/assisted-installer-agent-rhel9@sha256:49f93c9d88aac3a1a4bb9df2f46d52ea2581c05095a42b01993ea1559>
Apr 17 12:05:34 sno.xr11.kni-qe-9.lab.eng.rdu2.redhat.com systemd[1]: agent.service: Control process exited, code=exited, status=125/n/a
Apr 17 12:05:34 sno.xr11.kni-qe-9.lab.eng.rdu2.redhat.com systemd[1]: agent.service: Failed with result 'exit-code'.
Apr 17 12:05:34 sno.xr11.kni-qe-9.lab.eng.rdu2.redhat.com systemd[1]: Failed to start agent.service.
{code}
Once `docker` section is removed the image is pulled and the deployment continues.
Status: Verified
Apr 17 12:05:34 sno.xr11.kni-qe-9.lab.eng.rdu2.redhat.com podman[971303]: Trying to pull registry.redhat.io/multicluster-engine/assisted-installer-agent-rhel9@sha256:49f93c9d88aac3a1a4bb9df2f46d52ea2581c05095a42b01993ea15594b21aa9...
Apr 17 12:05:34 sno.xr11.kni-qe-9.lab.eng.rdu2.redhat.com podman[971303]: Error: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists
{code}
| |||
| #OCPBUGS-78101 | issue | 17 hours ago | [openshift-4.14] baremetal-machine-controller can't be built hermetically New |
Issue 17972263: [openshift-4.14] baremetal-machine-controller can't be built hermetically
Description: This is a clone of issue OCPBUGS-78100. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78099. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78098. The following is the description of the original issue: --- Description of problem:
ART can't build baremetal-machine-controller hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-78021 | issue | 34 hours ago | [openshift-4.15] ose-vsphere-cluster-api-controllers can't be built hermetically New |
Issue 17969353: [openshift-4.15] ose-vsphere-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78020. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78019. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78018. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78017. The following is the description of the original issue: --- Description of problem:
ART can't build ose-vsphere-cluster-api-controllers hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77197 | issue | 2 weeks ago | [openshift-4.15] ose-libvirt-machine-controllers can't be build hermetically New |
Issue 17925863: [openshift-4.15] ose-libvirt-machine-controllers can't be build hermetically
Description: Description of problem:
ART can't build ose-libvirt-machine-controllers hermetically.
The error is the following in the prefetch-dependencies konflux task:
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details. Please try running `go mod vendor` and committing the changes. Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-59211 | issue | 15 hours ago | Too many logs related to retried oauthaccesstokens from openshift-kube-storage-version-migrator namespace POST |
Issue 17134880: Too many logs related to retried oauthaccesstokens from openshift-kube-storage-version-migrator namespace
Description: Description of problem:
Too many logs related to multiple retried oauthaccesstokens from openshift-kube-storage-version-migrator namespace. Example logs are mentioned below:
{code:java}
2025-07-07T07:53:01.002259691Z W0707 07:53:01.002152 1 core.go:223] migration of sha256~xxxxxxxxxxxx will be retried: Operation cannot be fulfilled on oauthaccesstokens.oauth.openshift.io "sha256~xxxxxxxxxxxx": StorageError: invalid object, Code: 4, Key: /openshift.io/oauth/accesstokens/sha256~xxxxxxxxxxxx, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 1c3xxxx-7xxx-4axx-9xxx-fxxxxxxxxxx0, UID in object meta: {code}
The issue keeps re-occurring with different tokens.
Status: New
| |||
| #OCPBUGS-77649 | issue | 6 days ago | [openshift-4.22] hermetic build failure - monitoring-plugin Verified |
Issue 17948613: [openshift-4.22] hermetic build failure - monitoring-plugin
Description: Description of problem:
{code:none}
nmstate-console-plugin fails to build in hermetic mode due to yarn package manager incompatibility and Cypress test framework attempting network downloads during build process.
{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift Container Platform 4.14+
nmstate-console-plugin component across all affected versions
{code}
How reproducible:
{code:none}
Always - occurs when converting any nmstate-console-plugin build from network_mode: open to hermetic mode
{code}
Steps to Reproduce:
{code:none}
1. Remove network_mode: open from nmstate-console-plugin.yml configuration
2. Attempt hermetic build via Konflux build system
3. Build fails during npm install phase when Cypress tries to download binaries
{code}
Actual results:
{code:none}
npm error Installing Cypress (version: 12.17.4)
npm error The Cypress App could not be downloaded.
npm error URL: https://download.cypress.io/desktop/12.17.4?platform=linux&arch=x64
npm error Error: getaddrinfo ENOTFOUND download.cypress.io
Build fails because:
- Yarn package manager not supported by cachi2 for hermetic dependency management
- Cypress testing framework requires network access to download.cypress.io
- Custom yarn artifacts and dockerfile modifications conflict with hermetic requirements
{code}
Expected results:
{code:none}
nmstate-console-plugin builds successfully in hermetic mode without requiring network access during build process. Component should use npm package manager and disable Cypress binary
downloads via CYPRESS_INSTALL_BINARY=0 environment variable.
{code}
Additional info:
{code:none}
Root cause: Component uses yarn package manager which is incompatible with cachi2 hermetic builds, plus Cypress test framework downloads binaries during build.
Solution requires:
1. Upstream yarn→npm migration (similar to networking-console-plugin)
2. Set CYPRESS_INSTALL_BINARY=0 to disable test framework downloads
3. Remove yarn-specific artifacts and dockerfile modifications
4. Update ocp-build-data package manager configuration
Reference upstream work: https://github.com/openshift/nmstate-console-plugin/pull/169
Similar successful conversion: networking-console-plugin PRs #347 and #331
Tracking: ART-14616 (parent task with technical analysis)
{code}
Status: Verified
| |||
| #OCPBUGS-77676 | issue | 7 days ago | NNCP failing to reconfigure due to linux-bridge having extra ports after already configured CLOSED |
Issue 17950683: NNCP failing to reconfigure due to linux-bridge having extra ports after already configured
Description: Description of problem:
{code:none}
Linux bridge was configured by nmstate to be used by cnv-bridge cni for vms. After configuration ports were added to bridge by cni, some event triggered nmstate to reconfigure/validate configuration and now it is in degraded/FailedToConfigure state due to extra ports. {code}
Version-Release number of selected component (if applicable):
{code:none}
4.18{code}
How reproducible:
{code:none}
Unknown {code}
Steps to Reproduce:
{code:none}
1. Create a linux bridge with nmstate with nncp
spec:
desiredState:
interfaces:
- mtu: 9000
name: bond0.33
state: up
type: vlan
vlan:
base-iface: bond0
id: 33
- bridge:
options:
stp:
enabled: false
port:
- name: bond0.33
vlan: {}
description: Linux bridge with bond0 as a port on vlan 33
mtu: 9000
name: br0.33
state: up
type: linux-bridge
2. create a net-attach-def for br0.33
spec:
config: |
{
"cniVersion": "0.3.1",
"name": "vlan-33",
"type": "cnv-bridge",
"bridge": "br0.33",
"macspoofchk": true
}
3. After nmstate configures bridge, have cni add ports
4. Wait for an event that triggers nmstate to re-validate or re-configure (Not sure what triggers this nmstate to reconfigure/re-validate )
{code}
Actual results:
{code:none}
NNCP vlan-33-bridge Degraded FailedToConfigure
NNCE Failing
Logs: [2026-02-17T15:56:54Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: br0.33.interface.bridge.port desire '[{"name":"bond0.33","vlan":{}}]', current '[{"name":"bond0.33","stp-hairpin-mode":false,"stp-path-cost":100,"stp-priority":32,"vlan":{}},{"name":"veth8abc333d","stp-hairpin-mode":false,"stp-path-cost":2,"stp-priority":32,"vlan":{}}]'
{code}
Expected results:
{code:none}
For nmstate to not fail as its already configured bridge and allow CNI is adding ports. {code}
Additional info:
{code:none}
Allow linux bridges to use something similar to what can be configured for ovs bridges (allow-extra-patch-ports: true) {code}
Status: CLOSED
| |||
| #OCPBUGS-58262 | issue | 8 weeks ago | podman Error: trying to reuse blob - failed to verify certificate: x509: certificate signed by unknown authority ASSIGNED |
Issue 17106784: podman Error: trying to reuse blob - failed to verify certificate: x509: certificate signed by unknown authority
Description: Description of problem:
{code:none}
Can't reuse blob to push aws c2s bucket with x509 error since Dec. And on prow ci result, it failed in pull image from internal registry{code}
Version-Release number of selected component (if applicable):
{code:none}
4.18{code}
How reproducible:
{code:none}
always{code}
Steps to Reproduce:
{code:none}
1.Set up an aws c2s cluster with 4.18 using prow
https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.18-amd64-nightly-aws-sc2s-ipi-disc-priv-fips-f2/1865304153111138304/artifacts/aws-sc2s-ipi-disc-priv-fips-f2/openshift-extended-test/build-log.txt
build logs : 2024-12-07T11:25:33.257156973Z Replaced Dockerfile FROM image quay.io/openshifttest/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f
2024-12-07T11:25:35.613930151Z time="2024-12-07T11:25:35Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
2024-12-07T11:25:35.614776370Z I1207 11:25:35.614748 1 defaults.go:112] Defaulting to storage driver "overlay" with options [mountopt=metacopy=on].
2024-12-07T11:25:35.629453348Z Caching blobs under "/var/cache/blobs".
2024-12-07T11:25:35.632143910Z
2024-12-07T11:25:35.632143910Z Pulling image image-registry.openshift-image-registry.svc:5000/e2e-test-default-image-registry-zgmf2/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f ...
2024-12-07T11:25:35.636791870Z Trying to pull image-registry.openshift-image-registry.svc:5000/e2e-test-default-image-registry-zgmf2/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f...
2024-12-07T11:25:36.695243084Z Warning: Pull failed, retrying in 5s ...
2024-12-07T11:25:41.698639710Z Trying to pull image-registry.openshift-image-registry.svc:5000/e2e-test-default-image-registry-zgmf2/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f...
2024-12-07T11:25:41.848354202Z Warning: Pull failed, retrying in 5s ...
2024-12-07T11:25:46.852090834Z Trying to pull image-registry.openshift-image-registry.svc:5000/e2e-test-default-image-registry-zgmf2/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f...
2024-12-07T11:25:47.406620628Z Warning: Pull failed, retrying in 5s ...
2024-12-07T11:25:52.439067513Z error: build error: failed to pull image: After retrying 2 times, Pull image still failed due to error: copying system image from manifest list: parsing image configuration: Get "https://ci-op-5gnlt4mn-c6dd8-6xdhm-image-registry-us-isob-east-1-sxuku.s3.us-isob-east-1.sc2s.sgov.gov/docker/registry/v2/blobs/sha256/b9/b97242f89c8a29d13aea12843a08441a4bbfc33528f55b60366c1d8f6923d0d4/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAUW2DH7NUFSKUEAEQ%2F20241207%2Fus-isob-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241207T112547Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=6aba8b40b4b4f43e49074fcdae2db73c10c648c93ce9fb4d15bcd114925ef713": tls: failed to verify certificate: x509: certificate signed by unknown authority
2. But when reproduce it on prow ci cluster. we could push image at first try as always, but failed to push with reuse blob.
oc get builds
NAME TYPE FROM STATUS STARTED DURATION
test-registry-1 Docker Dockerfile Complete 2 hours ago 15s
test-registry-2 Docker Dockerfile Failed (PushImageToRegistryFailed) 2 hours ago 21s
oc logs -f test-registry-2-build
error: build error: Failed to push image: trying to reuse blob sha256:530afca65e2ea04227630ae746e0c85b2bd1a179379cbf2b6501b49c4cab2ccc at destination: Head "https://ci-op-qf55lf0q-3c6ac-j65q8-image-registry-us-iso-east-1-soajkc.s3.us-iso-east-1.c2s.ic.gov/docker/registry/v2/blobs/sha256/53/530afca65e2ea04227630ae746e0c85b2bd1a179379cbf2b6501b49c4cab2ccc/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAUW2DH7NUKYUBPOG2%2F20241210%2Fus-iso-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241210T070804Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=f5a44e20c1f5548e8cbec55f0faa0677fea306ecc8f1008a298a6171b8969a79": tls: failed to verify certificate: x509: certificate signed by unknown authority
3. Can't push a existing image to s3 bucket using podman
podman push image-registry.openshift-image-registry.svc:5000/wxj/test:latest
Getting image source signatures
Copying blob 1a8c6bfa0a12 done |
Copying blob 42e40303859e done |
Copying blob 9c6be2aa7dfd done |
Copying blob b80a8be6331b done |
Copying config cda9cd0379 done |
Writing manifest to image destination
sh-5.1# podman push image-registry.openshift-image-registry.svc:5000/wxj/test:latest
Getting image source signatures
Copying blob 9c6be2aa7dfd done |
Copying blob 42e40303859e done |
Copying blob 1a8c6bfa0a12 done |
Copying blob b80a8be6331b done |
Copying config cda9cd0379 [--------------------------------------] 8.0b / 20.7KiB | 7.8 MiB/s
Error: writing blob: checking whether a blob sha256:cda9cd0379eae8c992bf505926f7a9ea95f93b876cb55b15c9345b0d41a45388 exists in image-registry.openshift-image-registry.svc:5000/wxj/test: StatusCode: 403, "" {code}
Actual results:
{code:none}
Failed to pull or push reuse blob from/to image registry.{code}
Expected results:
{code:none}
Shouldn't fail to pull/push image from internal registry.{code}
Additional info:
{code:none}
must gather log https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.18-amd64-nightly-aws-sc2s-ipi-disc-priv-fips-f2/1865304153111138304/artifacts/aws-sc2s-ipi-disc-priv-fips-f2/gather-must-gather/{code}
Status: ASSIGNED
| |||
| #OCPBUGS-77763 | issue | 11 hours ago | [openshift-4.21] hermetic build failure - node-feature-discovery CLOSED |
Issue 17953316: [openshift-4.21] hermetic build failure - node-feature-discovery
Description: This is a clone of issue OCPBUGS-77651. The following is the description of the original issue: --- Description of problem:
{code:none}
Node Feature Discovery build fails in hermetic mode due to non-deterministic binary path handling in Dockerfile. The build uses "go install" command which places binaries in unpredictable GOPATH locations, and the Dockerfile copies from multiple inconsistent paths causing binary path conflicts.
{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift Container Platform 4.19+
node-feature-discovery component across all affected versions
{code}
How reproducible:
{code:none}
Always - when building node-feature-discovery in hermetic mode
{code}
Steps to Reproduce:
{code:none}
1. Build node-feature-discovery container in hermetic mode
2. Dockerfile copies binaries from multiple paths: /go/bin/* and /go/node-feature-discovery/bin/*
3. "go install" command places binaries in non-deterministic GOPATH locations during hermetic build process
{code}
Actual results:
{code:none}
Build fails because binaries are placed in unpredictable locations by "go install" and Dockerfile attempts to copy from multiple conflicting paths
Error: Inconsistent binary placement causing copy failures during Docker image build process when switching from "go install" to "go build" in
hermetic mode
{code}
Expected results:
{code:none}
Build succeeds using deterministic binary paths. Should use "go build" with explicit output paths instead of "go install" and copy binaries from single consistent location in Dockerfile.
{code}
Additional info:
{code:none}
Root cause: "go install" command places binaries in GOPATH-dependent locations that are not deterministic in hermetic builds, unlike "go build"
which allows explicit output path control.
Fix implemented: Remove redundant /go/bin/* copy command from Dockerfile and keep only the specific /go/node-feature-discovery/bin/* path. This
ensures consistent binary placement and eliminates path conflicts during hermetic builds.
Reference upstream work: https://github.com/openshift/node-feature-discovery/pull/163/changes
Related hermetic conversion tracking: ART-14586
{code}
Status: CLOSED
| |||
| #OCPBUGS-77414 | issue | 7 days ago | [openshift-4.22] hermetic build failure - networking-console-plugin ON_QA |
Issue 17936763: [openshift-4.22] hermetic build failure - networking-console-plugin
Description: Description of problem:
{code:none}
networking-console-plugin fails to build in hermetic mode due to yarn package manager incompatibility and Cypress test framework attempting network downloads during build process.
{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift Container Platform 4.16+ networking-console-plugin component across all affected versions
{code}
How reproducible:
{code:none}
Always - occurs when converting any networking-console-plugin build from network_mode: open to hermetic mode
{code}
Steps to Reproduce:
{code:none}
1. Remove network_mode: open from networking-console-plugin.yml configuration
2. Attempt hermetic build via Konflux build system
3. Build fails during npm install phase when Cypress tries to download binaries
{code}
Actual results:
{code:none}
npm error Installing Cypress (version: 12.17.4)
npm error The Cypress App could not be downloaded.
npm error URL: https://download.cypress.io/desktop/12.17.4?platform=linux&arch=x64
npm error Error: getaddrinfo ENOTFOUND download.cypress.io
Build fails because:
- Yarn package manager not supported by cachi2 for hermetic dependency management
- Cypress testing framework requires network access to download.cypress.io
- Custom yarn artifacts and dockerfile modifications conflict with hermetic requirements
{code}
Expected results:
{code:none}
networking-console-plugin builds successfully in hermetic mode without requiring network access during build process. Component should use npm package manager and disable Cypress binary
downloads via CYPRESS_INSTALL_BINARY=0 environment variable.
{code}
Additional info:
{code:none}
Root cause: Component uses yarn package manager which is incompatible with cachi2 hermetic builds, plus Cypress test framework downloads binaries during build.
Solution requires:
1. Upstream yarn→npm migration (similar to networking-console-plugin)
2. Set CYPRESS_INSTALL_BINARY=0 to disable test framework downloads
3. Remove yarn-specific artifacts and dockerfile modifications
4. Update ocp-build-data package manager configuration
Reference upstream work: https://github.com/openshift/networking-console-plugin/pull/357
Tracking: ART-14633 (parent task with technical analysis)
{code}
Status: ON_QA
| |||
| #OCPBUGS-78014 | issue | 34 hours ago | [openshift-4.12] sriov-cni can't be build hermetically New |
Issue 17969335: [openshift-4.12] sriov-cni can't be build hermetically
Description: This is a clone of issue OCPBUGS-78013. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78011. The following is the description of the original issue: --- Description of problem:
ART can't build sriov-cni hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-43501 | issue | 9 days ago | coreos-installer iso kargs show broken when using minimal ISO ASSIGNED |
Issue 16368909: coreos-installer iso kargs show broken when using minimal ISO
Description: Running the command coreos-installer iso kargs show no longer works in case of platform external.
{code:none}
$ coreos-installer iso kargs show agent.x86_64.iso
Writing manifest to image destination
Error: kargs don't match at all offsets! (expected 'rw ignition.firstboot ignition.platform.id=metal console=ttyS0 'coreos.live.rootfs_url=https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.17-9.4/builds/417.94.202407010929-0/x86_64/rhcos-417.94.202407010929-0-live-rootfs.x86_64.img'', but offset 112924489 has: '/images/assisted_installer_custom.img rw ignition.firstboot ignition.platform.id=metal console=ttyS0 coreos.live.rootfs_url=https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.17-9.4/builds/417.94.202407010929-0/x86_64/rhcos-417.94.202407010929-0-live-rootfs.x86_64.img'){code}
Version-Release number of selected component (if applicable):
{code:none}
https://github.com/openshift/installer/pull/8886{code}
Status: ASSIGNED
| |||
| #OCPBUGS-70154 | issue | 6 weeks ago | iDRAC 10 firmware updates succeeded but with servicing error New |
Issue 17714671: iDRAC 10 firmware updates succeeded but with servicing error
Description: Description of problem:
{code:none}
Though iDRAC 10 firmware updates succeeded, oc describe shows servicing error. in addition, node did not return to Ready state:
Normal PowerOff 52m metal3-baremetal-controller Host soft powered off
Normal ServicingError 43m metal3-baremetal-controller Node 68a7d19d-ebc0-42b6-8599-4c36a05a0bd8 failed step {'args': {'settings': [{'component': 'bmc', 'url': 'http://fw.apps.jhajyahy-vv.qe.devcluster.openshift.com/firmimg.d10'}]}, 'interface': 'firmware', 'step': 'update', 'abortable': False, 'priority': 0}: HTTP GET https://10.26.16.132/redfish/v1/ returned code 503. Base.1.18.GeneralError: The service is temporarily unavailable. Retry in 30 seconds. Extended information: [{'MessageId': 'Base.1.18.ServiceTemporarilyUnavailable', 'MessageArgs': ['30'], 'RelatedProperties': [], 'Message': 'The service is temporarily unavailable. Retry in 30 seconds.', 'Severity': 'Critical', 'Resolution': 'Wait for the indicated retry duration and retry the operation.'}]. Also failed to tear down servicing. {code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1. Perform FW update on IDRAC 10 server using HFC
2. Check "oc describe bmh -n openshift-machine-api <host>" command output
3.
{code}
Actual results:
{code:none}
passed with servicing errors{code}
Expected results:
{code:none}
To pass without servicing errors
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77981 | issue | 39 hours ago | [openshift-4.13] ose-machine-config-operator can't be built hermetically POST |
Issue 17967788: [openshift-4.13] ose-machine-config-operator can't be built hermetically
Description: This is a clone of issue OCPBUGS-77980. The following is the description of the original issue: --- Description of problem:
ART can't build ose-machine-config-operator hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-78018 | issue | 34 hours ago | [openshift-4.18] ose-vsphere-cluster-api-controllers can't be built hermetically POST |
Issue 17969349: [openshift-4.18] ose-vsphere-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-78017. The following is the description of the original issue: --- Description of problem:
ART can't build ose-vsphere-cluster-api-controllers hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-78182 | issue | 9 hours ago | [openshift-4.13] ose-machine-api-provider-gcp can't be built hermetically POST |
Issue 17974862: [openshift-4.13] ose-machine-api-provider-gcp can't be built hermetically
Description: This is a clone of issue OCPBUGS-78181. The following is the description of the original issue: --- Description of problem:
ART can't build ose-machine-api-provider-gcp hermetically
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-77213 | issue | 3 hours ago | [openshift-4.13] hermetic build failure - ose-ibm-vpc-block-csi-driver ON_QA |
Issue 17926440: [openshift-4.13] hermetic build failure - ose-ibm-vpc-block-csi-driver
Description: This is a clone of issue OCPBUGS-77212. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77211. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77210. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77209. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77207. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77206. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77205. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77203. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77167. The following is the description of the original issue: --- Description of problem:
{code:none}
IBM VPC Block CSI Driver build fails in hermetic mode due to non-deterministic GOPATH handling in Makefile. The build system rewrites $GOPATH causing the binary output path -o ${GOPATH}/bin/${EXE_DRIVER_NAME} to become invalid, resulting in build failures.{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift 4.19+ (affects hermetic build conversion) {code}
How reproducible:
{code:none}
Always - when building ibm-vpc-block-csi-driver in hermetic mode{code}
Steps to Reproduce:
{code:none}
1. Build ibm-vpc-block-csi-driver container in hermetic mode
2. Makefile uses absolute GOPATH-dependent output path: -o ${GOPATH}/bin/${EXE_DRIVER_NAME}
3. Build system rewrites $GOPATH during hermetic build process {code}
Actual results:
{code:none}
Build fails because binary cannot be found at expected GOPATH location after build system modifies $GOPATH
Error: building at STEP "COPY --from=builder /cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver /bin/ibm-vpc-block-csi-driver": checking on sources under [...]: copier: stat: "/cachi2/output/deps/gomod/bin/ibm-vpc-block-csi-driver": no such file or directory{code}
Expected results:
{code:none}
Build succeeds using relative path bin/${EXE_DRIVER_NAME} that doesn't depend on GOPATH manipulation, consistent with other CSI drivers like aws-ebs-csi-driver {code}
Additional info:
{code:none}
Fix implemented: Change Makefile to use relative path -o bin/${EXE_DRIVER_NAME} instead of absolute GOPATH-dependent path. This aligns with other CSI drivers and makes builds hermetic-compatible. Comment added to indicate this is a permanent carry patch for OpenShift builds.{code}
Status: Verified
| |||
| #OCPBUGS-77467 | issue | 11 days ago | [openshift-4.14] openshift-enterprise-hyperkube can't be built hermetically New |
Issue 17940935: [openshift-4.14] openshift-enterprise-hyperkube can't be built hermetically
Description: Description of problem:
ART can't build openshift-enterprise-hyperkube hermetically. Same applies for ose-installer-kube-apiserver-artifacts
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.
{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77979 | issue | 39 hours ago | [openshift-4.17] ose-machine-config-operator can't be built hermetically POST |
Issue 17967785: [openshift-4.17] ose-machine-config-operator can't be built hermetically
Description: This is a clone of issue OCPBUGS-77978. The following is the description of the original issue: --- Description of problem:
ART can't build ose-machine-config-operator hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-74133 | issue | 6 weeks ago | [4.19] oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9 POST |
Issue 17781204: [4.19] oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9
Description: This is a clone of issue OCPBUGS-73848. The following is the description of the original issue:
—
The oauth-proxy e2e-component tests fail on OCP 4.19 because RHEL 9 enforces fully qualified container image names ("short name mode").
The backend test image {{nginxdemos/nginx-hello:plain-text}} is rejected by Podman since it cannot determine which registry to pull from.
h2. Version-Release number
4.22
h2. How reproducible
Always
h2. Steps to Reproduce
Run the oauth-proxy e2e-component tests on OCP 4.22
The test creates a pod with a backend container using image {{nginxdemos/nginx-hello:plain-text}}
Observe the pod fails to start.
h2. Actual results
The backend container fails with:
{code:java}
ImageInspectError: short name mode is enforcing, but image name
nginxdemos/nginx-hello:plain-text returns ambiguous list
{code}
The pod never reaches Running state, and all e2e tests fail.
h2. Expected results
The e2e tests should pass. The backend container should start successfully using a fully qualified image reference.
h2. Additional info
_Root cause:_ RHEL 9 enforces "short name mode" for container images, requiring fully qualified registry paths.
_Fix:_ Replace {{nginxdemos/nginx-hello:plain-text}} with {{{}registry.k8s.io/e2e-test-images/agnhost:2.53{}}}, which is:
* Fully qualified (satisfies RHEL 9 short name policy)
* Multi-arch (supports ARM64)
* The standard test image adopted across OpenShift repositories ([openshift/origin PR #29672|https://github.com/openshift/origin/pull/29672])
* Mirrored in OpenShift CI infrastructure
_PR:_ [https://github.com/openshift/oauth-proxy/pull/343]
Status: POST
| |||
| #OCPBUGS-73851 | issue | 7 weeks ago | oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9 New |
Issue 17769459: oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9
Description: The oauth-proxy e2e-component tests fail on OCP 4.19 because RHEL 9 enforces fully qualified container image names ("short name mode"). The backend test image {{nginxdemos/nginx-hello:plain-text}} is rejected by Podman since it cannot determine which registry to pull from. h2. Version-Release number 4.19 h2. How reproducible Always h2. Steps to Reproduce h1. Run the oauth-proxy e2e-component tests on OCP 4.19 h1. The test creates a pod with a backend container using image {{nginxdemos/nginx-hello:plain-text}} h1. Observe the pod fails to start h2. Actual results The backend container fails with: {code} ImageInspectError: short name mode is enforcing, but image name nginxdemos/nginx-hello:plain-text returns ambiguous list {code} The pod never reaches Running state, and all e2e tests fail. h2. Expected results The e2e tests should pass. The backend container should start successfully using a fully qualified image reference. h2. Additional info _Root cause:_ RHEL 9 enforces "short name mode" for container images, requiring fully qualified registry paths. _Fix:_ Replace {{nginxdemos/nginx-hello:plain-text}} with {{registry.k8s.io/e2e-test-images/agnhost:2.53}}. _Backport of:_ [OCPBUGS-73848|https://issues.redhat.com/browse/OCPBUGS-73848]
Status: New
| |||
| #OCPBUGS-77862 | issue | 5 days ago | [openshift-4.17] openshift-enterprise-console-operator can't be built hermetically New |
Issue 17959746: [openshift-4.17] openshift-enterprise-console-operator can't be built hermetically
Description: This is a clone of issue OCPBUGS-77861. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-77860. The following is the description of the original issue: --- Description of problem:
ART can't build openshift-enterprise-console-operator hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-74694 | issue | 9 hours ago | Pods stuck terminating on cri-o unmounting overlayfs /merged path New |
Issue 17820540: Pods stuck terminating on cri-o unmounting overlayfs /merged path
Description: Description of problem:
{code:none}
Customer encounters pods stuck terminating with the following error. When listing mounts on the host the mount point does NOT show as mounted and no processes indicate using the path. However /proc/self/mountinfo does list the mountpoint. When we strace cri-o it's receiving -1EINVAL on the umount2() call to the relevant path.
Jan 24 06:51:04 worker crio[4437]: time="2026-01-24 06:51:04.487574617Z" level=warning msg="Failed to unmount container 9c634a49e087b3b7f6c48c8bf7fd6050d328bd5e985bd48e460de5743f98d78f: replacing mount point \"/var/lib/containers/storage/overlay/804f846ce959cc4fffeb930421231dc2298146fa8721732101b9870aee1217e5/merged\": device or resource busy" id=863adbef-0903-4953-a16c-d9018a970671 name=/runtime.v1.RuntimeService/StopPodSandbox
Jan 24 06:51:04 worker kubenswrapper[4476]: E0124 06:51:04.487776 4476 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to stop infra container for pod sandbox 9c634a49e087b3b7f6c48c8bf7fd6050d328bd5e985bd48e460de5743f98d78f: failed to unmount container 9c634a49e087b3b7f6c48c8bf7fd6050d328bd5e985bd48e460de5743f98d78f: replacing mount point \"/var/lib/containers/storage/overlay/804f846ce959cc4fffeb930421231dc2298146fa8721732101b9870aee1217e5/merged\": device or resource busy" podSandboxID="9c634a49e087b3b7f6c48c8bf7fd6050d328bd5e985bd48e460de5743f98d78f"
Jan 24 06:51:04 worker kubenswrapper[4476]: E0124 06:51:04.487830 4476 kuberuntime_manager.go:1479] "Failed to stop sandbox" podSandboxID={"Type":"cri-o","ID":"9c634a49e087b3b7f6c48c8bf7fd6050d328bd5e985bd48e460de5743f98d78f"}
Jan 24 06:51:04 worker kubenswrapper[4476]: E0124 06:51:04.487877 4476 kubelet.go:2041] "Unhandled Error" err="failed to \"KillPodSandbox\" for \"0f025850-3739-43c8-9e28-74c4bd792148\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to stop infra container for pod sandbox 9c634a49e087b3b7f6c48c8bf7fd6050d328bd5e985bd48e460de5743f98d78f: failed to unmount container 9c634a49e087b3b7f6c48c8bf7fd6050d328bd5e985bd48e460de5743f98d78f: replacing mount point \\\"/var/lib/containers/storage/overlay/804f846ce959cc4fffeb930421231dc2298146fa8721732101b9870aee1217e5/merged\\\": device or resource busy\"" logger="UnhandledError"
{code}
Version-Release number of selected component (if applicable):
{code:none}
OpenShift 4.18.24
cri-o-0-1.31.12-3.rhaos4.18.gitdc59c78
github.com/containers/storage v1.55.1-0.20250328180621-028e88d2378c
kernel-0-5.14.0-427.87.1.el9_4-x86_64 {code}
How reproducible:
{code:none}
Daily to weekly on two clusters in a loaded cluster with stackrox collectors and netobservability enabled. Sometimes happens in nodes that are only days old (elastic cloud environment).{code}
Steps to Reproduce:
{code:none}
Unknown {code}
Actual results:
{code:none}
Pods stuck in terminating state{code}
Expected results:
{code:none}
Pods terminate and cleanup {code}
Additional info:
{code:none}
This is a cluster running Stackrox 4.8.4 and Network Observability, both of which I understand to exert additional pressure on the kernel mount house keeping. However, the customer has plenty of other clusters with those components that don't encounter this problem so this may be not be valuable debugging information.
Container storage has to be reset and the node rebooted to clean things up. Often they just destroy the node or a machine health check does so.
The following kernel warnings are seen in the logs but we believe they're associated with debugging efforts rather than root cause. They happen around the time a debug toolbox container is created, additionally they do not happen in all nodes where the problem is observed. No other kernel messages are logged that seem related to this.
Jan 24 07:49:34 worker kernel: overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
Jan 24 07:49:34 worker kernel: overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.{code}
Status: New
| |||
| #OCPBUGS-78105 | issue | 17 hours ago | [openshift-4.14] cluster-network-operator can't be built hermetically POST |
Issue 17972270: [openshift-4.14] cluster-network-operator can't be built hermetically
Description: This is a clone of issue OCPBUGS-78104. The following is the description of the original issue: --- Description of problem:
ART can't build cluster-network-operator hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
Additional info:
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
# internal CI failure
# customer issue / SD
# internal RedHat testing failure
If it is an internal RedHat testing failure:
* Please share a kubeconfig or creds to a live cluster for the assignee to debug/troubleshoot along with reproducer steps (specially if it's a telco use case like ICNI, secondary bridges or BM+kubevirt).
If it is a CI failure:
* Did it happen in different CI lanes? If so please provide links to multiple failures with the same error instance
* Did it happen in both sdn and ovn jobs? If so please provide links to multiple failures with the same error instance
* Did it happen in other platforms (e.g. aws, azure, gcp, baremetal etc) ? If so please provide links to multiple failures with the same error instance
* When did the failure start happening? Please provide the UTC timestamp of the networking outage window from a sample failure run
* If it's a connectivity issue,
* What is the srcNode, srcIP and srcNamespace and srcPodName?
* What is the dstNode, dstIP and dstNamespace and dstPodName?
* What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
If it is a customer / SD issue:
* Provide enough information in the bug description that Engineering doesn’t need to read the entire case history.
* Don’t presume that Engineering has access to Salesforce.
* Do presume that Engineering will access attachments through supportshell.
* Describe what each relevant attachment is intended to demonstrate (failed pods, log errors, OVS issues, etc).
* Referring to the attached must-gather, sosreport or other attachment, please provide the following details:
** If the issue is in a customer namespace then provide a namespace inspect.
** If it is a connectivity issue:
*** What is the srcNode, srcNamespace, srcPodName and srcPodIP?
*** What is the dstNode, dstNamespace, dstPodName and dstPodIP?
*** What is the traffic path? (examples: pod2pod? pod2external?, pod2svc? pod2Node? etc)
*** Please provide the UTC timestamp networking outage window from must-gather
*** Please provide tcpdump pcaps taken during the outage filtered based on the above provided src/dst IPs
** If it is not a connectivity issue:
*** Describe the steps taken so far to analyze the logs from networking components (cluster-network-operator, OVNK, SDN, openvswitch, ovs-configure etc) and the actual component where the issue was seen based on the attached must-gather. Please attach snippets of relevant logs around the window when problem has happened if any.
* When showing the results from commands, include the entire command in the output.
* For OCPBUGS in which the issue has been identified, label with “sbr-triaged”
* For OCPBUGS in which the issue has not been identified and needs Engineering help for root cause, label with “sbr-untriaged”
* Do not set the priority, that is owned by Engineering and will be set when the bug is evaluated
* Note: bugs that do not meet these minimum standards will be closed with label “SDN-Jira-template”
* For guidance on using this template please see
[OCPBUGS Template Training for Networking components|https://source.redhat.com/groups/public/gss-openshift/cee_openshift_wiki/sbr_shift_networking#jive_content_id_Training_Materials]
Status: POST
| |||
| #OCPBUGS-74134 | issue | 6 weeks ago | [4.18] oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9 New |
Issue 17781205: [4.18] oauth-proxy e2e tests fail due to ambiguous container image short names on RHEL 9
Description: This is a clone of issue OCPBUGS-73848. The following is the description of the original issue:
—
The oauth-proxy e2e-component tests fail on OCP 4.18 because RHEL 9 enforces fully qualified container image names ("short name mode").
The backend test image {{nginxdemos/nginx-hello:plain-text}} is rejected by Podman since it cannot determine which registry to pull from.
h2. Version-Release number
4.22
h2. How reproducible
Always
h2. Steps to Reproduce
Run the oauth-proxy e2e-component tests on OCP 4.22
The test creates a pod with a backend container using image {{nginxdemos/nginx-hello:plain-text}}
Observe the pod fails to start.
h2. Actual results
The backend container fails with:
{code:java}
ImageInspectError: short name mode is enforcing, but image name
nginxdemos/nginx-hello:plain-text returns ambiguous list
{code}
The pod never reaches Running state, and all e2e tests fail.
h2. Expected results
The e2e tests should pass. The backend container should start successfully using a fully qualified image reference.
h2. Additional info
_Root cause:_ RHEL 9 enforces "short name mode" for container images, requiring fully qualified registry paths.
_Fix:_ Replace {{nginxdemos/nginx-hello:plain-text}} with {{{}registry.k8s.io/e2e-test-images/agnhost:2.53{}}}, which is:
* Fully qualified (satisfies RHEL 9 short name policy)
* Multi-arch (supports ARM64)
* The standard test image adopted across OpenShift repositories ([openshift/origin PR #29672|https://github.com/openshift/origin/pull/29672])
* Mirrored in OpenShift CI infrastructure
_PR:_ [https://github.com/openshift/oauth-proxy/pull/343]
Status: New
| |||
| #OCPBUGS-66129 | issue | 5 weeks ago | The gather job restart loop when upload incorrect password Verified |
then echo "Error: Required Parameters have not been provided. Exiting..." exit 1 | |||
| #OCPBUGS-77948 | issue | 4 days ago | TNF IPv6 Node Replacement Test misformats URL ASSIGNED |
Issue 17965686: TNF IPv6 Node Replacement Test misformats URL
Description: *Description of problem:*
{code:none}
The IPv6 recovery test fails node replacement because the IP url used for redfish doesn't put the IPv6 address in brackets.
{code}
*Version-Release number of selected component (if applicable):*
{code:none}
4.22 nightlies 3/6/26
{code}
*How reproducible:*
{code:none}
See IPv6 recovery test
{code}
*Steps to Reproduce:*
{code:none}
1. Look at IPv6 recovery test in CI
2. Check results of node replacement test
{code}
*Actual results:*
{code:none}
I0306 09:43:38.935142 19613 tnf_node_replacement.go:1489] BareMetalHost ostest-master-1 error message: Failed to validate power driver interface for node f5db7f08-753f-4ad9-87fa-82682bc2750d. Error: Invalid Redfish address https://fd2e:6f44:5dd8:c956::1:8000 set in driver_info/redfish_address on node f5db7f08-753f-4ad9-87fa-82682bc2750d
{code}
*Expected results:*
{code:none}
Test passes
{code}
Additional info:
{code:none}
{code}
Status: ASSIGNED
| |||
| #OCPBUGS-77458 | issue | 5 days ago | In rhel 10 podman inside nodes cannot reach outside the cluster New |
Issue 17940305: In rhel 10 podman inside nodes cannot reach outside the cluster
Description: Description of problem:
{code:none}
When in a rhel10 clutser we connect to a node, and we try to use podman inside the node, podman cannot reach the network outside the cluster.
If we do it in a rhel9 node there is no problem doing that.
{code}
Version-Release number of selected component (if applicable):
{code:none}
4.22 {code}
How reproducible:
{code:none}
Always{code}
Steps to Reproduce:
{code:none}
1. Enable the rhel10 stream in a 4.22 techpreview cluster
oc patch mcp worker --type=merge -p "{\"spec\":{\"osImageStream\":{\"name\":\"rhel-10\"}}}"
2. Wait until the worker pool is using rhel10
3. Debug into a worker node with "oc debug...."
4. Create a Containerfile with this content
sh-5.2# cat Containerfile
FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a1798a3b92a794a69d56eaf78c1521a1c4d2e52fd05057072780ec19ccabd45
RUN curl -LO https://pkgs.tailscale.com/stable/fedora/tailscale.repo
5. Build an image using the container file
sh-5.2# podman build --authfile /var/lib/kubelet/config.json . -t with9
The result is this error
STEP 2/2: RUN curl -LO https://pkgs.tailscale.com/stable/fedora/tailscale.repo
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0curl: (6) Could not resolve host: pkgs.tailscale.com
Error: building at STEP "RUN curl -LO https://pkgs.tailscale.com/stable/fedora/tailscale.repo": while running runtime: exit status 6{code}
Actual results:
{code:none}
The podman build command cannot reach the network outside the cluster {code}
Expected results:
{code:none}
The podman build command should be able to get the tailscale information and build the image{code}
Additional info:
{code:none}
This is the AI analysis:
Root Cause
The problem is a conflict between two FORWARD chains combined with a firewall backend change between netavark versions on RHEL 9 vs RHEL 10.
RHEL 9 (works) — netavark 1.17.2 with iptables-nft backend
There is one FORWARD chain in table ip filter with policy DROP, and netavark injects its rules directly into it:
chain FORWARD {
policy drop;
jump NETAVARK_FORWARD <-- netavark adds this to allow container traffic
<OVN rules>
}
Container traffic from 10.88.0.0/16 hits NETAVARK_FORWARD first and gets accepted. Packets are forwarded, masqueraded, and reach the internet.
RHEL 10 (fails) — netavark 1.16.0 with native nftables backend
There are two FORWARD chains at the same priority:
1. table ip filter → chain FORWARD (from OVN/kube — policy DROP):
chain FORWARD {
policy drop;
<only OVN/kube subnets accepted>
# NO netavark rules here!
}
2. table inet netavark → chain FORWARD (from netavark — policy ACCEPT):
chain FORWARD {
policy accept;
ct state invalid drop
jump NETAVARK-ISOLATION-1
}
The OVN table ip filter FORWARD chain has policy DROP and only allows traffic to/from 169.254.0.1, 172.30.0.0/16, and 10.128.0.0/14. The podman bridge subnet 10.88.0.0/16 is not in that list. Even though
netavark's own chain would accept it, the OVN/kube chain drops it first because both chains are at the same filter hook priority, and the ip family table processes before inet.
So container traffic from 10.88.0.0/16 → 10.0.0.2 (DNS) gets dropped by the OVN FORWARD chain before it can reach the internet.
Why --network host works
With --network host, containers use the host's network stack directly — no bridge, no forwarding, no FORWARD chain involved.
~
Workaround
Executing the "podman build" command with "--network host" will make the command work.{code}
Status: New
| |||
| #OCPBUGS-78100 | issue | 17 hours ago | [openshift-4.15] baremetal-machine-controller can't be built hermetically New |
Issue 17972261: [openshift-4.15] baremetal-machine-controller can't be built hermetically
Description: This is a clone of issue OCPBUGS-78099. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78098. The following is the description of the original issue: --- Description of problem:
ART can't build baremetal-machine-controller hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-76931 | issue | 3 weeks ago | GCP: Provider produced inconsistent result after apply - google_service_account ASSIGNED |
Issue 17884800: GCP: Provider produced inconsistent result after apply - google_service_account
Description: Description of problem:
{code:none}
The fix given for issue https://issues.redhat.com/browse/OCPBUGS-42014(pointing to same problem) was at service account key creation. but the actual problem lies at service account creation itself.
We suggest upgrading terraform and provider to the latest version in openshift-installer for 4.12+. The underlying issue is already fixed upstream and released in v6.45.0.
Note: we see this problem more often in lower openshift versions 4.12, 4.13 than in 4.16.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
Intermittent{code}
Steps to Reproduce:
{code:none}
1.Install the UPI openshift cluster in GCP platform
2.
3.
{code}
Actual results:
{code:none}
level=error msg=Error: Provider produced inconsistent result after apply
level=error
level=error msg=When applying changes to
level=error msg=module.master.google_service_account.master-node-sa[0], provider
level=error msg="provider[\"openshift/local/google\"]" produced an unexpected new value: Root
level=error msg=resource was present, but now absent.
level=error
level=error msg=This is a bug in the provider, which should be reported in the provider's own{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: ASSIGNED
| |||
| #OCPBUGS-59259 | issue | 5 weeks ago | Firmware upgrade failures get cleared from the BaremetalHost making it difficult to know what failed ASSIGNED |
Issue 17137858: Firmware upgrade failures get cleared from the BaremetalHost making it difficult to know what failed
Description: Description of problem:
I have the following BMH definition:
{code:none}
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
name: dell-server
namespace: hardware-inventory
annotations:
spec:
automatedCleaningMode: disabled
bmc:
disableCertificateVerification: True
address: idrac-virtualmedia://10.6.75.116/redfish/v1/Systems/System.Embedded.1
credentialsName: dell-bmc-credentials
bootMACAddress: B4:83:51:00:B4:88
online: true{code}
And I apply the following manifest to run the upgrade:
{code:java}
---
apiVersion: metal3.io/v1alpha1
kind: HostUpdatePolicy
metadata:
name: dell-server
namespace: hardware-inventory
spec:
firmwareSettings: onReboot
firmwareUpdates: onReboot
---
apiVersion: metal3.io/v1alpha1
kind: HostFirmwareComponents
metadata:
name: dell-server
namespace: hardware-inventory
spec:
updates:
- component: bmc
url: http://10.6.116.4:9999/iDRAC-with-Lifecycle-Controller_Firmware_R8V2F_LN64_7.20.30.00_A00.BIN
- component: bios
url: http://10.6.116.4:9999/BIOS_0HY8N_LN64_1.17.2.BIN {code}
Then I apply the _reboot.metal3.io=""_ annotation to get the firmware and bios upgraded. The problem I see is that this upgrade fails, this is what I see in the BMO logs:
{code:java}
{"level":"info","ts":1750857141.5618482,"logger":"controllers.BareMetalHost","msg":"using PreprovisioningImage","baremetalhost":{"name":"dell-server","namespace":"hardware-inventory"},"provisioningState":"available","Image":{"ImageURL":"http://metal3-image-customization-service.openshift-machine-api.svc.cluster.local/d235e339-76f8-4709-ad13-3517db75f539","KernelURL":"","ExtraKernelParams":"","Format":"iso"}}
{"level":"info","ts":1750857141.5847816,"logger":"provisioner.ironic","msg":"current provision state","host":"hardware-inventory~dell-server","lastError":"Firmware update failed for node c0471e4f-e9c8-4678-9893-77975de4ded2, firmware http://10.6.116.4:9999/iDRAC-with-Lifecycle-Controller_Firmware_R8V2F_LN64_7.20.30.00_A00.BIN. Error: Lifecycle Controller in use. This job will start when Lifecycle Controller is available.","current":"manageable","target":""} {code}
Now, that's okay. The issue I see is that this error will be cleared from the BMH, this is the BMH during the upgrade:
{code:java}
NAME STATE CONSUMER ONLINE ERROR AGE
dell-server preparing true 46h
dell-server preparing true preparation error 46h
dell-server preparing true preparation error 46h
dell-server available true 46h {code}
Version-Release number of selected component (if applicable):
{code:none}
OCP 4.18 {code}
How reproducible:
{code:none}
Always{code}
Steps to Reproduce:
{code:none}
Described above.{code}
Actual results:
{code:none}
Firmware upgrade fails. Error gets cleared out from the BMH. User cannot see why it failed unless watching the object during the procedure.{code}
Expected results:
{code:none}
Firmware upgrade fails. Error remains in the BMH until the user fixes whatever needs to be fixed.{code}
Additional info:
{code:none}
Slack thread: https://redhat-internal.slack.com/archives/CFP6ST0A3/p1750857275820059{code}
Status: ASSIGNED
| |||
| #OCPBUGS-77861 | issue | 5 days ago | [openshift-4.19] openshift-enterprise-console-operator can't be built hermetically POST |
Issue 17959742: [openshift-4.18] openshift-enterprise-console-operator can't be built hermetically
Description: This is a clone of issue OCPBUGS-77860. The following is the description of the original issue: --- Description of problem:
ART can't build openshift-enterprise-console-operator hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-77466 | issue | 11 days ago | [openshift-4.15] openshift-enterprise-hyperkube can't be built hermetically New |
Issue 17940923: [openshift-4.15] openshift-enterprise-hyperkube can't be built hermetically
Description: Description of problem:
ART can't build openshift-enterprise-hyperkube hermetically. Same applies for ose-installer-kube-apiserver-artifacts
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.
{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77775 | issue | 6 days ago | [openshift-4.19] ose-sriov-rdma-cni can't be built hermetically POST |
Issue 17954315: [openshift-4.19] ose-sriov-rdma-cni can't be built hermetically
Description: Description of problem:
ART can't build ose-sriov-rdma-cni hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: Verified
| |||
| #OCPBUGS-77199 | issue | 2 weeks ago | [openshift-4.13] ose-libvirt-machine-controllers can't be build hermetically New |
Issue 17925866: [openshift-4.13] ose-libvirt-machine-controllers can't be build hermetically
Description: Description of problem:
ART can't build ose-libvirt-machine-controllers hermetically.
The error is the following in the prefetch-dependencies konflux task:
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details. Please try running `go mod vendor` and committing the changes. Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-45017 | issue | 7 weeks ago | OpenShift 4.17 IPI Private cluster installation failure due to non-empty resource group ASSIGNED |
Issue 16485170: OpenShift 4.17 IPI Private cluster installation failure due to non-empty resource group
Description: Description of problem:
When creating a Private IPI Cluster using an existing Azure Vnet with preconfigured Subnets, as per
[https://docs.openshift.com/container-platform/4.17/installing/installing_azure/ipi/installing-azure-private.html]
and
[https://docs.openshift.com/container-platform/4.17/installing/installing_azure/ipi/installing-azure-vnet.html]
an error is returned:
{code:none}
FATAL failed to fetch Cluster API Manifests: failed to generate asset "Cluster API Manifests": failed to generate Azure manifests: failed to get azure ip availability: network.VirtualNetworksClient#CheckIPAddressAvailability: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="PrivateIPAddressNotInAnySubnet" Message="Private static IP address 10.0.0.100 does not belong to the range of any subnet in the virtual network /subscriptions/6e700e21-5667-435d-8f78-f421bedbe936/resourceGroups/RG-AS-OCPNP/providers/Microsoft.Network/virtualNetworks/VN-AS-OCPNP." Details=[]{code}
This error appears during the creation of the manifests, i.e.
{code:java}
$ openshift-install create manifests --dir <installation_directory> {code}
No cluster creation step was attempted yet.
For the above issue, the customer found the "fix" in the GitHub Repo[1] and it worked partially:
[1] [https://github.com/openshift/installer/pull/9144]
However, after cloning the git repo, adding the PR changes to that file; "pkg/asset/manifests/azure/cluster.go"
{code:java}
# ./openshift-install create manifests --dir installer_linux INFO Credentials loaded from file "/root/.azure/osServicePrincipal.json" INFO Consuming Install Config from target directory INFO Adding clusters... INFO Manifests created in: installer_linux/cluster-api, installer_linux/manifests and installer_linux/openshift{code}
We could see the internal LB was assigned the correct next available IP(which previously was not happening)
{code:java}
networkSpec:
apiServerLB:
backendPool:
name: ocpnp-pz6m2-internal
frontendIPs:
- name: ocpnp-pz6m2-internal-frontEnd
privateIP: 10.89.188.4
name: ocpnp-pz6m2-internal
type: Internal {code}
We used " remotes/origin/release-4.17 " branch, from [https://github.com/openshift/installer]
And then proceeded with a cluster build to test it further, but testing the cluster creation failed, this time with a new error message:
{code:java}
# ./openshift-install create cluster --dir installer_linux
INFO Consuming OpenShift Install (Manifests) from target directory
INFO Consuming Master Machines from target directory
INFO Consuming Worker Machines from target directory
INFO Consuming Common Manifests from target directory
INFO Consuming Openshift Manifests from target directory
INFO Credentials loaded from file "/root/.azure/osServicePrincipal.json"
FATAL failed to fetch Cluster Infrastructure Variables: failed to fetch dependency of "Cluster Infrastructure Variables": failed to generate asset "Platform Provisioning Check": platform.azure.resourceGroupName: Invalid value: "RG-AS-OCPNP": resource group must be empty but it has 7 resources like /subscriptions/6e700e21-5667-435d-8f78-f421bedbe936/resourceGroups/RG-AS-OCPNP/providers/Microsoft.Network/virtualNetworks/VN-AS-OCPNP, /subscriptions/6e700e21-5667-435d-8f78-f421bedbe936/resourceGroups/RG-AS-OCPNP/providers/Microsoft.Network/networkSecurityGroups/NSG-AS-OCPNP ...{code}
For some reason, when using an Existing Vnet / resource group etc, it's trying to validate the resource group is empty, which will never be the case.
This above error appeared during v4.17 cluster installation and also v4.16.x (tried .0, .9 and .21 installer versions) wherein creating the manifests worked fine, but the error about non-empty resource group persists.
{code:java}
root@ocpjump:~# ./openshift-install version
./openshift-install 4.16.9
built from commit deb993e72a920fb1c68a578d1b4c598071fefea4
release image quay.io/openshift-release-dev/ocp-release@sha256:115bba6836b9feffb81ad9101791619edd5f19d333580b7f62bd6721eeda82d2
release architecture amd64
root@ocpjump:~# ./openshift-install create manifests --dir installer_linux
INFO Credentials loaded from file "/root/.azure/osServicePrincipal.json"
INFO Consuming Install Config from target directory
INFO Manifests created in: installer_linux/manifests and installer_linux/openshift
root@ocpjump:~# ./openshift-install create cluster --dir installer_linux
INFO Consuming Common Manifests from target directory
INFO Consuming Master Machines from target directory
INFO Consuming Worker Machines from target directory
INFO Consuming OpenShift Install (Manifests) from target directory
INFO Consuming Openshift Manifests from target directory
INFO Credentials loaded from file "/root/.azure/osServicePrincipal.json"
FATAL failed to fetch Cluster Infrastructure Variables: failed to fetch dependency of "Cluster Infrastructure Variables": failed to generate asset "Platform Provisioning Check": platform.azure.resourceGroupName: Invalid value: "RG-AS-OCPNP": resource group must be empty but it has 7 resources like /subscriptions/6e700e21-5667-435d-8f78-f421bedbe936/resourceGroups/RG-AS-OCPNP/providers/Microsoft.Network/virtualNetworks/VN-AS-OCPNP, /subscriptions/6e700e21-5667-435d-8f78-f421bedbe936/resourceGroups/RG-AS-OCPNP/providers/Microsoft.Network/networkSecurityGroups/NSG-AS-OCPNP ...{code}
Perhaps existing vnet and subnets in a private install isn't a very widely used deployment mechanism in Azure given this bug seems present for a while now. This seems peculiar as this deployment type exhibits the best security and customisation with on-premise cloud interconnects.
Version-Release number of selected component (if applicable):
{code:none}
v4.17.Z {code}
Actual results:
{code:none}
The private IPI cluster installation fails for OpenShift v4.17 on Azure{code}
Expected results:
{code:none}
Private IPI OpenShift v4.17 cluster should be installed successfully on Azure{code}
Status: ASSIGNED
| |||
| #OCPBUGS-78103 | issue | 17 hours ago | [openshift-4.12] baremetal-machine-controller can't be built hermetically New |
Issue 17972334: [openshift-4.12] baremetal-machine-controller can't be built hermetically
Description: This is a clone of issue OCPBUGS-78102. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78101. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78100. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78099. The following is the description of the original issue: --- This is a clone of issue OCPBUGS-78098. The following is the description of the original issue: --- Description of problem:
ART can't build baremetal-machine-controller hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77469 | issue | 11 days ago | [openshift-4.12] openshift-enterprise-hyperkube can't be built hermetically New |
Issue 17940938: [openshift-4.12] openshift-enterprise-hyperkube can't be built hermetically
Description: Description of problem:
ART can't build openshift-enterprise-hyperkube hermetically. Same applies for ose-installer-kube-apiserver-artifacts
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.
{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77023 | issue | 2 weeks ago | Inspection error on Dell XR8620t New |
Issue 17899514: Inspection error on Dell XR8620t
Description: Description of problem:
{code:none}
After deprovisioning cluster and deleting BMH for my xr8620t, I recreated the BMH and hit an inspection failure.
errorMessage: 'Failed to inspect hardware. Reason: unable to start inspection: HTTP
POST https://10.x.x.x/redfish/v1/Systems/System.Embedded.1/VirtualMedia/1/Actions/VirtualMedia.InsertMedia
returned code 500. Base.1.12.GeneralError: A general error has occurred. See ExtendedInfo
for more information Extended information: [{''Message'': ''Virtual Media is detached
or Virtual Media devices are already in use.'', ''MessageArgs'': [], ''MessageArgs@odata.count'':
0, ''MessageId'': ''IDRAC.2.9.VRM0021'', ''RelatedProperties'': [], ''RelatedProperties@odata.count'':
0, ''Resolution'': ''Change the Virtual Media Attach Mode to Attach or Auto-Attach.Stop
existing Virtual Media sessions and retry the operation.'', ''Severity'': ''Informational''}]'
errorType: inspection error
I've confirmed that there is no virtual media attached. Setting online=true has no effect, and manually powering on the node from idrac ends up with the node quickly being powered off again.{code}
Version-Release number of selected component (if applicable):
{code:none}
OCP: 4.20.0-0.nightly-2025-07-20-021531
ACM: 2.14.0-131
MCE: 2.9.0-102{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: New
| |||
| #OCPBUGS-77903 | issue | 4 days ago | [openshift-4.18] ose-aws-cluster-api-controllers can't be built hermetically POST |
Issue 17962668: [openshift-4.18] ose-aws-cluster-api-controllers can't be built hermetically
Description: This is a clone of issue OCPBUGS-77902. The following is the description of the original issue: --- Description of problem:
ART can't build ose-aws-cluster-api-controllers
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.
{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-78169 | issue | 9 hours ago | [openshift-4.17] ose-machine-api-provider-openstack can't be built hermetically POST |
Issue 17974764: [openshift-4.17] ose-machine-api-provider-openstack can't be built hermetically
Description: This is a clone of issue OCPBUGS-78168. The following is the description of the original issue: --- Description of problem:
ART can't build ose-machine-api-provider-openstack hermetically.
the error is the following
{code:none}
Error: PackageRejected: The content of the vendor directory is not consistent with go.mod. Please check the logs for more details.
Please try running `go mod vendor` and committing the changes.
Note that you may need to `git add --force` ignored files in the vendor/ dir.{code}
Version-Release number of selected component (if applicable):
{code:none}
{code}
How reproducible:
{code:none}
{code}
Steps to Reproduce:
{code:none}
1.
2.
3.
{code}
Actual results:
{code:none}
{code}
Expected results:
{code:none}
{code}
Additional info:
{code:none}
{code}
Status: POST
| |||
| #OCPBUGS-60539 | issue | 5 weeks ago | node looses network connection when it reboots. If the system is rebooted into emergency mode (single user) and boot is resumed, the systems boots normally New |
Aug 01 04:39:59 openshift-master-0.utin-ocp-web.caas.apps.mil kubenswrapper[5286]: E0801 04:39:59.414124 5286 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/openshift-storage.cephfs.csi.ceph.com^0001-0011-openshift-storage-0000000000000001-fa12c8a7-1f30-11ed-b784-0a58ac1c0a0b podName:e4b533cc-8360-4861-86ae-8fe736c76db4 nodeName:}" failed. No retries permitted until 2025-08-01 04:42:01.414103867 +0000 UTC m=+52734.450871152 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "pvc-e5ce62d9-b588-4072-8ea9-d71a6db7aef6" (UniqueName: "kubernetes.io/csi/openshift-storage.cephfs.csi.ceph.com^0001-0011-openshift-storage-0000000000000001-fa12c8a7-1f30-11ed-b784-0a58ac1c0a0b") pod "e4b533cc-8360-4861-86ae-8fe736c76db4" (UID: "e4b533cc-8360-4861-86ae-8fe736c76db4") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name openshift-storage.cephfs.csi.ceph.com not found in the list of registered CSI drivers
>>>9 missing seconds
| |||
Found in 0.00% of runs (0.00% of failures) across 240107 total runs and 19723 jobs (30.02% failed) in 122ms - clear search | chart view - source code located on github