Job:
#OCPBUGS-33702issue38 hours agodev-scripts fails bootstrapping OCP 4.16 and greater with MIRROR_IMAGES=true AND INSTALLER_PROXY=true New
Issue 16005149: dev-scripts fails bootstrapping OCP 4.16 and greater with MIRROR_IMAGES=true AND INSTALLER_PROXY=true
Description: Description of problem:
 {code:none}
     Trying to execute https://github.com/openshift-metal3/dev-scripts to deploy an OCP 4.16 or 4.17 cluster (with the same configuration OCP 4.14 and 4.15 are instead working) with:
  MIRROR_IMAGES=true
  INSTALLER_PROXY=true
 
 the bootstrap process fails with:
 
  level=debug msg=    baremetalhost resource not yet available, will retry
 level=debug msg=    baremetalhost resource not yet available, will retry
 level=info msg=  baremetalhost: ostest-master-0: uninitialized
 level=info msg=  baremetalhost: ostest-master-0: registering
 level=info msg=  baremetalhost: ostest-master-1: uninitialized
 level=info msg=  baremetalhost: ostest-master-1: registering
 level=info msg=  baremetalhost: ostest-master-2: uninitialized
 level=info msg=  baremetalhost: ostest-master-2: registering
 level=info msg=  baremetalhost: ostest-master-1: inspecting
 level=info msg=  baremetalhost: ostest-master-2: inspecting
 level=info msg=  baremetalhost: ostest-master-0: inspecting
 E0514 12:16:51.985417   89709 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?allowWatchBookmarks=true&resourceVersion=5466&timeoutSeconds=547&watch=true": Service Unavailable
 W0514 12:16:52.979254   89709 reflector.go:539] k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?resourceVersion=5466": Service Unavailable
 E0514 12:16:52.979293   89709 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?resourceVersion=5466": Service Unavailable
 E0514 12:37:01.927140   89709 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?allowWatchBookmarks=true&resourceVersion=7800&timeoutSeconds=383&watch=true": Service Unavailable
 W0514 12:37:03.173425   89709 reflector.go:539] k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?resourceVersion=7800": Service Unavailable
 E0514 12:37:03.173473   89709 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?resourceVersion=7800": Service Unavailable
 level=debug msg=Fetching Bootstrap SSH Key Pair...
 level=debug msg=Loading Bootstrap SSH Key Pair...
 
 it looks like up to a certain point https://api.ostest.test.metalkube.org:6443 was reachable but then for some reason it started failing because its not using the proxy or is and it shouldn't be (???)
 
 The 3 master nodes are reported as:
 [root@ipi-ci-op-0qigcrln-b54ee-1790684582253694976 home]# oc get baremetalhosts -A
 NAMESPACE               NAME              STATE        CONSUMER                ONLINE   ERROR              AGE
 openshift-machine-api   ostest-master-0   inspecting   ostest-bbhxb-master-0   true     inspection error   24m
 openshift-machine-api   ostest-master-1   inspecting   ostest-bbhxb-master-1   true     inspection error   24m
 openshift-machine-api   ostest-master-2   inspecting   ostest-bbhxb-master-2   true     inspection error   24m
 
 With something like:
 
  status:
   errorCount: 5
   errorMessage: 'Failed to inspect hardware. Reason: unable to start inspection: Validation
     of image href http://0.0.0.0:8084/34427934-f1a6-48d6-9666-66872eec9ba2 failed,
     reason: Got HTTP code 503 instead of 200 in response to HEAD request.'
   errorType: inspection error
 
 on their status{code}
 Version-Release number of selected component (if applicable):
 {code:none}
     4.16, 4.17{code}
 How reproducible:
 {code:none}
     100%{code}
 Steps to Reproduce:
 {code:none}
     1. Try to create an OCP 4.16 cluster with dev-scrips with IP_STACK=v4, MIRROR_IMAGES=true and INSTALLER_PROXY=true
     2.
     3.
     {code}
 Actual results:
 {code:none}
     level=info msg=  baremetalhost: ostest-master-0: inspecting
 E0514 12:16:51.985417   89709 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?allowWatchBookmarks=true&resourceVersion=5466&timeoutSeconds=547&watch=true": Service Unavailable{code}
 Expected results:
 {code:none}
     Successful deployment{code}
 Additional info:
 {code:none}
 I'm using IP_STACK=v4, MIRROR_IMAGES=true and INSTALLER_PROXY=true
 with the same configuration (MIRROR_IMAGES=true and INSTALLER_PROXY=true) OCP 4.14 and OCP 4.15 are working.
 
 When removing INSTALLER_PROXY=true, OCP 4.16 is also working.
 
 I'm going to attach bootstrap gather logs{code}
Status: New
openshift-cluster-baremetal-operator-416-openshift-baremetal-operator-353-openshift-installer-8391-nightly-4.16-e2e-metal-ipi-ovn-ipv6 (all) - 2 runs, 50% failed, 100% of failures match = 50% impact
#1788964451169341440junit6 days ago
[must-gather      ] OUT 2024-05-10T18:56:34.664382317Z pod for plug-in image registry.redhat.io/openshift4/ose-must-gather:latest created
[must-gather-vbg6w] OUT 2024-05-10T19:06:24.696703513Z gather did not start: Get "https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-must-gather-mkg45/pods/must-gather-vbg6w": Service Unavailable
Delete "https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-must-gather-mkg45": Service Unavailable
Delete "https://api.ostest.test.metalkube.org:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/must-gather-6tmwz": Service Unavailable
#1788964451169341440junit6 days ago
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
error getting cluster version: Get "https://api.ostest.test.metalkube.org:6443/apis/config.openshift.io/v1/clusterversions/version": Service Unavailable
ClusterID:
#1788964451169341440junit6 days ago
ClusterVersion: Installing "" for <unknown>: <unknown>
error getting cluster operators: Get "https://api.ostest.test.metalkube.org:6443/apis/config.openshift.io/v1/clusteroperators": Service Unavailable
ClusterOperators:
#1788964451169341440junit6 days ago
error: gather did not start for pod must-gather-vbg6w: Get "https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-must-gather-mkg45/pods/must-gather-vbg6w": Service Unavailable
{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:84","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2024-05-10T19:06:24Z"}

Found in 0.37% of runs (0.85% of failures) across 273 total runs and 54 jobs (43.22% failed) in 800ms - clear search | chart view - source code located on github