#OCPBUGS-33702 | issue | 38 hours ago | dev-scripts fails bootstrapping OCP 4.16 and greater with MIRROR_IMAGES=true AND INSTALLER_PROXY=true New |
Issue 16005149: dev-scripts fails bootstrapping OCP 4.16 and greater with MIRROR_IMAGES=true AND INSTALLER_PROXY=true Description: Description of problem: {code:none} Trying to execute https://github.com/openshift-metal3/dev-scripts to deploy an OCP 4.16 or 4.17 cluster (with the same configuration OCP 4.14 and 4.15 are instead working) with: MIRROR_IMAGES=true INSTALLER_PROXY=true the bootstrap process fails with: level=debug msg= baremetalhost resource not yet available, will retry level=debug msg= baremetalhost resource not yet available, will retry level=info msg= baremetalhost: ostest-master-0: uninitialized level=info msg= baremetalhost: ostest-master-0: registering level=info msg= baremetalhost: ostest-master-1: uninitialized level=info msg= baremetalhost: ostest-master-1: registering level=info msg= baremetalhost: ostest-master-2: uninitialized level=info msg= baremetalhost: ostest-master-2: registering level=info msg= baremetalhost: ostest-master-1: inspecting level=info msg= baremetalhost: ostest-master-2: inspecting level=info msg= baremetalhost: ostest-master-0: inspecting E0514 12:16:51.985417 89709 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?allowWatchBookmarks=true&resourceVersion=5466&timeoutSeconds=547&watch=true": Service Unavailable W0514 12:16:52.979254 89709 reflector.go:539] k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?resourceVersion=5466": Service Unavailable E0514 12:16:52.979293 89709 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?resourceVersion=5466": Service Unavailable E0514 12:37:01.927140 89709 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?allowWatchBookmarks=true&resourceVersion=7800&timeoutSeconds=383&watch=true": Service Unavailable W0514 12:37:03.173425 89709 reflector.go:539] k8s.io/client-go/tools/watch/informerwatcher.go:146: failed to list *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?resourceVersion=7800": Service Unavailable E0514 12:37:03.173473 89709 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?resourceVersion=7800": Service Unavailable level=debug msg=Fetching Bootstrap SSH Key Pair... level=debug msg=Loading Bootstrap SSH Key Pair... it looks like up to a certain point https://api.ostest.test.metalkube.org:6443 was reachable but then for some reason it started failing because its not using the proxy or is and it shouldn't be (???) The 3 master nodes are reported as: [root@ipi-ci-op-0qigcrln-b54ee-1790684582253694976 home]# oc get baremetalhosts -A NAMESPACE NAME STATE CONSUMER ONLINE ERROR AGE openshift-machine-api ostest-master-0 inspecting ostest-bbhxb-master-0 true inspection error 24m openshift-machine-api ostest-master-1 inspecting ostest-bbhxb-master-1 true inspection error 24m openshift-machine-api ostest-master-2 inspecting ostest-bbhxb-master-2 true inspection error 24m With something like: status: errorCount: 5 errorMessage: 'Failed to inspect hardware. Reason: unable to start inspection: Validation of image href http://0.0.0.0:8084/34427934-f1a6-48d6-9666-66872eec9ba2 failed, reason: Got HTTP code 503 instead of 200 in response to HEAD request.' errorType: inspection error on their status{code} Version-Release number of selected component (if applicable): {code:none} 4.16, 4.17{code} How reproducible: {code:none} 100%{code} Steps to Reproduce: {code:none} 1. Try to create an OCP 4.16 cluster with dev-scrips with IP_STACK=v4, MIRROR_IMAGES=true and INSTALLER_PROXY=true 2. 3. {code} Actual results: {code:none} level=info msg= baremetalhost: ostest-master-0: inspecting E0514 12:16:51.985417 89709 reflector.go:147] k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: Get "https://api.ostest.test.metalkube.org:6443/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts?allowWatchBookmarks=true&resourceVersion=5466&timeoutSeconds=547&watch=true": Service Unavailable{code} Expected results: {code:none} Successful deployment{code} Additional info: {code:none} I'm using IP_STACK=v4, MIRROR_IMAGES=true and INSTALLER_PROXY=true with the same configuration (MIRROR_IMAGES=true and INSTALLER_PROXY=true) OCP 4.14 and OCP 4.15 are working. When removing INSTALLER_PROXY=true, OCP 4.16 is also working. I'm going to attach bootstrap gather logs{code} Status: New | |||
openshift-cluster-baremetal-operator-416-openshift-baremetal-operator-353-openshift-installer-8391-nightly-4.16-e2e-metal-ipi-ovn-ipv6 (all) - 2 runs, 50% failed, 100% of failures match = 50% impact | |||
#1788964451169341440 | junit | 6 days ago | |
[must-gather ] OUT 2024-05-10T18:56:34.664382317Z pod for plug-in image registry.redhat.io/openshift4/ose-must-gather:latest created [must-gather-vbg6w] OUT 2024-05-10T19:06:24.696703513Z gather did not start: Get "https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-must-gather-mkg45/pods/must-gather-vbg6w": Service Unavailable Delete "https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-must-gather-mkg45": Service Unavailable Delete "https://api.ostest.test.metalkube.org:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/must-gather-6tmwz": Service Unavailable | |||
#1788964451169341440 | junit | 6 days ago | |
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: error getting cluster version: Get "https://api.ostest.test.metalkube.org:6443/apis/config.openshift.io/v1/clusterversions/version": Service Unavailable ClusterID: | |||
#1788964451169341440 | junit | 6 days ago | |
ClusterVersion: Installing "" for <unknown>: <unknown> error getting cluster operators: Get "https://api.ostest.test.metalkube.org:6443/apis/config.openshift.io/v1/clusteroperators": Service Unavailable ClusterOperators: | |||
#1788964451169341440 | junit | 6 days ago | |
error: gather did not start for pod must-gather-vbg6w: Get "https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-must-gather-mkg45/pods/must-gather-vbg6w": Service Unavailable {"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:84","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2024-05-10T19:06:24Z"} |
Found in 0.37% of runs (0.85% of failures) across 273 total runs and 54 jobs (43.22% failed) in 800ms - clear search | chart view - source code located on github