=== RUN TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests/Validate_DSCI_webhook_denies_invalid_PEM_on_update testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests/Validate_DSCI_webhook_denies_invalid_PEM_on_update (300.01s) === RUN TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests 2026-07-05T17:26:12Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.23"}} test_context_test.go:1675: Timed out after 300.009s. Webhook should deny update with invalid PEM in customCABundle The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1664 with: Webhook should deny update with invalid PEM in customCABundle Expected an error to have occurred. Got: <nil>: nil --- FAIL: TestOdhOperator/DSCInitialization_and_DataScienceCluster_validation_E2E_Tests (300.08s) === RUN TestOdhOperator/components/group_1/ogx/Validate_update_operand_resources/deployment_ogx-k8s-operator-controller-manager testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/ogx/Validate_update_operand_resources/deployment_ogx-k8s-operator-controller-manager (5.02s) 2026/07/05 17:31:38 TEST FAILURE DETECTED: Validate update operand resources 2026/07/05 17:31:38 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:Validate update operand resources) === 2026/07/05 17:31:39 === CLUSTER STATE === 2026/07/05 17:31:39 Node: ip-10-0-131-75.ec2.internal 2026/07/05 17:31:39 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:31:39 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:31:39 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:31:39 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:31:39 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/05 17:31:39 Node: ip-10-0-137-37.ec2.internal 2026/07/05 17:31:39 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:31:39 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:31:39 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:31:39 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:31:39 Resources: 7500m CPU, 31236220Ki memory allocatable (of 8000m CPU, 32387196Ki memory capacity) 2026/07/05 17:31:39 Node: ip-10-0-143-231.ec2.internal 2026/07/05 17:31:39 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:31:39 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:31:39 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:31:39 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:31:39 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/05 17:31:39 === DEPLOYMENTS === 2026/07/05 17:31:39 Failed to collect deployment data: deployments not ready: redhat-ods-applications/ai-gateway-operator (0/1), redhat-ods-applications/kserve-controller-manager (0/2), redhat-ods-applications/maas-controller (0/1), redhat-ods-applications/model-registry-operator-controller-manager (0/1), redhat-ods-applications/odh-model-controller (0/1), redhat-ods-applications/ogx-k8s-operator-controller-manager (0/1) 2026/07/05 17:31:39 === PODS === 2026/07/05 17:31:39 Failed to collect pod data: redhat-ods-applications/ai-gateway-operator-f68c76b74-dd7cn: container manager not ready; redhat-ods-applications/kserve-controller-manager-588b4b787c-8lkpf: phase=Pending; redhat-ods-applications/kserve-controller-manager-588b4b787c-w4zf4: phase=Pending; redhat-ods-applications/maas-controller-5b858f66fc-xrp5h: phase=Pending; redhat-ods-applications/model-registry-operator-controller-manager-b67847846-jjt62: phase=Pending; redhat-ods-applications/odh-model-controller-566d8764f4-gmggg: phase=Pending; redhat-ods-applications/ogx-k8s-operator-controller-manager-94b957b46-57fdc: phase=Pending; redhat-ods-applications/selector-migration-sgh2h: phase=Failed 2026/07/05 17:31:39 === OPERATOR STATUS === 2026/07/05 17:31:39 Operator deployment: 3/3 ready 2026/07/05 17:31:39 === DSCI STATUS === 2026/07/05 17:31:39 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/05 17:31:39 === DSC STATUS === 2026/07/05 17:31:39 Failed to collect DSC data: default-dsc unhealthy conditions: Ready=False; ComponentsReady=False; KserveReady=False; ModelControllerReady=False; ModelRegistryReady=False; ModelsAsServiceReady=False; OGXReady=False 2026/07/05 17:31:39 === RECENT EVENTS === 2026/07/05 17:31:39 17:31:38 INFO Pod/kserve-controller-manager-588b4b787c-w4zf4: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-588b4b787c-w4zf4 to ip-10-0-131-75.ec2.internal 2026/07/05 17:31:39 17:31:38 INFO Pod/kserve-controller-manager-588b4b787c-8lkpf: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-588b4b787c-8lkpf to ip-10-0-143-231.ec2.internal 2026/07/05 17:31:39 17:31:38 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:31:39 17:31:38 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:31:39 17:31:38 INFO Pod/maas-controller-5b858f66fc-xrp5h: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-5b858f66fc-xrp5h to ip-10-0-143-231.ec2.internal 2026/07/05 17:31:39 17:31:38 INFO Pod/odh-model-controller-566d8764f4-gmggg: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-566d8764f4-gmggg to ip-10-0-143-231.ec2.internal 2026/07/05 17:31:39 17:31:38 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-588b4b787c from 0 to 2 2026/07/05 17:31:39 17:31:38 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-bc44d7c7b from 0 to 1 2026/07/05 17:31:39 17:31:38 INFO ReplicaSet/kserve-controller-manager-588b4b787c: SuccessfulCreate - Created pod: kserve-controller-manager-588b4b787c-w4zf4 2026/07/05 17:31:39 17:31:38 INFO ReplicaSet/kserve-controller-manager-588b4b787c: SuccessfulCreate - Created pod: kserve-controller-manager-588b4b787c-8lkpf 2026/07/05 17:31:39 17:31:38 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-566d8764f4 from 0 to 1 2026/07/05 17:31:39 17:31:38 INFO ReplicaSet/odh-model-controller-566d8764f4: SuccessfulCreate - Created pod: odh-model-controller-566d8764f4-gmggg 2026/07/05 17:31:39 17:31:38 WARN Pod/odh-model-controller-566d8764f4-gmggg: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/05 17:31:39 17:31:38 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-5b858f66fc from 0 to 1 2026/07/05 17:31:39 17:31:38 INFO ReplicaSet/maas-controller-5b858f66fc: SuccessfulCreate - Created pod: maas-controller-5b858f66fc-xrp5h 2026/07/05 17:31:39 17:31:37 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Pulling - Pulling image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:e1bd98206aee94e76215e2a9ea4987e9c6f4597bdba1358921496b275e91abc0" 2026/07/05 17:31:39 17:31:37 WARN Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 2026/07/05 17:31:39 17:31:36 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: Scheduled - Successfully assigned redhat-ods-applications/ogx-k8s-operator-controller-manager-94b957b46-57fdc to ip-10-0-143-231.ec2.internal 2026/07/05 17:31:39 17:31:36 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-b67847846-jjt62 to ip-10-0-143-231.ec2.internal 2026/07/05 17:31:39 17:31:36 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:31:39 17:31:36 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-b67847846 from 0 to 1 2026/07/05 17:31:39 17:31:36 INFO ReplicaSet/model-registry-operator-controller-manager-b67847846: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-b67847846-jjt62 2026/07/05 17:31:39 17:31:36 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: AddedInterface - Add eth0 [10.133.0.94/23] from ovn-kubernetes 2026/07/05 17:31:39 17:31:36 INFO ReplicaSet/ogx-k8s-operator-controller-manager-94b957b46: SuccessfulCreate - Created pod: ogx-k8s-operator-controller-manager-94b957b46-57fdc 2026/07/05 17:31:39 17:31:36 INFO Deployment/ogx-k8s-operator-controller-manager: ScalingReplicaSet - Scaled up replica set ogx-k8s-operator-controller-manager-94b957b46 from 0 to 1 2026/07/05 17:31:39 17:31:34 INFO Lease/ai-gateway-lock: LeaderElection - ai-gateway-operator-f68c76b74-dd7cn_0b026c3b-2f14-4ee1-bdd0-1f325834b9cc became leader 2026/07/05 17:31:39 17:31:34 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Started - Started container manager 2026/07/05 17:31:39 17:31:34 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Created - Created container: manager 2026/07/05 17:31:39 17:31:34 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Pulled - Successfully pulled image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" in 349ms (349ms including waiting). Image size: 205008753 bytes. 2026/07/05 17:31:39 17:31:33 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Pulling - Pulling image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" 2026/07/05 17:31:39 17:31:33 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Started - Started container copy-manifests 2026/07/05 17:31:39 17:31:33 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Created - Created container: copy-manifests 2026/07/05 17:31:39 17:31:32 INFO Pod/notebook-controller-deployment-6b68db7cbd-5wjj4: Killing - Stopping container manager 2026/07/05 17:31:39 17:31:32 INFO Pod/kubeflow-training-operator-7c59bdd544-cdgsb: Killing - Stopping container training-operator 2026/07/05 17:31:39 17:31:32 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-8ltwl: Killing - Stopping container manager 2026/07/05 17:31:39 17:31:32 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Pulled - Successfully pulled image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" in 3.678s (3.678s including waiting). Image size: 205008753 bytes. 2026/07/05 17:31:39 17:31:32 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-hwdjl: Killing - Stopping container manager 2026/07/05 17:31:39 17:31:32 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-rlpwh: Killing - Stopping container manager 2026/07/05 17:31:39 17:31:32 INFO Pod/model-registry-operator-controller-manager-b67847846-kljq2: Killing - Stopping container manager 2026/07/05 17:31:39 17:31:32 INFO Pod/maas-api-955664fbd-fv5z6: Killing - Stopping container maas-api 2026/07/05 17:31:39 17:31:32 INFO Pod/maas-controller-5b858f66fc-mxjzp: Killing - Stopping container manager 2026/07/05 17:31:39 17:31:31 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:31:39 17:31:31 INFO Pod/kserve-controller-manager-588b4b787c-xpwqz: Killing - Stopping container manager 2026/07/05 17:31:39 17:31:31 INFO Pod/model-serving-api-5c897df776-gvttw: Killing - Stopping container server 2026/07/05 17:31:39 17:31:31 INFO Pod/odh-model-controller-566d8764f4-zxdr4: Killing - Stopping container manager 2026/07/05 17:31:39 17:31:31 INFO Pod/llmisvc-controller-manager-bc44d7c7b-zffh2: Killing - Stopping container manager 2026/07/05 17:31:39 17:31:30 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:31:39 17:31:30 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:31:39 17:31:30 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:31:39 17:31:29 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hsl6h: Killing - Stopping container manager 2026/07/05 17:31:39 17:31:29 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Pulling - Pulling image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" 2026/07/05 17:31:39 17:31:29 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: AddedInterface - Add eth0 [10.133.0.93/23] from ovn-kubernetes 2026/07/05 17:31:39 17:31:28 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Scheduled - Successfully assigned redhat-ods-applications/ai-gateway-operator-f68c76b74-dd7cn to ip-10-0-143-231.ec2.internal 2026/07/05 17:31:39 17:31:28 INFO Pod/selector-migration-sgh2h: Started - Started container migrate 2026/07/05 17:31:39 17:31:28 INFO Pod/selector-migration-sgh2h: Created - Created container: migrate 2026/07/05 17:31:39 17:31:28 INFO Pod/selector-migration-sgh2h: Pulled - Container image "registry.redhat.io/openshift4/ose-cli@sha256:bc35a9fc663baf0d6493cc57e89e77a240a36c43cf38fb78d8e61d3b87cf5cc5" already present on machine 2026/07/05 17:31:39 17:31:28 INFO Pod/selector-migration-sgh2h: AddedInterface - Add eth0 [10.133.0.92/23] from ovn-kubernetes 2026/07/05 17:31:39 17:31:28 INFO Deployment/ai-gateway-operator: ScalingReplicaSet - Scaled up replica set ai-gateway-operator-f68c76b74 from 0 to 1 2026/07/05 17:31:39 17:31:28 INFO ReplicaSet/ai-gateway-operator-f68c76b74: SuccessfulCreate - Created pod: ai-gateway-operator-f68c76b74-dd7cn 2026/07/05 17:31:39 17:31:27 INFO Pod/selector-migration-sgh2h: Scheduled - Successfully assigned redhat-ods-applications/selector-migration-sgh2h to ip-10-0-143-231.ec2.internal 2026/07/05 17:31:39 17:31:27 INFO Job/selector-migration: SuccessfulCreate - Created pod: selector-migration-sgh2h 2026/07/05 17:31:39 17:30:04 INFO CronJob/maas-api-key-cleanup: SuccessfulDelete - Deleted job maas-api-key-cleanup-29721165 2026/07/05 17:31:39 17:30:04 INFO Job/maas-api-key-cleanup-29721210: Completed - Job completed 2026/07/05 17:31:39 17:30:04 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29721210, condition: Complete 2026/07/05 17:31:39 17:30:01 INFO Pod/maas-api-key-cleanup-29721210-r94rz: Started - Started container cleanup 2026/07/05 17:31:39 17:30:01 INFO Pod/maas-api-key-cleanup-29721210-r94rz: Created - Created container: cleanup 2026/07/05 17:31:39 17:30:00 INFO Pod/maas-api-key-cleanup-29721210-r94rz: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29721210-r94rz to ip-10-0-143-231.ec2.internal 2026/07/05 17:31:39 17:30:00 INFO Job/maas-api-key-cleanup-29721210: SuccessfulCreate - Created pod: maas-api-key-cleanup-29721210-r94rz 2026/07/05 17:31:39 17:30:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29721210 2026/07/05 17:31:39 17:30:00 INFO Pod/maas-api-key-cleanup-29721210-r94rz: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/05 17:31:39 17:30:00 INFO Pod/maas-api-key-cleanup-29721210-r94rz: AddedInterface - Add eth0 [10.133.0.91/23] from ovn-kubernetes 2026/07/05 17:31:39 === RESOURCE QUOTAS === 2026/07/05 17:31:39 No resource quotas found in monitored namespaces 2026/07/05 17:31:39 Classification for Validate update operand resources: infrastructure/storage (code=1006, confidence=medium) 2026/07/05 17:31:39 Evidence: event Pod/odh-model-controller-566d8764f4-gmggg: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/odh-model-controller-566d8764f4-gmggg: FailedMount - MountVolume.SetUp failed for volume \"cert\" : secret \"odh-model-controller-webhook-cert\" not found"],"confidence":"medium"} 2026/07/05 17:31:39 Diagnostics complete for failed test: Validate update operand resources === RUN TestOdhOperator/components/group_1/ogx/Validate_update_operand_resources testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/ogx/Validate_update_operand_resources (5.54s) === RUN TestOdhOperator/components/group_1/ogx === PAUSE TestOdhOperator/components/group_1/ogx === CONT TestOdhOperator/components/group_1/ogx resource_applier_test.go:252: Failed after 5.008s. The function passed to Consistently failed at /workspace/tests/e2e/resource_applier_test.go:121 with: Expected resource 'redhat-ods-applications/ogx-k8s-operator-controller-manager' of kind 'Deployment' to match condition {.spec.replicas == 2 []}. Expected <string>: map[apiVersion:apps/v1 kind:Deployment metadata:map[annotations:map[deployment.kubernetes.io/revision:1 internal.config.kubernetes.io/previousKinds:Deployment internal.config.kubernetes.io/previousNames:ogx-k8s-operator-controller-manager internal.config.kubernetes.io/previousNamespaces:redhat-ods-applications platform.opendatahub.io/instance.generation:1 platform.opendatahub.io/instance.name:default-ogx platform.opendatahub.io/instance.uid:8061b776-c124-4374-9ad5-d9382d2f0c63 platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-05T17:31:36Z generation:1 labels:map[app.kubernetes.io/name:ogx-k8s-operator app.kubernetes.io/part-of:ogx app.opendatahub.io/ogx:true control-plane:controller-manager platform.opendatahub.io/part-of:ogx] managedFields:[map[apiVersion:apps/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:internal.config.kubernetes.io/previousKinds:map[] f:internal.config.kubernetes.io/previousNames:map[] f:internal.config.kubernetes.io/previousNamespaces:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:app.kubernetes.io/name:map[] f:app.kubernetes.io/part-of:map[] f:app.opendatahub.io/ogx:map[] f:control-plane:map[] f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"8061b776-c124-4374-9ad5-d9382d2f0c63"}:map[]]] f:spec:map[f:replicas:map[] f:selector:map[] f:template:map[f:metadata:map[f:annotations:map[f:kubectl.kubernetes.io/default-container:map[]] f:labels:map[f:app.kubernetes.io/name:map[] f:app.kubernetes.io/part-of:map[] f:app.opendatahub.io/ogx:map[] f:control-plane:map[]]] f:spec:map[f:containers:map[k:{"name":"manager"}:map[.:map[] f:args:map[] f:command:map[] f:env:map[k:{"name":"GOMEMLIMIT"}:map[.:map[] f:name:map[] f:value:map[]] k:{"name":"OPERATOR_VERSION"}:map[.:map[] f:name:map[] f:value:map[]] k:{"name":"RELATED_IMAGE_RH_DISTRIBUTION"}:map[.:map[] f:name:map[] f:value:map[]]] f:image:map[] f:imagePullPolicy:map[] f:livenessProbe:map[f:httpGet:map[f:path:map[] f:port:map[]] f:initialDelaySeconds:map[] f:periodSeconds:map[]] f:name:map[] f:ports:map[k:{"containerPort":9443,"protocol":"TCP"}:map[.:map[] f:containerPort:map[] f:name:map[] f:protocol:map[]]] f:readinessProbe:map[f:httpGet:map[f:path:map[] f:port:map[]] f:initialDelaySeconds:map[] f:periodSeconds:map[]] f:resources:map[f:limits:map[f:cpu:map[] f:memory:map[]] f:requests:map[f:cpu:map[] f:memory:map[]]] f:securityContext:map[f:allowPrivilegeEscalation:map[] f:capabilities:map[f:drop:map[]]] f:volumeMounts:map[k:{"mountPath":"/tmp/k8s-webhook-server/serving-certs"}:map[.:map[] f:mountPath:map[] f:name:map[] f:readOnly:map[]]]]] f:securityContext:map[f:runAsNonRoot:map[]] f:serviceAccountName:map[] f:terminationGracePeriodSeconds:map[] f:topologySpreadConstraints:map[k:{"topologyKey":"kubernetes.io/hostname","whenUnsatisfiable":"ScheduleAnyway"}:map[.:map[] f:labelSelector:map[] f:maxSkew:map[] f:topologyKey:map[] f:whenUnsatisfiable:map[]] k:{"topologyKey":"topology.kubernetes.io/zone","whenUnsatisfiable":"ScheduleAnyway"}:map[.:map[] f:labelSelector:map[] f:maxSkew:map[] f:topologyKey:map[] f:whenUnsatisfiable:map[]]] f:volumes:map[k:{"name":"cert"}:map[.:map[] f:name:map[] f:secret:map[f:defaultMode:map[] f:secretName:map[]]]]]]]] manager:ogx operation:Apply time:2026-07-05T17:31:36Z] map[apiVersion:apps/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:deployment.kubernetes.io/revision:map[]]] f:status:map[f:conditions:map[.:map[] k:{"type":"Available"}:map[.:map[] f:lastTransitionTime:map[] f:lastUpdateTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Progressing"}:map[.:map[] f:lastTransitionTime:map[] f:lastUpdateTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]]] f:observedGeneration:map[] f:replicas:map[] f:unavailableReplicas:map[] f:updatedReplicas:map[]]] manager:kube-controller-manager operation:Update subresource:status time:2026-07-05T17:31:36Z]] name:ogx-k8s-operator-controller-manager namespace:redhat-ods-applications ownerReferences:[map[apiVersion:components.platform.opendatahub.io/v1alpha1 blockOwnerDeletion:true controller:true kind:OGX name:default-ogx uid:8061b776-c124-4374-9ad5-d9382d2f0c63]] resourceVersion:121390 uid:e979aed2-fb84-4293-8028-27ab59bcf892] spec:map[progressDeadlineSeconds:600 replicas:1 revisionHistoryLimit:10 selector:map[matchLabels:map[app.kubernetes.io/name:ogx-k8s-operator app.kubernetes.io/part-of:ogx app.opendatahub.io/ogx:true control-plane:controller-manager]] strategy:map[rollingUpdate:map[maxSurge:25% maxUnavailable:25%] type:RollingUpdate] template:map[metadata:map[annotations:map[kubectl.kubernetes.io/default-container:manager] labels:map[app.kubernetes.io/name:ogx-k8s-operator app.kubernetes.io/part-of:ogx app.opendatahub.io/ogx:true control-plane:controller-manager]] spec:map[containers:[map[args:[--leader-elect] command:[/manager] env:[map[name:GOMEMLIMIT value:800MiB] map[name:OPERATOR_VERSION value:"0.11.0"] map[name:RELATED_IMAGE_RH_DISTRIBUTION value:registry.redhat.io/rhoai/odh-ogx-core-rhel9@sha256:8cca913e20b0f22de690ad0975365d933beccadf5110b698832fad3c5124518b]] image:registry.redhat.io/rhoai/odh-ogx-k8s-operator-rhel9@sha256:f7b82e87ac7ed8b10126a45828a11e8c3a77e0e867783b5ef49e896a5c37d810 imagePullPolicy:Always livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8081 scheme:HTTP] initialDelaySeconds:15 periodSeconds:20 successThreshold:1 timeoutSeconds:1] name:manager ports:[map[containerPort:9443 name:webhook-server protocol:TCP]] readinessProbe:map[failureThreshold:3 httpGet:map[path:/readyz port:8081 scheme:HTTP] initialDelaySeconds:5 periodSeconds:10 successThreshold:1 timeoutSeconds:1] resources:map[limits:map[cpu:500m memory:1Gi] requests:map[cpu:10m memory:256Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]]] terminationMessagePath:/dev/termination-log terminationMessagePolicy:File volumeMounts:[map[mountPath:/tmp/k8s-webhook-server/serving-certs name:cert readOnly:true]]]] dnsPolicy:ClusterFirst restartPolicy:Always schedulerName:default-scheduler securityContext:map[runAsNonRoot:true] serviceAccount:ogx-k8s-operator-controller-manager serviceAccountName:ogx-k8s-operator-controller-manager terminationGracePeriodSeconds:10 topologySpreadConstraints:[map[labelSelector:map[matchLabels:map[app.kubernetes.io/name:ogx-k8s-operator]] maxSkew:1 topologyKey:topology.kubernetes.io/zone whenUnsatisfiable:ScheduleAnyway] map[labelSelector:map[matchLabels:map[app.kubernetes.io/name:ogx-k8s-operator]] maxSkew:1 topologyKey:kubernetes.io/hostname whenUnsatisfiable:ScheduleAnyway]] volumes:[map[name:cert secret:map[defaultMode:420 secretName:ogx-k8s-operator-webhook-cert]]]]]] status:map[conditions:[map[lastTransitionTime:2026-07-05T17:31:36Z lastUpdateTime:2026-07-05T17:31:36Z message:Deployment does not have minimum availability. reason:MinimumReplicasUnavailable status:False type:Available] map[lastTransitionTime:2026-07-05T17:31:36Z lastUpdateTime:2026-07-05T17:31:36Z message:ReplicaSet "ogx-k8s-operator-controller-manager-94b957b46" is progressing. reason:ReplicaSetUpdated status:True type:Progressing]] observedGeneration:1 replicas:1 unavailableReplicas:1 updatedReplicas:1]] to match expression <string>: .spec.replicas == 2 --- FAIL: TestOdhOperator/components/group_1/ogx (16.17s) === RUN TestOdhOperator/components/group_1/kserve/Validate_resource_deletion_recovery/Deployment_deletion_recovery/deployment_llmisvc-controller-manager testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/kserve/Validate_resource_deletion_recovery/Deployment_deletion_recovery/deployment_llmisvc-controller-manager (95.02s) 2026/07/05 17:35:38 TEST FAILURE DETECTED: Deployment deletion recovery 2026/07/05 17:35:38 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:Deployment deletion recovery) === 2026/07/05 17:35:38 === CLUSTER STATE === 2026/07/05 17:35:38 Node: ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:35:38 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:35:38 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:35:38 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:35:38 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/05 17:35:38 Node: ip-10-0-137-37.ec2.internal 2026/07/05 17:35:38 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:35:38 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:35:38 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:35:38 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:35:38 Resources: 7500m CPU, 31236220Ki memory allocatable (of 8000m CPU, 32387196Ki memory capacity) 2026/07/05 17:35:38 Node: ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:35:38 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:35:38 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:35:38 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:35:38 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/05 17:35:38 === DEPLOYMENTS === 2026/07/05 17:35:38 Failed to collect deployment data: deployments not ready: redhat-ods-applications/llmisvc-controller-manager (0/1), redhat-ods-applications/rhods-dashboard (1/2) 2026/07/05 17:35:38 === PODS === 2026/07/05 17:35:38 Failed to collect pod data: redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-h9kc7: container manager not ready; redhat-ods-applications/rhods-dashboard-6c65779f4b-fmwmv: container automl-ui not ready; redhat-ods-applications/selector-migration-sgh2h: phase=Failed 2026/07/05 17:35:38 === OPERATOR STATUS === 2026/07/05 17:35:38 Operator deployment: 3/3 ready 2026/07/05 17:35:38 === DSCI STATUS === 2026/07/05 17:35:38 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/05 17:35:38 === DSC STATUS === 2026/07/05 17:35:38 Failed to collect DSC data: default-dsc unhealthy conditions: Ready=False; ComponentsReady=False; DashboardReady=False; KserveReady=False 2026/07/05 17:35:38 === RECENT EVENTS === 2026/07/05 17:35:38 17:35:31 INFO Pod/kuberay-operator-74d759f546-ljsv9: Killing - Stopping container kuberay-operator 2026/07/05 17:35:38 17:35:30 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkExecutorRunning - Executor spark-pi-8b56089f3359686c-exec-1 is running 2026/07/05 17:35:38 17:35:29 INFO Pod/spark-pi-8b56089f3359686c-exec-1: Started - Started container spark-kubernetes-executor 2026/07/05 17:35:38 17:35:29 INFO Pod/spark-pi-8b56089f3359686c-exec-1: AddedInterface - Add eth0 [10.133.0.121/23] from ovn-kubernetes 2026/07/05 17:35:38 17:35:29 INFO Pod/spark-pi-8b56089f3359686c-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/05 17:35:38 17:35:29 INFO Pod/spark-pi-8b56089f3359686c-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/05 17:35:38 17:35:28 INFO Pod/spark-pi-8b56089f3359686c-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-8b56089f3359686c-exec-1 to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:35:28 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-bc44d7c7b-h9kc7_3d60e20d-76d4-403f-839c-9b4c308da487 became leader 2026/07/05 17:35:38 17:35:28 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkExecutorPending - Executor spark-pi-8b56089f3359686c-exec-1 is pending 2026/07/05 17:35:38 17:35:28 INFO Lease/ray-operator-leader: LeaderElection - kuberay-operator-74d759f546-ljsv9_13210562-1da5-4cca-8090-9b58f50ca77f became leader 2026/07/05 17:35:38 17:35:19 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkDriverRunning - Driver spark-pi-d959b8e96nfs74re5dgg-driver is running 2026/07/05 17:35:38 17:35:19 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Started - Started container spark-kubernetes-driver 2026/07/05 17:35:38 17:35:19 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Created - Created container: spark-kubernetes-driver 2026/07/05 17:35:38 17:35:19 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Pulled - Successfully pulled image "quay.io/opendatahub/data-processing:Spark-v4.0.1" in 20.724s (20.724s including waiting). Image size: 1778928657 bytes. 2026/07/05 17:35:38 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" in 600ms (600ms including waiting). Image size: 177813676 bytes. 2026/07/05 17:35:38 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" in 490ms (490ms including waiting). Image size: 185088364 bytes. 2026/07/05 17:35:38 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: mlflow-ui 2026/07/05 17:35:38 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container mlflow-ui 2026/07/05 17:35:38 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" 2026/07/05 17:35:38 17:35:14 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" in 526ms (526ms including waiting). Image size: 187798114 bytes. 2026/07/05 17:35:38 17:35:14 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" 2026/07/05 17:35:38 17:35:14 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container maas-ui 2026/07/05 17:35:38 17:35:14 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: maas-ui 2026/07/05 17:35:38 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container gen-ai-ui 2026/07/05 17:35:38 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" in 529ms (529ms including waiting). Image size: 182434600 bytes. 2026/07/05 17:35:38 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: model-registry-ui 2026/07/05 17:35:38 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container model-registry-ui 2026/07/05 17:35:38 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" 2026/07/05 17:35:38 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" in 488ms (488ms including waiting). Image size: 241238855 bytes. 2026/07/05 17:35:38 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: gen-ai-ui 2026/07/05 17:35:38 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" 2026/07/05 17:35:38 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" in 616ms (616ms including waiting). Image size: 799818131 bytes. 2026/07/05 17:35:38 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container rhods-dashboard 2026/07/05 17:35:38 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: rhods-dashboard 2026/07/05 17:35:38 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" 2026/07/05 17:35:38 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container kube-rbac-proxy 2026/07/05 17:35:38 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: kube-rbac-proxy 2026/07/05 17:35:38 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:ac5537f459cd812a12e8f6ae13001150e29954913007b1790a44b15fb75f4926" already present on machine 2026/07/05 17:35:38 17:35:11 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" 2026/07/05 17:35:38 17:35:11 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: AddedInterface - Add eth0 [10.134.0.72/23] from ovn-kubernetes 2026/07/05 17:35:38 17:35:10 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-6c65779f4b-fmwmv to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:35:10 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Pulled - Successfully pulled image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" in 613ms (613ms including waiting). Image size: 173525457 bytes. 2026/07/05 17:35:38 17:35:10 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Created - Created container: manager 2026/07/05 17:35:38 17:35:10 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Started - Started container manager 2026/07/05 17:35:38 17:35:10 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" in 500ms (500ms including waiting). Image size: 236362435 bytes. 2026/07/05 17:35:38 17:35:10 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Started - Started container manager 2026/07/05 17:35:38 17:35:10 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Created - Created container: manager 2026/07/05 17:35:38 17:35:09 INFO Pod/notebook-controller-deployment-6b68db7cbd-dp87m: Pulling - Pulling image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" 2026/07/05 17:35:38 17:35:09 INFO Pod/kuberay-operator-74d759f546-ljsv9: Pulled - Container image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" already present on machine 2026/07/05 17:35:38 17:35:09 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" 2026/07/05 17:35:38 17:35:09 INFO Pod/kuberay-operator-74d759f546-ljsv9: Started - Started container kuberay-operator 2026/07/05 17:35:38 17:35:09 INFO Pod/notebook-controller-deployment-6b68db7cbd-dp87m: Pulled - Successfully pulled image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" in 469ms (469ms including waiting). Image size: 168349711 bytes. 2026/07/05 17:35:38 17:35:09 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Started - Started container controller 2026/07/05 17:35:38 17:35:09 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Created - Created container: controller 2026/07/05 17:35:38 17:35:09 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Pulled - Successfully pulled image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" in 1m23.925s (1m23.925s including waiting). Image size: 2736477781 bytes. 2026/07/05 17:35:38 17:35:09 INFO Pod/kuberay-operator-74d759f546-ljsv9: Created - Created container: kuberay-operator 2026/07/05 17:35:38 17:35:09 INFO Pod/notebook-controller-deployment-6b68db7cbd-dp87m: AddedInterface - Add eth0 [10.134.0.68/23] from ovn-kubernetes 2026/07/05 17:35:38 17:35:09 INFO Pod/kuberay-operator-74d759f546-ljsv9: AddedInterface - Add eth0 [10.134.0.71/23] from ovn-kubernetes 2026/07/05 17:35:38 17:35:09 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: AddedInterface - Add eth0 [10.134.0.70/23] from ovn-kubernetes 2026/07/05 17:35:38 17:35:09 WARN Pod/notebook-controller-deployment-6b68db7cbd-dp87m: Failed - Error: configmap "notebook-controller-config" not found 2026/07/05 17:35:38 17:35:09 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Pulling - Pulling image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" 2026/07/05 17:35:38 17:35:09 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: AddedInterface - Add eth0 [10.134.0.69/23] from ovn-kubernetes 2026/07/05 17:35:38 17:35:07 WARN Pod/kuberay-operator-74d759f546-g4x5g: Unhealthy - Readiness probe failed: Get "http://10.134.0.66:8080/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/05 17:35:38 17:34:59 WARN Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Unhealthy - Readiness probe failed: Get "http://10.134.0.61:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/05 17:35:38 17:34:59 WARN Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Unhealthy - Readiness probe failed: Get "http://10.134.0.60:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/05 17:35:38 17:34:58 WARN Pod/rhods-dashboard-6c65779f4b-fmwmv: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/07/05 17:35:38 17:34:58 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/05 17:35:38 17:34:58 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Created - Created container: webhook 2026/07/05 17:35:38 17:34:58 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Started - Started container webhook 2026/07/05 17:35:38 17:34:58 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: AddedInterface - Add eth0 [10.133.0.119/23] from ovn-kubernetes 2026/07/05 17:35:38 17:34:58 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Pulling - Pulling image "quay.io/opendatahub/data-processing:Spark-v4.0.1" 2026/07/05 17:35:38 17:34:58 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: AddedInterface - Add eth0 [10.133.0.120/23] from ovn-kubernetes 2026/07/05 17:35:38 17:34:56 WARN Pod/kuberay-operator-74d759f546-g4x5g: Unhealthy - Readiness probe failed: Get "http://10.134.0.66:8080/metrics": dial tcp 10.134.0.66:8080: connect: connection refused 2026/07/05 17:35:38 17:34:54 WARN Pod/kuberay-operator-74d759f546-llk9f: Unhealthy - Readiness probe failed: Get "http://10.133.0.118:8080/metrics": dial tcp 10.133.0.118:8080: connect: connection refused 2026/07/05 17:35:38 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" 2026/07/05 17:35:38 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" in 462ms (462ms including waiting). Image size: 185088364 bytes. 2026/07/05 17:35:38 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: mlflow-ui 2026/07/05 17:35:38 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" in 394ms (394ms including waiting). Image size: 177813676 bytes. 2026/07/05 17:35:38 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container mlflow-ui 2026/07/05 17:35:38 17:34:43 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-d959b8e96nfs74re5dgg-driver to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" 2026/07/05 17:35:38 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: maas-ui 2026/07/05 17:35:38 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" 2026/07/05 17:35:38 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container gen-ai-ui 2026/07/05 17:35:38 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: gen-ai-ui 2026/07/05 17:35:38 17:34:43 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkApplicationSubmitted - SparkApplication spark-pi-d959b8e96nfs74re5dgg was submitted successfully 2026/07/05 17:35:38 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" in 381ms (381ms including waiting). Image size: 241238855 bytes. 2026/07/05 17:35:38 17:34:43 WARN Pod/spark-pi-d959b8e96nfs74re5dgg-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-89627a9f3358b879-conf-map" not found 2026/07/05 17:35:38 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container model-registry-ui 2026/07/05 17:35:38 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: model-registry-ui 2026/07/05 17:35:38 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" 2026/07/05 17:35:38 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" in 366ms (366ms including waiting). Image size: 187798114 bytes. 2026/07/05 17:35:38 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container maas-ui 2026/07/05 17:35:38 17:34:43 WARN Pod/spark-pi-d959b8e96nfs74re5dgg-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d959b8e96nfs74re5dgg-3380d59f3358a115-driver-podspec-conf-map" not found 2026/07/05 17:35:38 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" 2026/07/05 17:35:38 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" in 447ms (447ms including waiting). Image size: 182434600 bytes. 2026/07/05 17:35:38 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container rhods-dashboard 2026/07/05 17:35:38 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: rhods-dashboard 2026/07/05 17:35:38 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" in 456ms (456ms including waiting). Image size: 799818131 bytes. 2026/07/05 17:35:38 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: kube-rbac-proxy 2026/07/05 17:35:38 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:ac5537f459cd812a12e8f6ae13001150e29954913007b1790a44b15fb75f4926" already present on machine 2026/07/05 17:35:38 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container kube-rbac-proxy 2026/07/05 17:35:38 17:34:41 WARN Pod/rhods-dashboard-6c65779f4b-fmwmv: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/07/05 17:35:38 17:34:41 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-6c65779f4b-lstrb to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:38 17:34:41 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: AddedInterface - Add eth0 [10.132.0.50/23] from ovn-kubernetes 2026/07/05 17:35:38 17:34:41 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" 2026/07/05 17:35:38 17:34:41 INFO ReplicaSet/rhods-dashboard-6c65779f4b: SuccessfulCreate - Created pod: rhods-dashboard-6c65779f4b-fmwmv 2026/07/05 17:35:38 17:34:41 INFO ReplicaSet/rhods-dashboard-6c65779f4b: SuccessfulCreate - Created pod: rhods-dashboard-6c65779f4b-lstrb 2026/07/05 17:35:38 17:34:40 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-6c65779f4b from 0 to 2 2026/07/05 17:35:38 17:34:36 INFO Pod/kuberay-operator-74d759f546-ljsv9: Scheduled - Successfully assigned redhat-ods-applications/kuberay-operator-74d759f546-ljsv9 to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:34:36 INFO Deployment/kuberay-operator: ScalingReplicaSet - Scaled up replica set kuberay-operator-74d759f546 from 0 to 1 2026/07/05 17:35:38 17:34:36 INFO ReplicaSet/kuberay-operator-74d759f546: SuccessfulCreate - Created pod: kuberay-operator-74d759f546-ljsv9 2026/07/05 17:35:38 17:34:35 INFO Pod/kuberay-operator-74d759f546-g4x5g: Killing - Stopping container kuberay-operator 2026/07/05 17:35:38 17:34:35 INFO Pod/kuberay-operator-74d759f546-llk9f: Killing - Stopping container kuberay-operator 2026/07/05 17:35:38 17:34:06 WARN Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Unhealthy - Readiness probe failed: Get "http://10.133.0.104:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/05 17:35:38 17:34:05 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-h9kc7 to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:34:05 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Scheduled - Successfully assigned redhat-ods-applications/odh-notebook-controller-manager-6f96b94f4c-7jxl6 to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:34:05 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-wx56c to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:34:05 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-wx56c 2026/07/05 17:35:38 17:34:05 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 1 to 2 2026/07/05 17:35:38 17:34:05 INFO ReplicaSet/llmisvc-controller-manager-bc44d7c7b: SuccessfulCreate - Created pod: llmisvc-controller-manager-bc44d7c7b-h9kc7 2026/07/05 17:35:38 17:34:05 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-bc44d7c7b from 0 to 1 2026/07/05 17:35:38 17:34:05 INFO Deployment/odh-notebook-controller-manager: ScalingReplicaSet - Scaled up replica set odh-notebook-controller-manager-6f96b94f4c from 0 to 1 2026/07/05 17:35:38 17:34:05 INFO ReplicaSet/odh-notebook-controller-manager-6f96b94f4c: SuccessfulCreate - Created pod: odh-notebook-controller-manager-6f96b94f4c-7jxl6 2026/07/05 17:35:38 17:34:04 WARN Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Unhealthy - Readiness probe failed: Get "http://10.134.0.60:8081/readyz": dial tcp 10.134.0.60:8081: connect: connection refused 2026/07/05 17:35:38 17:34:04 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: Killing - Stopping container manager 2026/07/05 17:35:38 17:34:04 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Killing - Stopping container manager 2026/07/05 17:35:38 17:34:03 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Killing - Stopping container manager 2026/07/05 17:35:38 17:34:03 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: Killing - Stopping container manager 2026/07/05 17:35:38 17:34:02 INFO Pod/kuberay-operator-74d759f546-llk9f: Pulled - Successfully pulled image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" in 1.407s (1.407s including waiting). Image size: 179205940 bytes. 2026/07/05 17:35:38 17:34:02 INFO Lease/ray-operator-leader: LeaderElection - kuberay-operator-74d759f546-g4x5g_34d876db-58de-4ab5-ab74-052534dfbdba became leader 2026/07/05 17:35:38 17:34:02 INFO Pod/kuberay-operator-74d759f546-llk9f: Created - Created container: kuberay-operator 2026/07/05 17:35:38 17:34:02 INFO Pod/kuberay-operator-74d759f546-llk9f: Started - Started container kuberay-operator 2026/07/05 17:35:38 17:34:01 INFO Pod/kuberay-operator-74d759f546-llk9f: AddedInterface - Add eth0 [10.133.0.118/23] from ovn-kubernetes 2026/07/05 17:35:38 17:34:01 INFO Pod/kuberay-operator-74d759f546-llk9f: Pulling - Pulling image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" 2026/07/05 17:35:38 17:34:00 INFO Pod/kuberay-operator-74d759f546-llk9f: Scheduled - Successfully assigned redhat-ods-applications/kuberay-operator-74d759f546-llk9f to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:34:00 INFO Pod/notebook-controller-deployment-6b68db7cbd-dp87m: Scheduled - Successfully assigned redhat-ods-applications/notebook-controller-deployment-6b68db7cbd-dp87m to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:34:00 INFO Deployment/notebook-controller-deployment: ScalingReplicaSet - Scaled up replica set notebook-controller-deployment-6b68db7cbd from 0 to 1 2026/07/05 17:35:38 17:34:00 INFO ReplicaSet/notebook-controller-deployment-6b68db7cbd: SuccessfulCreate - Created pod: notebook-controller-deployment-6b68db7cbd-dp87m 2026/07/05 17:35:38 17:34:00 INFO Deployment/kuberay-operator: ScalingReplicaSet - Scaled up replica set kuberay-operator-74d759f546 from 1 to 2 2026/07/05 17:35:38 17:34:00 INFO ReplicaSet/kuberay-operator-74d759f546: SuccessfulCreate - Created pod: kuberay-operator-74d759f546-llk9f 2026/07/05 17:35:38 17:33:59 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: Killing - Stopping container manager 2026/07/05 17:35:38 17:33:59 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Killing - Stopping container manager 2026/07/05 17:35:38 17:33:45 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-jdvzx to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:33:45 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Pulling - Pulling image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" 2026/07/05 17:35:38 17:33:45 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-jdvzx 2026/07/05 17:35:38 17:33:45 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 1 to 2 2026/07/05 17:35:38 17:33:45 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: AddedInterface - Add eth0 [10.134.0.67/23] from ovn-kubernetes 2026/07/05 17:35:38 17:33:44 INFO Pod/kuberay-operator-74d759f546-g4x5g: Created - Created container: kuberay-operator 2026/07/05 17:35:38 17:33:44 INFO Pod/kuberay-operator-74d759f546-g4x5g: Started - Started container kuberay-operator 2026/07/05 17:35:38 17:33:44 INFO Pod/kuberay-operator-74d759f546-g4x5g: Pulled - Successfully pulled image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" in 1.631s (1.631s including waiting). Image size: 179205940 bytes. 2026/07/05 17:35:38 17:33:42 INFO Pod/kuberay-operator-74d759f546-g4x5g: Pulling - Pulling image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" 2026/07/05 17:35:38 17:33:42 INFO Pod/kuberay-operator-74d759f546-g4x5g: AddedInterface - Add eth0 [10.134.0.66/23] from ovn-kubernetes 2026/07/05 17:35:38 17:33:41 INFO Pod/kuberay-operator-74d759f546-g4x5g: Scheduled - Successfully assigned redhat-ods-applications/kuberay-operator-74d759f546-g4x5g to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:33:41 WARN Pod/kuberay-operator-74d759f546-g4x5g: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 2026/07/05 17:35:38 17:33:41 INFO ReplicaSet/kuberay-operator-74d759f546: SuccessfulCreate - Created pod: kuberay-operator-74d759f546-g4x5g 2026/07/05 17:35:38 17:33:41 INFO Deployment/kuberay-operator: ScalingReplicaSet - Scaled up replica set kuberay-operator-74d759f546 from 0 to 1 2026/07/05 17:35:38 17:33:38 INFO Lease/spark-operator-webhook-lock: LeaderElection - spark-operator-webhook-5f8768dbfd-rgktb_b0ba836a-be37-4a30-a983-1f197f6c49e2 became leader 2026/07/05 17:35:38 17:33:38 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-588b4b787c-4nm25_e7ea5d10-a5b5-4cb5-b915-1e85d8cc1b75 became leader 2026/07/05 17:35:38 17:33:36 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-5dd7d48f87-s2gzm_70a06366-6c28-43ad-8a34-79526b6f18e7 became leader 2026/07/05 17:35:38 17:33:30 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5f7b7f7655-mf2jg_d4d6fa96-e67f-491c-ba63-6964158add9e became leader 2026/07/05 17:35:38 17:33:27 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:33:27 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:33:27 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd98nx4m: Killing - Stopping container manager 2026/07/05 17:35:38 17:33:27 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: Killing - Stopping container manager 2026/07/05 17:35:38 17:33:26 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:33:26 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:33:25 INFO ReplicaSet/rhods-dashboard-6c65779f4b: SuccessfulDelete - Deleted pod: rhods-dashboard-6c65779f4b-5zgbz 2026/07/05 17:35:38 17:33:25 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled down replica set rhods-dashboard-6c65779f4b from 2 to 1 2026/07/05 17:35:38 17:33:25 INFO Pod/kubeflow-training-operator-7c59bdd544-qns74: Killing - Stopping container training-operator 2026/07/05 17:35:38 17:33:24 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:33:24 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: AddedInterface - Add eth0 [10.134.0.65/23] from ovn-kubernetes 2026/07/05 17:35:38 17:33:22 INFO Pod/kuberay-operator-74d759f546-b82w5: Killing - Stopping container kuberay-operator 2026/07/05 17:35:38 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Started - Started container manager 2026/07/05 17:35:38 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Created - Created container: manager 2026/07/05 17:35:38 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" in 385ms (385ms including waiting). Image size: 238427353 bytes. 2026/07/05 17:35:38 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" 2026/07/05 17:35:38 17:33:21 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-588b4b787c-4nm25 to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:33:21 INFO ReplicaSet/kserve-controller-manager-588b4b787c: SuccessfulCreate - Created pod: kserve-controller-manager-588b4b787c-4nm25 2026/07/05 17:35:38 17:33:21 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-588b4b787c from 0 to 1 2026/07/05 17:35:38 17:33:19 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Created - Created container: controller 2026/07/05 17:35:38 17:33:19 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Started - Started container webhook 2026/07/05 17:35:38 17:33:19 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: AddedInterface - Add eth0 [10.133.0.116/23] from ovn-kubernetes 2026/07/05 17:35:38 17:33:19 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Created - Created container: webhook 2026/07/05 17:35:38 17:33:19 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/05 17:35:38 17:33:19 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Started - Started container controller 2026/07/05 17:35:38 17:33:19 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/05 17:35:38 17:33:19 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: AddedInterface - Add eth0 [10.133.0.117/23] from ovn-kubernetes 2026/07/05 17:35:38 17:33:19 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-8648b6bb44-l65kd_20a05650-424e-43df-a54f-c97a806596e4 became leader 2026/07/05 17:35:38 17:33:18 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:33:18 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:33:18 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-rgktb to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:33:18 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-s2gzm to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:33:18 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-rgktb 2026/07/05 17:35:38 17:33:18 WARN ReplicaSet/spark-operator-controller-5dd7d48f87: FailedCreate - Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 2026/07/05 17:35:38 17:33:18 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: Killing - Stopping container manager 2026/07/05 17:35:38 17:33:18 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-s2gzm 2026/07/05 17:35:38 17:33:18 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 0 to 1 2026/07/05 17:35:38 17:33:18 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 0 to 1 2026/07/05 17:35:38 17:33:13 INFO Pod/feast-operator-controller-manager-7b78bd74b9-cpccm: Killing - Stopping container manager 2026/07/05 17:35:38 17:33:12 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd98nx4m: Scheduled - Successfully assigned redhat-ods-applications/data-science-pipelines-operator-controller-manager-595ffd98nx4m to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:33:12 INFO Lease/66483dbd.feast.dev: LeaderElection - feast-operator-controller-manager-7b78bd74b9-cpccm_715eee42-83b7-4cd9-a524-f0188710b2f6 became leader 2026/07/05 17:35:38 17:33:12 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd98nx4m: AddedInterface - Add eth0 [10.133.0.115/23] from ovn-kubernetes 2026/07/05 17:35:38 17:33:12 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd98nx4m: Pulled - Container image "quay.io/rhoai/odh-data-science-pipelines-operator-controller-rhel9@sha256:d9eaf8dab3521c7b97f1ed5d48481a4306122bb7bc4b751a965ebc9479dc2d93" already present on machine 2026/07/05 17:35:38 17:33:12 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd98nx4m: Created - Created container: manager 2026/07/05 17:35:38 17:33:12 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd98nx4m: Started - Started container manager 2026/07/05 17:35:38 17:33:12 INFO ReplicaSet/data-science-pipelines-operator-controller-manager-595ffd9458: SuccessfulCreate - Created pod: data-science-pipelines-operator-controller-manager-595ffd98nx4m 2026/07/05 17:35:38 17:33:12 INFO Deployment/data-science-pipelines-operator-controller-manager: ScalingReplicaSet - Scaled up replica set data-science-pipelines-operator-controller-manager-595ffd9458 from 0 to 1 2026/07/05 17:35:38 17:33:11 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: Killing - Stopping container manager 2026/07/05 17:35:38 17:33:11 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: Killing - Stopping container manager 2026/07/05 17:35:38 17:33:10 INFO Pod/odh-model-controller-566d8764f4-gmggg: Killing - Stopping container manager 2026/07/05 17:35:38 17:33:10 INFO ReplicaSet/odh-model-controller-566d8764f4: SuccessfulDelete - Deleted pod: odh-model-controller-566d8764f4-gmggg 2026/07/05 17:35:38 17:33:10 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled down replica set odh-model-controller-566d8764f4 from 1 to 0 2026/07/05 17:35:38 17:33:07 INFO Pod/spark-operator-webhook-5f8768dbfd-nfws7: Killing - Stopping container webhook 2026/07/05 17:35:38 17:33:07 INFO Pod/spark-operator-controller-5dd7d48f87-bv2wn: Killing - Stopping container controller 2026/07/05 17:35:38 17:33:06 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:33:06 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:33:03 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: Started - Started container manager 2026/07/05 17:35:38 17:33:03 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: Created - Created container: manager 2026/07/05 17:35:38 17:33:03 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: Pulled - Successfully pulled image "quay.io/rhoai/odh-trainer-rhel9@sha256:27961006c645f0800ecab6ec75c48d98de4aa1ea1d65b37652056f3cd5684ac5" in 2.418s (2.418s including waiting). Image size: 157432247 bytes. 2026/07/05 17:35:38 17:33:01 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: Pulling - Pulling image "quay.io/rhoai/odh-trainer-rhel9@sha256:27961006c645f0800ecab6ec75c48d98de4aa1ea1d65b37652056f3cd5684ac5" 2026/07/05 17:35:38 17:33:01 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: AddedInterface - Add eth0 [10.133.0.114/23] from ovn-kubernetes 2026/07/05 17:35:38 17:33:00 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-8648b6bb44-l65kd to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:33:00 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-8648b6bb44 from 0 to 1 2026/07/05 17:35:38 17:33:00 INFO ReplicaSet/kubeflow-trainer-controller-manager-8648b6bb44: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-8648b6bb44-l65kd 2026/07/05 17:35:38 17:32:59 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: Killing - Stopping container manager 2026/07/05 17:35:38 17:32:59 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l: Killing - Stopping container manager 2026/07/05 17:35:38 17:32:58 INFO Pod/odh-model-controller-5f7b7f7655-mf2jg: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5f7b7f7655-mf2jg to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:32:58 INFO Pod/odh-model-controller-5f7b7f7655-mf2jg: AddedInterface - Add eth0 [10.133.0.113/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:58 INFO ReplicaSet/odh-model-controller-5f7b7f7655: SuccessfulCreate - Created pod: odh-model-controller-5f7b7f7655-mf2jg 2026/07/05 17:35:38 17:32:58 INFO Pod/odh-model-controller-5f7b7f7655-mf2jg: Created - Created container: manager 2026/07/05 17:35:38 17:32:58 INFO Pod/odh-model-controller-5f7b7f7655-mf2jg: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" in 344ms (344ms including waiting). Image size: 238959021 bytes. 2026/07/05 17:35:38 17:32:58 INFO Pod/odh-model-controller-5f7b7f7655-mf2jg: Started - Started container manager 2026/07/05 17:35:38 17:32:58 INFO Pod/odh-model-controller-5f7b7f7655-mf2jg: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" 2026/07/05 17:35:38 17:32:57 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: Killing - Stopping container manager 2026/07/05 17:35:38 17:32:57 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5f7b7f7655 from 0 to 1 2026/07/05 17:35:38 17:32:56 INFO Pod/kubeflow-training-operator-7c59bdd544-qns74: AddedInterface - Add eth0 [10.133.0.112/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:56 INFO Pod/feast-operator-controller-manager-7b78bd74b9-cpccm: Pulled - Container image "quay.io/rhoai/odh-feast-operator-rhel9@sha256:fe1b375defb9ab9e79f523fd63103931941bfa0d7ff530ddd0d0db6fce412c8d" already present on machine 2026/07/05 17:35:38 17:32:56 INFO Pod/feast-operator-controller-manager-7b78bd74b9-cpccm: AddedInterface - Add eth0 [10.133.0.111/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:56 INFO Pod/kubeflow-training-operator-7c59bdd544-qns74: Started - Started container training-operator 2026/07/05 17:35:38 17:32:56 INFO Pod/feast-operator-controller-manager-7b78bd74b9-cpccm: Created - Created container: manager 2026/07/05 17:35:38 17:32:56 INFO Pod/feast-operator-controller-manager-7b78bd74b9-cpccm: Started - Started container manager 2026/07/05 17:35:38 17:32:56 INFO Pod/kubeflow-training-operator-7c59bdd544-qns74: Pulled - Container image "quay.io/rhoai/odh-training-operator-rhel9@sha256:752923f0362c0070dfdcdfb6fc0a10577cdda34007a88375c00a602ba96f2b66" already present on machine 2026/07/05 17:35:38 17:32:56 INFO Pod/kubeflow-training-operator-7c59bdd544-qns74: Created - Created container: training-operator 2026/07/05 17:35:38 17:32:55 INFO Pod/kubeflow-training-operator-7c59bdd544-qns74: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-training-operator-7c59bdd544-qns74 to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:32:55 INFO Pod/feast-operator-controller-manager-7b78bd74b9-cpccm: Scheduled - Successfully assigned redhat-ods-applications/feast-operator-controller-manager-7b78bd74b9-cpccm to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:32:55 INFO ReplicaSet/feast-operator-controller-manager-7b78bd74b9: SuccessfulCreate - Created pod: feast-operator-controller-manager-7b78bd74b9-cpccm 2026/07/05 17:35:38 17:32:55 INFO Pod/kubeflow-training-operator-7c59bdd544-r8lkq: Killing - Stopping container training-operator 2026/07/05 17:35:38 17:32:55 INFO ReplicaSet/kubeflow-training-operator-7c59bdd544: SuccessfulCreate - Created pod: kubeflow-training-operator-7c59bdd544-qns74 2026/07/05 17:35:38 17:32:55 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: Killing - Stopping container training-operator 2026/07/05 17:35:38 17:32:55 INFO Deployment/feast-operator-controller-manager: ScalingReplicaSet - Scaled up replica set feast-operator-controller-manager-7b78bd74b9 from 0 to 1 2026/07/05 17:35:38 17:32:55 INFO Deployment/kubeflow-training-operator: ScalingReplicaSet - Scaled up replica set kubeflow-training-operator-7c59bdd544 from 0 to 1 2026/07/05 17:35:38 17:32:54 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: Killing - Stopping container manager 2026/07/05 17:35:38 17:32:54 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: Killing - Stopping container manager 2026/07/05 17:35:38 17:32:48 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-b67847846-7vq2t_9fd9b1d9-cc9f-403f-b680-73e37b839fde became leader 2026/07/05 17:35:38 17:32:46 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: Created - Created container: manager 2026/07/05 17:35:38 17:32:46 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: Started - Started container manager 2026/07/05 17:35:38 17:32:45 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: AddedInterface - Add eth0 [10.133.0.110/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:45 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: Pulling - Pulling image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" 2026/07/05 17:35:38 17:32:45 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: Pulled - Successfully pulled image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" in 536ms (536ms including waiting). Image size: 173525457 bytes. 2026/07/05 17:35:38 17:32:44 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: Scheduled - Successfully assigned redhat-ods-applications/odh-notebook-controller-manager-6f96b94f4c-tmjpf to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:32:44 INFO Deployment/odh-notebook-controller-manager: ScalingReplicaSet - Scaled up replica set odh-notebook-controller-manager-6f96b94f4c from 1 to 2 2026/07/05 17:35:38 17:32:44 INFO ReplicaSet/odh-notebook-controller-manager-6f96b94f4c: SuccessfulCreate - Created pod: odh-notebook-controller-manager-6f96b94f4c-tmjpf 2026/07/05 17:35:38 17:32:39 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" 2026/07/05 17:35:38 17:32:39 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" in 515ms (515ms including waiting). Image size: 177813676 bytes. 2026/07/05 17:35:38 17:32:39 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" in 507ms (507ms including waiting). Image size: 185088364 bytes. 2026/07/05 17:35:38 17:32:39 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Started - Started container mlflow-ui 2026/07/05 17:35:38 17:32:39 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Created - Created container: mlflow-ui 2026/07/05 17:35:38 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" in 382ms (382ms including waiting). Image size: 177813676 bytes. 2026/07/05 17:35:38 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" in 483ms (483ms including waiting). Image size: 187798114 bytes. 2026/07/05 17:35:38 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Created - Created container: maas-ui 2026/07/05 17:35:38 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Started - Started container maas-ui 2026/07/05 17:35:38 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" 2026/07/05 17:35:38 17:32:38 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-588b4b787c-gk7pm_4323bc22-edb8-450f-a82c-231f54f5def9 became leader 2026/07/05 17:35:38 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Created - Created container: mlflow-ui 2026/07/05 17:35:38 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" in 453ms (453ms including waiting). Image size: 185088364 bytes. 2026/07/05 17:35:38 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" 2026/07/05 17:35:38 17:32:38 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-bc44d7c7b-p794f_7c1236ca-7649-45ed-902d-f628ca1c3adf became leader 2026/07/05 17:35:38 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Started - Started container mlflow-ui 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Created - Created container: gen-ai-ui 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Started - Started container maas-ui 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Created - Created container: maas-ui 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" in 558ms (558ms including waiting). Image size: 187798114 bytes. 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Started - Started container gen-ai-ui 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Created - Created container: gen-ai-ui 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" in 428ms (428ms including waiting). Image size: 241238855 bytes. 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" in 428ms (428ms including waiting). Image size: 182434600 bytes. 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Created - Created container: model-registry-ui 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Started - Started container model-registry-ui 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" in 412ms (412ms including waiting). Image size: 241238855 bytes. 2026/07/05 17:35:38 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Started - Started container gen-ai-ui 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Started - Started container rhods-dashboard 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Created - Created container: rhods-dashboard 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Started - Started container kube-rbac-proxy 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Created - Created container: kube-rbac-proxy 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:ac5537f459cd812a12e8f6ae13001150e29954913007b1790a44b15fb75f4926" already present on machine 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Started - Started container rhods-dashboard 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Created - Created container: rhods-dashboard 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" in 451ms (451ms including waiting). Image size: 799818131 bytes. 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: AddedInterface - Add eth0 [10.134.0.64/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Started - Started container model-registry-ui 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Created - Created container: model-registry-ui 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" in 475ms (475ms including waiting). Image size: 182434600 bytes. 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Started - Started container kube-rbac-proxy 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Created - Created container: kube-rbac-proxy 2026/07/05 17:35:38 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:ac5537f459cd812a12e8f6ae13001150e29954913007b1790a44b15fb75f4926" already present on machine 2026/07/05 17:35:38 17:32:35 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-6c65779f4b-5zgbz to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:32:35 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-6c65779f4b-xcrkd to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:38 17:32:35 INFO ReplicaSet/rhods-dashboard-6c65779f4b: SuccessfulCreate - Created pod: rhods-dashboard-6c65779f4b-xcrkd 2026/07/05 17:35:38 17:32:35 INFO ReplicaSet/rhods-dashboard-6c65779f4b: SuccessfulCreate - Created pod: rhods-dashboard-6c65779f4b-5zgbz 2026/07/05 17:35:38 17:32:35 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" in 431ms (431ms including waiting). Image size: 799818131 bytes. 2026/07/05 17:35:38 17:32:35 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" 2026/07/05 17:35:38 17:32:35 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: AddedInterface - Add eth0 [10.132.0.49/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:34 WARN ReplicaSet/rhods-dashboard-6c65779f4b: FailedCreate - Error creating: pods "rhods-dashboard-6c65779f4b-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 2026/07/05 17:35:38 17:32:33 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: Created - Created container: manager 2026/07/05 17:35:38 17:32:33 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: Pulled - Successfully pulled image "quay.io/rhoai/odh-data-science-pipelines-operator-controller-rhel9@sha256:d9eaf8dab3521c7b97f1ed5d48481a4306122bb7bc4b751a965ebc9479dc2d93" in 1.832s (1.832s including waiting). Image size: 176639426 bytes. 2026/07/05 17:35:38 17:32:33 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-6c65779f4b from 0 to 2 2026/07/05 17:35:38 17:32:33 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: Started - Started container manager 2026/07/05 17:35:38 17:32:32 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: Started - Started container manager 2026/07/05 17:35:38 17:32:32 INFO Pod/maas-api-557679ff4c-rn27g: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:86e3f628c409b1afa2773d3e0ed3ab7f977a4da82d6576d4b9cd3b3ddbadb447" in 2.09s (2.09s including waiting). Image size: 175014370 bytes. 2026/07/05 17:35:38 17:32:32 INFO Pod/maas-api-557679ff4c-rn27g: Started - Started container maas-api 2026/07/05 17:35:38 17:32:32 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: Created - Created container: manager 2026/07/05 17:35:38 17:32:32 INFO Pod/maas-api-557679ff4c-rn27g: Created - Created container: maas-api 2026/07/05 17:35:38 17:32:32 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:e1bd98206aee94e76215e2a9ea4987e9c6f4597bdba1358921496b275e91abc0" in 2.079s (2.079s including waiting). Image size: 211352054 bytes. 2026/07/05 17:35:38 17:32:31 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: Scheduled - Successfully assigned redhat-ods-applications/data-science-pipelines-operator-controller-manager-595ffd9dtr6j to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:32:31 INFO Deployment/data-science-pipelines-operator-controller-manager: ScalingReplicaSet - Scaled up replica set data-science-pipelines-operator-controller-manager-595ffd9458 from 1 to 2 2026/07/05 17:35:38 17:32:31 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: AddedInterface - Add eth0 [10.134.0.63/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:31 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: Pulling - Pulling image "quay.io/rhoai/odh-data-science-pipelines-operator-controller-rhel9@sha256:d9eaf8dab3521c7b97f1ed5d48481a4306122bb7bc4b751a965ebc9479dc2d93" 2026/07/05 17:35:38 17:32:31 INFO ReplicaSet/data-science-pipelines-operator-controller-manager-595ffd9458: SuccessfulCreate - Created pod: data-science-pipelines-operator-controller-manager-595ffd9dtr6j 2026/07/05 17:35:38 17:32:30 INFO Pod/maas-api-557679ff4c-rn27g: AddedInterface - Add eth0 [10.132.0.45/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:30 INFO Pod/dashboard-redirect-b564449f8-jxpdv: Pulled - Container image "registry.redhat.io/ubi9/nginx-126@sha256:f0a79ccf21b8780a7534d78ee0e49d1852654f5f0f52e17ebdbe185a71d93253" already present on machine 2026/07/05 17:35:38 17:32:30 INFO Pod/dashboard-redirect-b564449f8-jxpdv: Started - Started container nginx 2026/07/05 17:35:38 17:32:30 INFO Pod/dashboard-redirect-b564449f8-jxpdv: AddedInterface - Add eth0 [10.132.0.44/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:30 INFO Pod/maas-api-557679ff4c-rn27g: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:86e3f628c409b1afa2773d3e0ed3ab7f977a4da82d6576d4b9cd3b3ddbadb447" 2026/07/05 17:35:38 17:32:30 INFO Pod/dashboard-redirect-b564449f8-jxpdv: Created - Created container: nginx 2026/07/05 17:35:38 17:32:30 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: Pulling - Pulling image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:e1bd98206aee94e76215e2a9ea4987e9c6f4597bdba1358921496b275e91abc0" 2026/07/05 17:35:38 17:32:30 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: AddedInterface - Add eth0 [10.132.0.48/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:29 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-b67847846-7vq2t to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:38 17:32:29 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: Pulled - Successfully pulled image "quay.io/rhoai/odh-trainer-rhel9@sha256:27961006c645f0800ecab6ec75c48d98de4aa1ea1d65b37652056f3cd5684ac5" in 4.405s (4.405s including waiting). Image size: 157432247 bytes. 2026/07/05 17:35:38 17:32:29 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: Pulled - Successfully pulled image "quay.io/rhoai/odh-training-operator-rhel9@sha256:752923f0362c0070dfdcdfb6fc0a10577cdda34007a88375c00a602ba96f2b66" in 3.732s (3.732s including waiting). Image size: 203250147 bytes. 2026/07/05 17:35:38 17:32:29 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: Pulled - Successfully pulled image "quay.io/rhoai/odh-feast-operator-rhel9@sha256:fe1b375defb9ab9e79f523fd63103931941bfa0d7ff530ddd0d0db6fce412c8d" in 3.758s (3.758s including waiting). Image size: 185429423 bytes. 2026/07/05 17:35:38 17:32:29 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: Started - Started container training-operator 2026/07/05 17:35:38 17:32:29 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-b67847846 from 0 to 1 2026/07/05 17:35:38 17:32:29 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: Created - Created container: manager 2026/07/05 17:35:38 17:32:29 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Killing - Stopping container manager 2026/07/05 17:35:38 17:32:29 INFO Pod/model-registry-operator-controller-manager-b67847846-st27k: Killing - Stopping container manager 2026/07/05 17:35:38 17:32:29 INFO ReplicaSet/model-registry-operator-controller-manager-b67847846: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-b67847846-7vq2t 2026/07/05 17:35:38 17:32:29 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: Started - Started container manager 2026/07/05 17:35:38 17:32:29 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: Created - Created container: manager 2026/07/05 17:35:38 17:32:29 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: Created - Created container: training-operator 2026/07/05 17:35:38 17:32:29 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: Started - Started container manager 2026/07/05 17:35:38 17:32:28 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:32:28 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:32:28 INFO Pod/maas-api-557679ff4c-rn27g: Scheduled - Successfully assigned redhat-ods-applications/maas-api-557679ff4c-rn27g to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:38 17:32:28 INFO ReplicaSet/maas-api-557679ff4c: SuccessfulCreate - Created pod: maas-api-557679ff4c-rn27g 2026/07/05 17:35:38 17:32:27 INFO Pod/dashboard-redirect-b564449f8-vk6vt: Pulled - Container image "registry.redhat.io/ubi9/nginx-126@sha256:f0a79ccf21b8780a7534d78ee0e49d1852654f5f0f52e17ebdbe185a71d93253" already present on machine 2026/07/05 17:35:38 17:32:27 INFO Pod/dashboard-redirect-b564449f8-vk6vt: AddedInterface - Add eth0 [10.134.0.62/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:27 INFO Pod/dashboard-redirect-b564449f8-vk6vt: Created - Created container: nginx 2026/07/05 17:35:38 17:32:27 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-557679ff4c from 0 to 1 2026/07/05 17:35:38 17:32:27 INFO Pod/dashboard-redirect-b564449f8-vk6vt: Started - Started container nginx 2026/07/05 17:35:38 17:32:27 WARN ReplicaSet/maas-api-557679ff4c: FailedCreate - Error creating: pods "maas-api-557679ff4c-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/05 17:35:38 17:32:26 INFO Pod/dashboard-redirect-b564449f8-vk6vt: Scheduled - Successfully assigned redhat-ods-applications/dashboard-redirect-b564449f8-vk6vt to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:32:26 INFO Pod/dashboard-redirect-b564449f8-jxpdv: Scheduled - Successfully assigned redhat-ods-applications/dashboard-redirect-b564449f8-jxpdv to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:38 17:32:26 INFO ReplicaSet/dashboard-redirect-b564449f8: SuccessfulCreate - Created pod: dashboard-redirect-b564449f8-vk6vt 2026/07/05 17:35:38 17:32:26 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Pulled - Successfully pulled image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" in 1.459s (1.459s including waiting). Image size: 168349711 bytes. 2026/07/05 17:35:38 17:32:26 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Created - Created container: manager 2026/07/05 17:35:38 17:32:26 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Started - Started container manager 2026/07/05 17:35:38 17:32:26 INFO Deployment/dashboard-redirect: ScalingReplicaSet - Scaled up replica set dashboard-redirect-b564449f8 from 0 to 2 2026/07/05 17:35:38 17:32:26 INFO ReplicaSet/dashboard-redirect-b564449f8: SuccessfulCreate - Created pod: dashboard-redirect-b564449f8-jxpdv 2026/07/05 17:35:38 17:32:25 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-training-operator-7c59bdd544-t6h2g to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:38 17:32:25 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" in 1.628s (1.628s including waiting). Image size: 236362435 bytes. 2026/07/05 17:35:38 17:32:25 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Pulling - Pulling image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" 2026/07/05 17:35:38 17:32:25 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: Pulling - Pulling image "quay.io/rhoai/odh-training-operator-rhel9@sha256:752923f0362c0070dfdcdfb6fc0a10577cdda34007a88375c00a602ba96f2b66" 2026/07/05 17:35:38 17:32:25 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: AddedInterface - Add eth0 [10.132.0.42/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:25 INFO Deployment/kubeflow-training-operator: ScalingReplicaSet - Scaled up replica set kubeflow-training-operator-7c59bdd544 from 1 to 2 2026/07/05 17:35:38 17:32:25 INFO ReplicaSet/kubeflow-training-operator-7c59bdd544: SuccessfulCreate - Created pod: kubeflow-training-operator-7c59bdd544-t6h2g 2026/07/05 17:35:38 17:32:25 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: Pulling - Pulling image "quay.io/rhoai/odh-feast-operator-rhel9@sha256:fe1b375defb9ab9e79f523fd63103931941bfa0d7ff530ddd0d0db6fce412c8d" 2026/07/05 17:35:38 17:32:25 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: Pulling - Pulling image "quay.io/rhoai/odh-trainer-rhel9@sha256:27961006c645f0800ecab6ec75c48d98de4aa1ea1d65b37652056f3cd5684ac5" 2026/07/05 17:35:38 17:32:25 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: AddedInterface - Add eth0 [10.134.0.61/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:25 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: Created - Created container: manager 2026/07/05 17:35:38 17:32:25 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: AddedInterface - Add eth0 [10.132.0.41/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:25 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: AddedInterface - Add eth0 [10.132.0.43/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:25 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: Started - Started container manager 2026/07/05 17:35:38 17:32:24 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: Scheduled - Successfully assigned redhat-ods-applications/feast-operator-controller-manager-7b78bd74b9-hn47x to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:38 17:32:24 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Scheduled - Successfully assigned redhat-ods-applications/notebook-controller-deployment-6b68db7cbd-zzfr7 to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:32:24 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:38 17:32:24 INFO ReplicaSet/kubeflow-trainer-controller-manager-8648b6bb44: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-8648b6bb44-bmxvv 2026/07/05 17:35:38 17:32:24 INFO Deployment/notebook-controller-deployment: ScalingReplicaSet - Scaled up replica set notebook-controller-deployment-6b68db7cbd from 1 to 2 2026/07/05 17:35:38 17:32:24 INFO ReplicaSet/notebook-controller-deployment-6b68db7cbd: SuccessfulCreate - Created pod: notebook-controller-deployment-6b68db7cbd-zzfr7 2026/07/05 17:35:38 17:32:24 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" 2026/07/05 17:35:38 17:32:24 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: AddedInterface - Add eth0 [10.132.0.40/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:24 INFO ReplicaSet/feast-operator-controller-manager-7b78bd74b9: SuccessfulCreate - Created pod: feast-operator-controller-manager-7b78bd74b9-hn47x 2026/07/05 17:35:38 17:32:24 INFO Deployment/feast-operator-controller-manager: ScalingReplicaSet - Scaled up replica set feast-operator-controller-manager-7b78bd74b9 from 1 to 2 2026/07/05 17:35:38 17:32:24 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-8648b6bb44 from 1 to 2 2026/07/05 17:35:38 17:32:23 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-cr5pl to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:38 17:32:23 INFO ReplicaSet/llmisvc-controller-manager-bc44d7c7b: SuccessfulCreate - Created pod: llmisvc-controller-manager-bc44d7c7b-cr5pl 2026/07/05 17:35:38 17:32:23 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-bc44d7c7b from 1 to 2 2026/07/05 17:35:38 17:32:22 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Created - Created container: manager 2026/07/05 17:35:38 17:32:22 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" in 2.207s (2.207s including waiting). Image size: 236362435 bytes. 2026/07/05 17:35:38 17:32:22 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Started - Started container manager 2026/07/05 17:35:38 17:32:21 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: Created - Created container: manager 2026/07/05 17:35:38 17:32:21 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: Started - Started container manager 2026/07/05 17:35:38 17:32:20 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-p794f to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:32:20 INFO ReplicaSet/llmisvc-controller-manager-bc44d7c7b: SuccessfulCreate - Created pod: llmisvc-controller-manager-bc44d7c7b-p794f 2026/07/05 17:35:38 17:32:20 INFO Pod/dashboard-redirect-b564449f8-hhjwd: Killing - Stopping container nginx 2026/07/05 17:35:38 17:32:20 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" 2026/07/05 17:35:38 17:32:20 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: AddedInterface - Add eth0 [10.134.0.60/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:20 INFO Pod/dashboard-redirect-b564449f8-26p8f: Killing - Stopping container nginx 2026/07/05 17:35:38 17:32:20 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" in 374ms (374ms including waiting). Image size: 238427353 bytes. 2026/07/05 17:35:38 17:32:20 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" 2026/07/05 17:35:38 17:32:20 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: AddedInterface - Add eth0 [10.134.0.59/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:19 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-588b4b787c-gk7pm to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:32:19 INFO ReplicaSet/kserve-controller-manager-588b4b787c: SuccessfulCreate - Created pod: kserve-controller-manager-588b4b787c-gk7pm 2026/07/05 17:35:38 17:32:19 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-588b4b787c from 0 to 1 2026/07/05 17:35:38 17:32:19 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-bc44d7c7b from 0 to 1 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container autorag-ui 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container autorag-ui 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container model-registry-ui 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container kube-rbac-proxy 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container eval-hub-ui 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container automl-ui 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container automl-ui 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container rhods-dashboard 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container eval-hub-ui 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container mlflow-ui 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container mlflow-ui 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container gen-ai-ui 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container model-registry-ui 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container kube-rbac-proxy 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container maas-ui 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container gen-ai-ui 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container maas-ui 2026/07/05 17:35:38 17:32:18 INFO Lease/66483dbd.feast.dev: LeaderElection - feast-operator-controller-manager-7b78bd74b9-zv5kt_e58adde2-b1a6-4b05-8af0-fe2ec487a921 became leader 2026/07/05 17:35:38 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container rhods-dashboard 2026/07/05 17:35:38 17:32:15 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-8648b6bb44-zfg9l_a10b4c02-6c38-4e68-86bd-ae5cfcbc32e8 became leader 2026/07/05 17:35:38 17:32:14 INFO Pod/ai-gateway-operator-64fcd4bb69-csb97: Created - Created container: copy-manifests 2026/07/05 17:35:38 17:32:14 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: Started - Started container manager 2026/07/05 17:35:38 17:32:14 INFO Pod/ai-gateway-operator-64fcd4bb69-csb97: Pulled - Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-operator:odh-stable" in 11.043s (11.043s including waiting). Image size: 204996419 bytes. 2026/07/05 17:35:38 17:32:14 INFO Pod/ai-gateway-operator-64fcd4bb69-csb97: Started - Started container copy-manifests 2026/07/05 17:35:38 17:32:14 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: Created - Created container: manager 2026/07/05 17:35:38 17:32:12 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: Pulled - Successfully pulled image "quay.io/rhoai/odh-data-science-pipelines-operator-controller-rhel9@sha256:d9eaf8dab3521c7b97f1ed5d48481a4306122bb7bc4b751a965ebc9479dc2d93" in 10.867s (10.867s including waiting). Image size: 176639426 bytes. 2026/07/05 17:35:38 17:32:05 WARN Pod/ai-gateway-operator-f68c76b74-dd7cn: Unhealthy - Readiness probe failed: Get "http://10.133.0.93:8081/readyz": dial tcp 10.133.0.93:8081: connect: connection refused 2026/07/05 17:35:38 17:32:04 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Killing - Stopping container manager 2026/07/05 17:35:38 17:32:03 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Started - Started container manager 2026/07/05 17:35:38 17:32:03 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Created - Created container: manager 2026/07/05 17:35:38 17:32:03 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Pulled - Successfully pulled image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" in 2.94s (2.94s including waiting). Image size: 173525457 bytes. 2026/07/05 17:35:38 17:32:03 INFO Pod/ai-gateway-operator-64fcd4bb69-csb97: Pulling - Pulling image "quay.io/opendatahub/odh-ai-gateway-operator:odh-stable" 2026/07/05 17:35:38 17:32:02 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: Created - Created container: manager 2026/07/05 17:35:38 17:32:02 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: Started - Started container manager 2026/07/05 17:35:38 17:32:02 INFO Job/selector-migration: Completed - Job completed 2026/07/05 17:35:38 17:32:02 INFO Pod/ai-gateway-operator-64fcd4bb69-csb97: AddedInterface - Add eth0 [10.133.0.109/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:02 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: Pulled - Successfully pulled image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" in 1.768s (1.768s including waiting). Image size: 168349711 bytes. 2026/07/05 17:35:38 17:32:01 INFO Pod/ai-gateway-operator-64fcd4bb69-csb97: Scheduled - Successfully assigned redhat-ods-applications/ai-gateway-operator-64fcd4bb69-csb97 to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:32:01 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: Pulling - Pulling image "quay.io/rhoai/odh-data-science-pipelines-operator-controller-rhel9@sha256:d9eaf8dab3521c7b97f1ed5d48481a4306122bb7bc4b751a965ebc9479dc2d93" 2026/07/05 17:35:38 17:32:01 INFO Deployment/ai-gateway-operator: ScalingReplicaSet - Scaled up replica set ai-gateway-operator-64fcd4bb69 from 0 to 1 2026/07/05 17:35:38 17:32:01 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: AddedInterface - Add eth0 [10.133.0.106/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:01 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: Pulling - Pulling image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" 2026/07/05 17:35:38 17:32:01 INFO ReplicaSet/ai-gateway-operator-64fcd4bb69: SuccessfulCreate - Created pod: ai-gateway-operator-64fcd4bb69-csb97 2026/07/05 17:35:38 17:32:00 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: Scheduled - Successfully assigned redhat-ods-applications/notebook-controller-deployment-6b68db7cbd-6wpjx to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:38 17:32:00 INFO ReplicaSet/notebook-controller-deployment-6b68db7cbd: SuccessfulCreate - Created pod: notebook-controller-deployment-6b68db7cbd-6wpjx 2026/07/05 17:35:38 17:32:00 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: AddedInterface - Add eth0 [10.133.0.104/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:00 INFO Pod/kubeflow-training-operator-7c59bdd544-r8lkq: AddedInterface - Add eth0 [10.133.0.105/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:00 INFO Deployment/notebook-controller-deployment: ScalingReplicaSet - Scaled up replica set notebook-controller-deployment-6b68db7cbd from 0 to 1 2026/07/05 17:35:38 17:32:00 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: AddedInterface - Add eth0 [10.132.0.39/23] from ovn-kubernetes 2026/07/05 17:35:38 17:32:00 INFO Pod/kubeflow-training-operator-7c59bdd544-r8lkq: Started - Started container training-operator 2026/07/05 17:35:38 17:32:00 INFO Pod/kubeflow-training-operator-7c59bdd544-r8lkq: Created - Created container: training-operator 2026/07/05 17:35:38 17:32:00 WARN ReplicaSet/notebook-controller-deployment-6b68db7cbd: FailedCreate - Error creating: pods "notebook-controller-deployment-6b68db7cbd-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 2026/07/05 17:35:38 17:32:00 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Pulling - Pulling image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" 2026/07/05 17:35:38 17:32:00 INFO Pod/kubeflow-training-operator-7c59bdd544-r8lkq: Pulled - Container image "quay.io/rhoai/odh-training-operator-rhel9@sha256:752923f0362c0070dfdcdfb6fc0a10577cdda34007a88375c00a602ba96f2b66" already present on machine 2026/07/05 17:35:38 17:31:59 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:59 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:59 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: Scheduled - Successfully assigned redhat-ods-applications/data-science-pipelines-operator-controller-manager-595ffd92phgg to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:59 WARN Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: FailedMount - MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 2026/07/05 17:35:38 17:31:59 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: Pulled - Successfully pulled image "quay.io/rhoai/odh-feast-operator-rhel9@sha256:fe1b375defb9ab9e79f523fd63103931941bfa0d7ff530ddd0d0db6fce412c8d" in 3.501s (3.501s including waiting). Image size: 185429423 bytes. 2026/07/05 17:35:38 17:31:59 INFO Deployment/data-science-pipelines-operator-controller-manager: ScalingReplicaSet - Scaled up replica set data-science-pipelines-operator-controller-manager-595ffd9458 from 0 to 1 2026/07/05 17:35:38 17:31:59 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: Created - Created container: manager 2026/07/05 17:35:38 17:31:59 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: Started - Started container manager 2026/07/05 17:35:38 17:31:59 INFO ReplicaSet/data-science-pipelines-operator-controller-manager-595ffd9458: SuccessfulCreate - Created pod: data-science-pipelines-operator-controller-manager-595ffd92phgg 2026/07/05 17:35:38 17:31:58 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:58 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:58 INFO Pod/kubeflow-training-operator-7c59bdd544-r8lkq: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-training-operator-7c59bdd544-r8lkq to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:58 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:31:58 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-566d8764f4-gmggg_ae5c89f4-1a6a-4fb5-a95b-dd524613ed05 became leader 2026/07/05 17:35:38 17:31:58 INFO ReplicaSet/kubeflow-training-operator-7c59bdd544: SuccessfulCreate - Created pod: kubeflow-training-operator-7c59bdd544-r8lkq 2026/07/05 17:35:38 17:31:58 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-8648b6bb44 from 0 to 1 2026/07/05 17:35:38 17:31:58 INFO ReplicaSet/kubeflow-trainer-controller-manager-8648b6bb44: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-8648b6bb44-zfg9l 2026/07/05 17:35:38 17:31:58 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l: Created - Created container: manager 2026/07/05 17:35:38 17:31:58 INFO Deployment/kubeflow-training-operator: ScalingReplicaSet - Scaled up replica set kubeflow-training-operator-7c59bdd544 from 0 to 1 2026/07/05 17:35:38 17:31:58 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:27961006c645f0800ecab6ec75c48d98de4aa1ea1d65b37652056f3cd5684ac5" already present on machine 2026/07/05 17:35:38 17:31:58 WARN Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: FailedMount - MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 2026/07/05 17:35:38 17:31:58 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l: Started - Started container manager 2026/07/05 17:35:38 17:31:58 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l: AddedInterface - Add eth0 [10.134.0.57/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:57 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Scheduled - Successfully assigned redhat-ods-applications/odh-notebook-controller-manager-6f96b94f4c-kswrx to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:57 INFO Deployment/odh-notebook-controller-manager: ScalingReplicaSet - Scaled up replica set odh-notebook-controller-manager-6f96b94f4c from 0 to 1 2026/07/05 17:35:38 17:31:57 INFO ReplicaSet/odh-notebook-controller-manager-6f96b94f4c: SuccessfulCreate - Created pod: odh-notebook-controller-manager-6f96b94f4c-kswrx 2026/07/05 17:35:38 17:31:56 INFO Pod/selector-migration-v7sw6: Scheduled - Successfully assigned redhat-ods-applications/selector-migration-v7sw6 to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:56 INFO Pod/selector-migration-v7sw6: Started - Started container migrate 2026/07/05 17:35:38 17:31:56 INFO Job/selector-migration: SuccessfulCreate - Created pod: selector-migration-v7sw6 2026/07/05 17:35:38 17:31:56 INFO Pod/selector-migration-v7sw6: AddedInterface - Add eth0 [10.133.0.102/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:56 INFO Pod/selector-migration-v7sw6: Pulled - Container image "registry.redhat.io/openshift4/ose-cli@sha256:bc35a9fc663baf0d6493cc57e89e77a240a36c43cf38fb78d8e61d3b87cf5cc5" already present on machine 2026/07/05 17:35:38 17:31:56 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-5b858f66fc-xrp5h_91470823-f2fd-46f8-a3c9-0c8162123242 became leader 2026/07/05 17:35:38 17:31:56 INFO Pod/selector-migration-v7sw6: Created - Created container: migrate 2026/07/05 17:35:38 17:31:56 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: Pulling - Pulling image "quay.io/rhoai/odh-feast-operator-rhel9@sha256:fe1b375defb9ab9e79f523fd63103931941bfa0d7ff530ddd0d0db6fce412c8d" 2026/07/05 17:35:38 17:31:56 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: AddedInterface - Add eth0 [10.133.0.101/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:55 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: Scheduled - Successfully assigned redhat-ods-applications/feast-operator-controller-manager-7b78bd74b9-zv5kt to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:55 INFO Lease/54e06e98.ogx.io: LeaderElection - ogx-k8s-operator-controller-manager-94b957b46-57fdc_fdba6bf6-3748-4914-b7ad-62048d56cc40 became leader 2026/07/05 17:35:38 17:31:55 INFO ReplicaSet/feast-operator-controller-manager-7b78bd74b9: SuccessfulCreate - Created pod: feast-operator-controller-manager-7b78bd74b9-zv5kt 2026/07/05 17:35:38 17:31:55 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-b67847846-jjt62_6dedcfe4-2e7a-469c-ab16-194787f85493 became leader 2026/07/05 17:35:38 17:31:55 INFO Deployment/feast-operator-controller-manager: ScalingReplicaSet - Scaled up replica set feast-operator-controller-manager-7b78bd74b9 from 0 to 1 2026/07/05 17:35:38 17:31:54 INFO Pod/model-registry-operator-controller-manager-b67847846-st27k: Started - Started container manager 2026/07/05 17:35:38 17:31:54 INFO Pod/model-registry-operator-controller-manager-b67847846-st27k: Created - Created container: manager 2026/07/05 17:35:38 17:31:54 INFO Pod/model-registry-operator-controller-manager-b67847846-st27k: AddedInterface - Add eth0 [10.134.0.56/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:54 INFO Pod/model-registry-operator-controller-manager-b67847846-st27k: Pulled - Container image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:e1bd98206aee94e76215e2a9ea4987e9c6f4597bdba1358921496b275e91abc0" already present on machine 2026/07/05 17:35:38 17:31:53 INFO Pod/model-registry-operator-controller-manager-b67847846-st27k: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-b67847846-st27k to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:31:53 INFO ReplicaSet/model-registry-operator-controller-manager-b67847846: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-b67847846-st27k 2026/07/05 17:35:38 17:31:53 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-b67847846 from 1 to 2 2026/07/05 17:35:38 17:31:48 WARN Job/selector-migration: FailedCreate - Error creating: pods "selector-migration-" is forbidden: error looking up service account redhat-ods-applications/feast-operator-controller-manager: serviceaccount "feast-operator-controller-manager" not found 2026/07/05 17:35:38 17:31:48 WARN ReplicaSet/kserve-controller-manager-588b4b787c: FailedCreate - Error creating: pods "kserve-controller-manager-588b4b787c-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 2026/07/05 17:35:38 17:31:48 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-588b4b787c from 0 to 2 2026/07/05 17:35:38 17:31:43 WARN ReplicaSet/kserve-controller-manager-588b4b787c: FailedCreate - Error creating: pods "kserve-controller-manager-588b4b787c-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 2026/07/05 17:35:38 17:31:43 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-588b4b787c from 0 to 2 2026/07/05 17:35:38 17:31:42 INFO Pod/model-serving-api-5c897df776-4mkx7: Created - Created container: server 2026/07/05 17:35:38 17:31:42 INFO Pod/model-serving-api-5c897df776-4mkx7: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:c690422f9b5d3fc40fe327208b7af310035080730395bb9c4e393965dce7b53f" in 1.597s (1.597s including waiting). Image size: 172852455 bytes. 2026/07/05 17:35:38 17:31:42 INFO Pod/model-serving-api-5c897df776-4mkx7: Started - Started container server 2026/07/05 17:35:38 17:31:41 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:41 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:41 INFO Pod/kserve-controller-manager-588b4b787c-w4zf4: Created - Created container: manager 2026/07/05 17:35:38 17:31:41 INFO Pod/kserve-controller-manager-588b4b787c-w4zf4: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" in 2.116s (2.116s including waiting). Image size: 238427353 bytes. 2026/07/05 17:35:38 17:31:41 INFO Pod/kserve-controller-manager-588b4b787c-w4zf4: Started - Started container manager 2026/07/05 17:35:38 17:31:41 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9m5w6l: Killing - Stopping container manager 2026/07/05 17:35:38 17:31:40 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: Created - Created container: manager 2026/07/05 17:35:38 17:31:40 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: Killing - Stopping container manager 2026/07/05 17:35:38 17:31:40 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: Started - Started container manager 2026/07/05 17:35:38 17:31:40 INFO Pod/model-serving-api-5c897df776-4mkx7: Pulling - Pulling image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:c690422f9b5d3fc40fe327208b7af310035080730395bb9c4e393965dce7b53f" 2026/07/05 17:35:38 17:31:40 INFO Pod/model-serving-api-5c897df776-4mkx7: AddedInterface - Add eth0 [10.133.0.100/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:40 INFO Pod/odh-model-controller-566d8764f4-gmggg: Created - Created container: manager 2026/07/05 17:35:38 17:31:40 INFO Pod/odh-model-controller-566d8764f4-gmggg: Started - Started container manager 2026/07/05 17:35:38 17:31:39 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:39 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:39 INFO Pod/model-serving-api-5c897df776-4mkx7: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5c897df776-4mkx7 to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:39 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:e1bd98206aee94e76215e2a9ea4987e9c6f4597bdba1358921496b275e91abc0" in 1.993s (1.993s including waiting). Image size: 211352054 bytes. 2026/07/05 17:35:38 17:31:39 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: AddedInterface - Add eth0 [10.133.0.99/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:39 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: Created - Created container: manager 2026/07/05 17:35:38 17:31:39 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: Pulled - Successfully pulled image "quay.io/rhoai/odh-ogx-k8s-operator-rhel9@sha256:f7b82e87ac7ed8b10126a45828a11e8c3a77e0e867783b5ef49e896a5c37d810" in 421ms (421ms including waiting). Image size: 195397925 bytes. 2026/07/05 17:35:38 17:31:39 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: Pulling - Pulling image "quay.io/rhoai/odh-ogx-k8s-operator-rhel9@sha256:f7b82e87ac7ed8b10126a45828a11e8c3a77e0e867783b5ef49e896a5c37d810" 2026/07/05 17:35:38 17:31:39 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: AddedInterface - Add eth0 [10.133.0.95/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-8lkpf: AddedInterface - Add eth0 [10.133.0.98/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:39 INFO Pod/maas-controller-5b858f66fc-xrp5h: AddedInterface - Add eth0 [10.133.0.97/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:39 INFO Pod/maas-controller-5b858f66fc-xrp5h: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5ad153372250a40333b74ab5ef8a66b84d831b9c9ec8e9bf0c313802c85ffae1" 2026/07/05 17:35:38 17:31:39 INFO Pod/maas-controller-5b858f66fc-xrp5h: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5ad153372250a40333b74ab5ef8a66b84d831b9c9ec8e9bf0c313802c85ffae1" in 567ms (567ms including waiting). Image size: 212862053 bytes. 2026/07/05 17:35:38 17:31:39 INFO Pod/maas-controller-5b858f66fc-xrp5h: Created - Created container: manager 2026/07/05 17:35:38 17:31:39 INFO Pod/maas-controller-5b858f66fc-xrp5h: Started - Started container manager 2026/07/05 17:35:38 17:31:39 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Created - Created container: manager 2026/07/05 17:35:38 17:31:39 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Started - Started container manager 2026/07/05 17:35:38 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-8lkpf: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" 2026/07/05 17:35:38 17:31:39 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5c897df776 from 0 to 1 2026/07/05 17:35:38 17:31:39 WARN Pod/model-serving-api-5c897df776-4mkx7: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/05 17:35:38 17:31:39 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" 2026/07/05 17:35:38 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-8lkpf: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" in 309ms (309ms including waiting). Image size: 238427353 bytes. 2026/07/05 17:35:38 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-8lkpf: Created - Created container: manager 2026/07/05 17:35:38 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-8lkpf: Started - Started container manager 2026/07/05 17:35:38 17:31:39 INFO Pod/odh-model-controller-566d8764f4-gmggg: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" in 487ms (487ms including waiting). Image size: 238959021 bytes. 2026/07/05 17:35:38 17:31:39 INFO Pod/odh-model-controller-566d8764f4-gmggg: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" 2026/07/05 17:35:38 17:31:39 INFO Pod/odh-model-controller-566d8764f4-gmggg: AddedInterface - Add eth0 [10.133.0.96/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:39 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: Started - Started container manager 2026/07/05 17:35:38 17:31:39 WARN ReplicaSet/model-serving-api-5c897df776: FailedCreate - Error creating: pods "model-serving-api-5c897df776-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/05 17:35:38 17:31:39 INFO ReplicaSet/model-serving-api-5c897df776: SuccessfulCreate - Created pod: model-serving-api-5c897df776-4mkx7 2026/07/05 17:35:38 17:31:39 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" in 377ms (377ms including waiting). Image size: 236362435 bytes. 2026/07/05 17:35:38 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-w4zf4: AddedInterface - Add eth0 [10.134.0.55/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-w4zf4: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" 2026/07/05 17:35:38 17:31:38 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-tqjqw to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:38 INFO Pod/kserve-controller-manager-588b4b787c-w4zf4: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-588b4b787c-w4zf4 to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:38 17:31:38 INFO Pod/kserve-controller-manager-588b4b787c-8lkpf: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-588b4b787c-8lkpf to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:38 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:38 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:38 INFO Pod/maas-controller-5b858f66fc-xrp5h: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-5b858f66fc-xrp5h to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:38 INFO Pod/odh-model-controller-566d8764f4-gmggg: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-566d8764f4-gmggg to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:38 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-5b858f66fc from 0 to 1 2026/07/05 17:35:38 17:31:38 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-588b4b787c from 0 to 2 2026/07/05 17:35:38 17:31:38 INFO ReplicaSet/kserve-controller-manager-588b4b787c: SuccessfulCreate - Created pod: kserve-controller-manager-588b4b787c-8lkpf 2026/07/05 17:35:38 17:31:38 INFO ReplicaSet/llmisvc-controller-manager-bc44d7c7b: SuccessfulCreate - Created pod: llmisvc-controller-manager-bc44d7c7b-tqjqw 2026/07/05 17:35:38 17:31:38 INFO ReplicaSet/odh-model-controller-566d8764f4: SuccessfulCreate - Created pod: odh-model-controller-566d8764f4-gmggg 2026/07/05 17:35:38 17:31:38 WARN Pod/odh-model-controller-566d8764f4-gmggg: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/05 17:35:38 17:31:38 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-bc44d7c7b from 0 to 1 2026/07/05 17:35:38 17:31:38 INFO ReplicaSet/maas-controller-5b858f66fc: SuccessfulCreate - Created pod: maas-controller-5b858f66fc-xrp5h 2026/07/05 17:35:38 17:31:38 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-566d8764f4 from 0 to 1 2026/07/05 17:35:38 17:31:38 INFO ReplicaSet/kserve-controller-manager-588b4b787c: SuccessfulCreate - Created pod: kserve-controller-manager-588b4b787c-w4zf4 2026/07/05 17:35:38 17:31:37 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Pulling - Pulling image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:e1bd98206aee94e76215e2a9ea4987e9c6f4597bdba1358921496b275e91abc0" 2026/07/05 17:35:38 17:31:37 WARN Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 2026/07/05 17:35:38 17:31:36 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: Scheduled - Successfully assigned redhat-ods-applications/ogx-k8s-operator-controller-manager-94b957b46-57fdc to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:36 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-b67847846-jjt62 to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:36 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:36 INFO Deployment/ogx-k8s-operator-controller-manager: ScalingReplicaSet - Scaled up replica set ogx-k8s-operator-controller-manager-94b957b46 from 0 to 1 2026/07/05 17:35:38 17:31:36 INFO ReplicaSet/ogx-k8s-operator-controller-manager-94b957b46: SuccessfulCreate - Created pod: ogx-k8s-operator-controller-manager-94b957b46-57fdc 2026/07/05 17:35:38 17:31:36 INFO ReplicaSet/model-registry-operator-controller-manager-b67847846: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-b67847846-jjt62 2026/07/05 17:35:38 17:31:36 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-b67847846 from 0 to 1 2026/07/05 17:35:38 17:31:36 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: AddedInterface - Add eth0 [10.133.0.94/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:34 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Pulled - Successfully pulled image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" in 349ms (349ms including waiting). Image size: 205008753 bytes. 2026/07/05 17:35:38 17:31:34 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Created - Created container: manager 2026/07/05 17:35:38 17:31:34 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Started - Started container manager 2026/07/05 17:35:38 17:31:34 INFO Lease/ai-gateway-lock: LeaderElection - ai-gateway-operator-f68c76b74-dd7cn_0b026c3b-2f14-4ee1-bdd0-1f325834b9cc became leader 2026/07/05 17:35:38 17:31:33 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Created - Created container: copy-manifests 2026/07/05 17:35:38 17:31:33 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Pulling - Pulling image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" 2026/07/05 17:35:38 17:31:33 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Started - Started container copy-manifests 2026/07/05 17:35:38 17:31:32 INFO Pod/model-registry-operator-controller-manager-b67847846-kljq2: Killing - Stopping container manager 2026/07/05 17:35:38 17:31:32 INFO Pod/notebook-controller-deployment-6b68db7cbd-5wjj4: Killing - Stopping container manager 2026/07/05 17:35:38 17:31:32 INFO Pod/kubeflow-training-operator-7c59bdd544-cdgsb: Killing - Stopping container training-operator 2026/07/05 17:35:38 17:31:32 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-rlpwh: Killing - Stopping container manager 2026/07/05 17:35:38 17:31:32 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Pulled - Successfully pulled image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" in 3.678s (3.678s including waiting). Image size: 205008753 bytes. 2026/07/05 17:35:38 17:31:32 INFO Pod/maas-controller-5b858f66fc-mxjzp: Killing - Stopping container manager 2026/07/05 17:35:38 17:31:32 INFO Pod/maas-api-955664fbd-fv5z6: Killing - Stopping container maas-api 2026/07/05 17:35:38 17:31:32 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-hwdjl: Killing - Stopping container manager 2026/07/05 17:35:38 17:31:32 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-8ltwl: Killing - Stopping container manager 2026/07/05 17:35:38 17:31:31 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:31 INFO Pod/model-serving-api-5c897df776-gvttw: Killing - Stopping container server 2026/07/05 17:35:38 17:31:31 INFO Pod/kserve-controller-manager-588b4b787c-xpwqz: Killing - Stopping container manager 2026/07/05 17:35:38 17:31:31 INFO Pod/odh-model-controller-566d8764f4-zxdr4: Killing - Stopping container manager 2026/07/05 17:35:38 17:31:31 INFO Pod/llmisvc-controller-manager-bc44d7c7b-zffh2: Killing - Stopping container manager 2026/07/05 17:35:38 17:31:30 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:30 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:30 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:38 17:31:29 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Pulling - Pulling image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" 2026/07/05 17:35:38 17:31:29 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: AddedInterface - Add eth0 [10.133.0.93/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:29 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hsl6h: Killing - Stopping container manager 2026/07/05 17:35:38 17:31:28 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Scheduled - Successfully assigned redhat-ods-applications/ai-gateway-operator-f68c76b74-dd7cn to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:28 INFO Pod/selector-migration-sgh2h: Started - Started container migrate 2026/07/05 17:35:38 17:31:28 INFO Pod/selector-migration-sgh2h: Created - Created container: migrate 2026/07/05 17:35:38 17:31:28 INFO Pod/selector-migration-sgh2h: Pulled - Container image "registry.redhat.io/openshift4/ose-cli@sha256:bc35a9fc663baf0d6493cc57e89e77a240a36c43cf38fb78d8e61d3b87cf5cc5" already present on machine 2026/07/05 17:35:38 17:31:28 INFO Pod/selector-migration-sgh2h: AddedInterface - Add eth0 [10.133.0.92/23] from ovn-kubernetes 2026/07/05 17:35:38 17:31:28 INFO ReplicaSet/ai-gateway-operator-f68c76b74: SuccessfulCreate - Created pod: ai-gateway-operator-f68c76b74-dd7cn 2026/07/05 17:35:38 17:31:28 INFO Deployment/ai-gateway-operator: ScalingReplicaSet - Scaled up replica set ai-gateway-operator-f68c76b74 from 0 to 1 2026/07/05 17:35:38 17:31:27 INFO Pod/selector-migration-sgh2h: Scheduled - Successfully assigned redhat-ods-applications/selector-migration-sgh2h to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:38 17:31:27 INFO Job/selector-migration: SuccessfulCreate - Created pod: selector-migration-sgh2h 2026/07/05 17:35:38 === RESOURCE QUOTAS === 2026/07/05 17:35:38 No resource quotas found in monitored namespaces 2026/07/05 17:35:38 Classification for Deployment deletion recovery: infrastructure/probe-failure (code=1014, confidence=medium) 2026/07/05 17:35:38 Evidence: event Pod/kuberay-operator-74d759f546-g4x5g: Unhealthy - Readiness probe failed: Get "http://10.134.0.66:8080/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"probe-failure","error_code":1014,"evidence":["event Pod/kuberay-operator-74d759f546-g4x5g: Unhealthy - Readiness probe failed: Get \"http://10.134.0.66:8080/metrics\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"],"confidence":"medium"} 2026/07/05 17:35:38 Diagnostics complete for failed test: Deployment deletion recovery === RUN TestOdhOperator/components/group_1/kserve/Validate_resource_deletion_recovery/Deployment_deletion_recovery testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/kserve/Validate_resource_deletion_recovery/Deployment_deletion_recovery (140.18s) 2026/07/05 17:35:38 TEST FAILURE DETECTED: Validate resource deletion recovery 2026/07/05 17:35:38 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:Validate resource deletion recovery) === 2026/07/05 17:35:39 === CLUSTER STATE === 2026/07/05 17:35:39 Node: ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:35:39 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:35:39 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:35:39 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:35:39 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/05 17:35:39 Node: ip-10-0-137-37.ec2.internal 2026/07/05 17:35:39 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:35:39 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:35:39 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:35:39 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:35:39 Resources: 7500m CPU, 31236220Ki memory allocatable (of 8000m CPU, 32387196Ki memory capacity) 2026/07/05 17:35:39 Node: ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:35:39 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:35:39 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:35:39 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:35:39 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/05 17:35:39 === DEPLOYMENTS === 2026/07/05 17:35:39 Failed to collect deployment data: deployments not ready: redhat-ods-applications/llmisvc-controller-manager (0/1), redhat-ods-applications/rhods-dashboard (1/2) 2026/07/05 17:35:39 === PODS === 2026/07/05 17:35:39 Failed to collect pod data: redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-h9kc7: container manager not ready; redhat-ods-applications/rhods-dashboard-6c65779f4b-fmwmv: container automl-ui not ready; redhat-ods-applications/selector-migration-sgh2h: phase=Failed 2026/07/05 17:35:39 === OPERATOR STATUS === 2026/07/05 17:35:39 Operator deployment: 3/3 ready 2026/07/05 17:35:39 === DSCI STATUS === 2026/07/05 17:35:39 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/05 17:35:39 === DSC STATUS === 2026/07/05 17:35:39 Failed to collect DSC data: default-dsc unhealthy conditions: Ready=False; ComponentsReady=False; DashboardReady=False; KserveReady=False 2026/07/05 17:35:39 === RECENT EVENTS === 2026/07/05 17:35:39 17:35:31 INFO Pod/kuberay-operator-74d759f546-ljsv9: Killing - Stopping container kuberay-operator 2026/07/05 17:35:39 17:35:30 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkExecutorRunning - Executor spark-pi-8b56089f3359686c-exec-1 is running 2026/07/05 17:35:39 17:35:29 INFO Pod/spark-pi-8b56089f3359686c-exec-1: Started - Started container spark-kubernetes-executor 2026/07/05 17:35:39 17:35:29 INFO Pod/spark-pi-8b56089f3359686c-exec-1: AddedInterface - Add eth0 [10.133.0.121/23] from ovn-kubernetes 2026/07/05 17:35:39 17:35:29 INFO Pod/spark-pi-8b56089f3359686c-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/05 17:35:39 17:35:29 INFO Pod/spark-pi-8b56089f3359686c-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/05 17:35:39 17:35:28 INFO Pod/spark-pi-8b56089f3359686c-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-8b56089f3359686c-exec-1 to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:35:28 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-bc44d7c7b-h9kc7_3d60e20d-76d4-403f-839c-9b4c308da487 became leader 2026/07/05 17:35:39 17:35:28 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkExecutorPending - Executor spark-pi-8b56089f3359686c-exec-1 is pending 2026/07/05 17:35:39 17:35:28 INFO Lease/ray-operator-leader: LeaderElection - kuberay-operator-74d759f546-ljsv9_13210562-1da5-4cca-8090-9b58f50ca77f became leader 2026/07/05 17:35:39 17:35:19 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkDriverRunning - Driver spark-pi-d959b8e96nfs74re5dgg-driver is running 2026/07/05 17:35:39 17:35:19 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Started - Started container spark-kubernetes-driver 2026/07/05 17:35:39 17:35:19 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Created - Created container: spark-kubernetes-driver 2026/07/05 17:35:39 17:35:19 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Pulled - Successfully pulled image "quay.io/opendatahub/data-processing:Spark-v4.0.1" in 20.724s (20.724s including waiting). Image size: 1778928657 bytes. 2026/07/05 17:35:39 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" in 600ms (600ms including waiting). Image size: 177813676 bytes. 2026/07/05 17:35:39 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" in 490ms (490ms including waiting). Image size: 185088364 bytes. 2026/07/05 17:35:39 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: mlflow-ui 2026/07/05 17:35:39 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container mlflow-ui 2026/07/05 17:35:39 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" 2026/07/05 17:35:39 17:35:14 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" in 526ms (526ms including waiting). Image size: 187798114 bytes. 2026/07/05 17:35:39 17:35:14 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" 2026/07/05 17:35:39 17:35:14 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container maas-ui 2026/07/05 17:35:39 17:35:14 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: maas-ui 2026/07/05 17:35:39 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container gen-ai-ui 2026/07/05 17:35:39 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" in 529ms (529ms including waiting). Image size: 182434600 bytes. 2026/07/05 17:35:39 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: model-registry-ui 2026/07/05 17:35:39 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container model-registry-ui 2026/07/05 17:35:39 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" 2026/07/05 17:35:39 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" in 488ms (488ms including waiting). Image size: 241238855 bytes. 2026/07/05 17:35:39 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: gen-ai-ui 2026/07/05 17:35:39 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" 2026/07/05 17:35:39 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" in 616ms (616ms including waiting). Image size: 799818131 bytes. 2026/07/05 17:35:39 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container rhods-dashboard 2026/07/05 17:35:39 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: rhods-dashboard 2026/07/05 17:35:39 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" 2026/07/05 17:35:39 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container kube-rbac-proxy 2026/07/05 17:35:39 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: kube-rbac-proxy 2026/07/05 17:35:39 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:ac5537f459cd812a12e8f6ae13001150e29954913007b1790a44b15fb75f4926" already present on machine 2026/07/05 17:35:39 17:35:11 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" 2026/07/05 17:35:39 17:35:11 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: AddedInterface - Add eth0 [10.134.0.72/23] from ovn-kubernetes 2026/07/05 17:35:39 17:35:10 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-6c65779f4b-fmwmv to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:35:10 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Pulled - Successfully pulled image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" in 613ms (613ms including waiting). Image size: 173525457 bytes. 2026/07/05 17:35:39 17:35:10 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Created - Created container: manager 2026/07/05 17:35:39 17:35:10 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Started - Started container manager 2026/07/05 17:35:39 17:35:10 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" in 500ms (500ms including waiting). Image size: 236362435 bytes. 2026/07/05 17:35:39 17:35:10 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Started - Started container manager 2026/07/05 17:35:39 17:35:10 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Created - Created container: manager 2026/07/05 17:35:39 17:35:09 INFO Pod/notebook-controller-deployment-6b68db7cbd-dp87m: Pulling - Pulling image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" 2026/07/05 17:35:39 17:35:09 INFO Pod/kuberay-operator-74d759f546-ljsv9: Pulled - Container image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" already present on machine 2026/07/05 17:35:39 17:35:09 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" 2026/07/05 17:35:39 17:35:09 INFO Pod/kuberay-operator-74d759f546-ljsv9: Started - Started container kuberay-operator 2026/07/05 17:35:39 17:35:09 INFO Pod/notebook-controller-deployment-6b68db7cbd-dp87m: Pulled - Successfully pulled image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" in 469ms (469ms including waiting). Image size: 168349711 bytes. 2026/07/05 17:35:39 17:35:09 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Started - Started container controller 2026/07/05 17:35:39 17:35:09 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Created - Created container: controller 2026/07/05 17:35:39 17:35:09 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Pulled - Successfully pulled image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" in 1m23.925s (1m23.925s including waiting). Image size: 2736477781 bytes. 2026/07/05 17:35:39 17:35:09 INFO Pod/kuberay-operator-74d759f546-ljsv9: Created - Created container: kuberay-operator 2026/07/05 17:35:39 17:35:09 INFO Pod/notebook-controller-deployment-6b68db7cbd-dp87m: AddedInterface - Add eth0 [10.134.0.68/23] from ovn-kubernetes 2026/07/05 17:35:39 17:35:09 INFO Pod/kuberay-operator-74d759f546-ljsv9: AddedInterface - Add eth0 [10.134.0.71/23] from ovn-kubernetes 2026/07/05 17:35:39 17:35:09 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: AddedInterface - Add eth0 [10.134.0.70/23] from ovn-kubernetes 2026/07/05 17:35:39 17:35:09 WARN Pod/notebook-controller-deployment-6b68db7cbd-dp87m: Failed - Error: configmap "notebook-controller-config" not found 2026/07/05 17:35:39 17:35:09 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Pulling - Pulling image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" 2026/07/05 17:35:39 17:35:09 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: AddedInterface - Add eth0 [10.134.0.69/23] from ovn-kubernetes 2026/07/05 17:35:39 17:35:07 WARN Pod/kuberay-operator-74d759f546-g4x5g: Unhealthy - Readiness probe failed: Get "http://10.134.0.66:8080/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/05 17:35:39 17:34:59 WARN Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Unhealthy - Readiness probe failed: Get "http://10.134.0.61:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/05 17:35:39 17:34:59 WARN Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Unhealthy - Readiness probe failed: Get "http://10.134.0.60:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/05 17:35:39 17:34:58 WARN Pod/rhods-dashboard-6c65779f4b-fmwmv: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/07/05 17:35:39 17:34:58 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/05 17:35:39 17:34:58 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Created - Created container: webhook 2026/07/05 17:35:39 17:34:58 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Started - Started container webhook 2026/07/05 17:35:39 17:34:58 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: AddedInterface - Add eth0 [10.133.0.119/23] from ovn-kubernetes 2026/07/05 17:35:39 17:34:58 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Pulling - Pulling image "quay.io/opendatahub/data-processing:Spark-v4.0.1" 2026/07/05 17:35:39 17:34:58 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: AddedInterface - Add eth0 [10.133.0.120/23] from ovn-kubernetes 2026/07/05 17:35:39 17:34:56 WARN Pod/kuberay-operator-74d759f546-g4x5g: Unhealthy - Readiness probe failed: Get "http://10.134.0.66:8080/metrics": dial tcp 10.134.0.66:8080: connect: connection refused 2026/07/05 17:35:39 17:34:54 WARN Pod/kuberay-operator-74d759f546-llk9f: Unhealthy - Readiness probe failed: Get "http://10.133.0.118:8080/metrics": dial tcp 10.133.0.118:8080: connect: connection refused 2026/07/05 17:35:39 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" 2026/07/05 17:35:39 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" in 462ms (462ms including waiting). Image size: 185088364 bytes. 2026/07/05 17:35:39 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: mlflow-ui 2026/07/05 17:35:39 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" in 394ms (394ms including waiting). Image size: 177813676 bytes. 2026/07/05 17:35:39 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container mlflow-ui 2026/07/05 17:35:39 17:34:43 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-d959b8e96nfs74re5dgg-driver to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" 2026/07/05 17:35:39 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: maas-ui 2026/07/05 17:35:39 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" 2026/07/05 17:35:39 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container gen-ai-ui 2026/07/05 17:35:39 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: gen-ai-ui 2026/07/05 17:35:39 17:34:43 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkApplicationSubmitted - SparkApplication spark-pi-d959b8e96nfs74re5dgg was submitted successfully 2026/07/05 17:35:39 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" in 381ms (381ms including waiting). Image size: 241238855 bytes. 2026/07/05 17:35:39 17:34:43 WARN Pod/spark-pi-d959b8e96nfs74re5dgg-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-89627a9f3358b879-conf-map" not found 2026/07/05 17:35:39 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container model-registry-ui 2026/07/05 17:35:39 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: model-registry-ui 2026/07/05 17:35:39 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" 2026/07/05 17:35:39 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" in 366ms (366ms including waiting). Image size: 187798114 bytes. 2026/07/05 17:35:39 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container maas-ui 2026/07/05 17:35:39 17:34:43 WARN Pod/spark-pi-d959b8e96nfs74re5dgg-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d959b8e96nfs74re5dgg-3380d59f3358a115-driver-podspec-conf-map" not found 2026/07/05 17:35:39 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" 2026/07/05 17:35:39 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" in 447ms (447ms including waiting). Image size: 182434600 bytes. 2026/07/05 17:35:39 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container rhods-dashboard 2026/07/05 17:35:39 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: rhods-dashboard 2026/07/05 17:35:39 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" in 456ms (456ms including waiting). Image size: 799818131 bytes. 2026/07/05 17:35:39 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: kube-rbac-proxy 2026/07/05 17:35:39 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:ac5537f459cd812a12e8f6ae13001150e29954913007b1790a44b15fb75f4926" already present on machine 2026/07/05 17:35:39 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container kube-rbac-proxy 2026/07/05 17:35:39 17:34:41 WARN Pod/rhods-dashboard-6c65779f4b-fmwmv: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/07/05 17:35:39 17:34:41 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-6c65779f4b-lstrb to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:39 17:34:41 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: AddedInterface - Add eth0 [10.132.0.50/23] from ovn-kubernetes 2026/07/05 17:35:39 17:34:41 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" 2026/07/05 17:35:39 17:34:41 INFO ReplicaSet/rhods-dashboard-6c65779f4b: SuccessfulCreate - Created pod: rhods-dashboard-6c65779f4b-fmwmv 2026/07/05 17:35:39 17:34:41 INFO ReplicaSet/rhods-dashboard-6c65779f4b: SuccessfulCreate - Created pod: rhods-dashboard-6c65779f4b-lstrb 2026/07/05 17:35:39 17:34:40 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-6c65779f4b from 0 to 2 2026/07/05 17:35:39 17:34:36 INFO Pod/kuberay-operator-74d759f546-ljsv9: Scheduled - Successfully assigned redhat-ods-applications/kuberay-operator-74d759f546-ljsv9 to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:34:36 INFO Deployment/kuberay-operator: ScalingReplicaSet - Scaled up replica set kuberay-operator-74d759f546 from 0 to 1 2026/07/05 17:35:39 17:34:36 INFO ReplicaSet/kuberay-operator-74d759f546: SuccessfulCreate - Created pod: kuberay-operator-74d759f546-ljsv9 2026/07/05 17:35:39 17:34:35 INFO Pod/kuberay-operator-74d759f546-g4x5g: Killing - Stopping container kuberay-operator 2026/07/05 17:35:39 17:34:35 INFO Pod/kuberay-operator-74d759f546-llk9f: Killing - Stopping container kuberay-operator 2026/07/05 17:35:39 17:34:06 WARN Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Unhealthy - Readiness probe failed: Get "http://10.133.0.104:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/05 17:35:39 17:34:05 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-h9kc7 to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:34:05 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Scheduled - Successfully assigned redhat-ods-applications/odh-notebook-controller-manager-6f96b94f4c-7jxl6 to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:34:05 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-wx56c to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:34:05 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-wx56c 2026/07/05 17:35:39 17:34:05 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 1 to 2 2026/07/05 17:35:39 17:34:05 INFO ReplicaSet/llmisvc-controller-manager-bc44d7c7b: SuccessfulCreate - Created pod: llmisvc-controller-manager-bc44d7c7b-h9kc7 2026/07/05 17:35:39 17:34:05 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-bc44d7c7b from 0 to 1 2026/07/05 17:35:39 17:34:05 INFO Deployment/odh-notebook-controller-manager: ScalingReplicaSet - Scaled up replica set odh-notebook-controller-manager-6f96b94f4c from 0 to 1 2026/07/05 17:35:39 17:34:05 INFO ReplicaSet/odh-notebook-controller-manager-6f96b94f4c: SuccessfulCreate - Created pod: odh-notebook-controller-manager-6f96b94f4c-7jxl6 2026/07/05 17:35:39 17:34:04 WARN Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Unhealthy - Readiness probe failed: Get "http://10.134.0.60:8081/readyz": dial tcp 10.134.0.60:8081: connect: connection refused 2026/07/05 17:35:39 17:34:04 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: Killing - Stopping container manager 2026/07/05 17:35:39 17:34:04 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Killing - Stopping container manager 2026/07/05 17:35:39 17:34:03 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Killing - Stopping container manager 2026/07/05 17:35:39 17:34:03 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: Killing - Stopping container manager 2026/07/05 17:35:39 17:34:02 INFO Pod/kuberay-operator-74d759f546-llk9f: Pulled - Successfully pulled image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" in 1.407s (1.407s including waiting). Image size: 179205940 bytes. 2026/07/05 17:35:39 17:34:02 INFO Lease/ray-operator-leader: LeaderElection - kuberay-operator-74d759f546-g4x5g_34d876db-58de-4ab5-ab74-052534dfbdba became leader 2026/07/05 17:35:39 17:34:02 INFO Pod/kuberay-operator-74d759f546-llk9f: Created - Created container: kuberay-operator 2026/07/05 17:35:39 17:34:02 INFO Pod/kuberay-operator-74d759f546-llk9f: Started - Started container kuberay-operator 2026/07/05 17:35:39 17:34:01 INFO Pod/kuberay-operator-74d759f546-llk9f: AddedInterface - Add eth0 [10.133.0.118/23] from ovn-kubernetes 2026/07/05 17:35:39 17:34:01 INFO Pod/kuberay-operator-74d759f546-llk9f: Pulling - Pulling image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" 2026/07/05 17:35:39 17:34:00 INFO Pod/kuberay-operator-74d759f546-llk9f: Scheduled - Successfully assigned redhat-ods-applications/kuberay-operator-74d759f546-llk9f to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:34:00 INFO Pod/notebook-controller-deployment-6b68db7cbd-dp87m: Scheduled - Successfully assigned redhat-ods-applications/notebook-controller-deployment-6b68db7cbd-dp87m to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:34:00 INFO Deployment/notebook-controller-deployment: ScalingReplicaSet - Scaled up replica set notebook-controller-deployment-6b68db7cbd from 0 to 1 2026/07/05 17:35:39 17:34:00 INFO ReplicaSet/notebook-controller-deployment-6b68db7cbd: SuccessfulCreate - Created pod: notebook-controller-deployment-6b68db7cbd-dp87m 2026/07/05 17:35:39 17:34:00 INFO Deployment/kuberay-operator: ScalingReplicaSet - Scaled up replica set kuberay-operator-74d759f546 from 1 to 2 2026/07/05 17:35:39 17:34:00 INFO ReplicaSet/kuberay-operator-74d759f546: SuccessfulCreate - Created pod: kuberay-operator-74d759f546-llk9f 2026/07/05 17:35:39 17:33:59 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: Killing - Stopping container manager 2026/07/05 17:35:39 17:33:59 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Killing - Stopping container manager 2026/07/05 17:35:39 17:33:45 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-jdvzx to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:33:45 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Pulling - Pulling image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" 2026/07/05 17:35:39 17:33:45 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-jdvzx 2026/07/05 17:35:39 17:33:45 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 1 to 2 2026/07/05 17:35:39 17:33:45 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: AddedInterface - Add eth0 [10.134.0.67/23] from ovn-kubernetes 2026/07/05 17:35:39 17:33:44 INFO Pod/kuberay-operator-74d759f546-g4x5g: Created - Created container: kuberay-operator 2026/07/05 17:35:39 17:33:44 INFO Pod/kuberay-operator-74d759f546-g4x5g: Started - Started container kuberay-operator 2026/07/05 17:35:39 17:33:44 INFO Pod/kuberay-operator-74d759f546-g4x5g: Pulled - Successfully pulled image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" in 1.631s (1.631s including waiting). Image size: 179205940 bytes. 2026/07/05 17:35:39 17:33:42 INFO Pod/kuberay-operator-74d759f546-g4x5g: Pulling - Pulling image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" 2026/07/05 17:35:39 17:33:42 INFO Pod/kuberay-operator-74d759f546-g4x5g: AddedInterface - Add eth0 [10.134.0.66/23] from ovn-kubernetes 2026/07/05 17:35:39 17:33:41 INFO Pod/kuberay-operator-74d759f546-g4x5g: Scheduled - Successfully assigned redhat-ods-applications/kuberay-operator-74d759f546-g4x5g to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:33:41 WARN Pod/kuberay-operator-74d759f546-g4x5g: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 2026/07/05 17:35:39 17:33:41 INFO ReplicaSet/kuberay-operator-74d759f546: SuccessfulCreate - Created pod: kuberay-operator-74d759f546-g4x5g 2026/07/05 17:35:39 17:33:41 INFO Deployment/kuberay-operator: ScalingReplicaSet - Scaled up replica set kuberay-operator-74d759f546 from 0 to 1 2026/07/05 17:35:39 17:33:38 INFO Lease/spark-operator-webhook-lock: LeaderElection - spark-operator-webhook-5f8768dbfd-rgktb_b0ba836a-be37-4a30-a983-1f197f6c49e2 became leader 2026/07/05 17:35:39 17:33:38 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-588b4b787c-4nm25_e7ea5d10-a5b5-4cb5-b915-1e85d8cc1b75 became leader 2026/07/05 17:35:39 17:33:36 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-5dd7d48f87-s2gzm_70a06366-6c28-43ad-8a34-79526b6f18e7 became leader 2026/07/05 17:35:39 17:33:30 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5f7b7f7655-mf2jg_d4d6fa96-e67f-491c-ba63-6964158add9e became leader 2026/07/05 17:35:39 17:33:27 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:33:27 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:33:27 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd98nx4m: Killing - Stopping container manager 2026/07/05 17:35:39 17:33:27 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: Killing - Stopping container manager 2026/07/05 17:35:39 17:33:26 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:33:26 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:33:25 INFO ReplicaSet/rhods-dashboard-6c65779f4b: SuccessfulDelete - Deleted pod: rhods-dashboard-6c65779f4b-5zgbz 2026/07/05 17:35:39 17:33:25 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled down replica set rhods-dashboard-6c65779f4b from 2 to 1 2026/07/05 17:35:39 17:33:25 INFO Pod/kubeflow-training-operator-7c59bdd544-qns74: Killing - Stopping container training-operator 2026/07/05 17:35:39 17:33:24 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:33:24 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: AddedInterface - Add eth0 [10.134.0.65/23] from ovn-kubernetes 2026/07/05 17:35:39 17:33:22 INFO Pod/kuberay-operator-74d759f546-b82w5: Killing - Stopping container kuberay-operator 2026/07/05 17:35:39 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Started - Started container manager 2026/07/05 17:35:39 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Created - Created container: manager 2026/07/05 17:35:39 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" in 385ms (385ms including waiting). Image size: 238427353 bytes. 2026/07/05 17:35:39 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" 2026/07/05 17:35:39 17:33:21 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-588b4b787c-4nm25 to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:33:21 INFO ReplicaSet/kserve-controller-manager-588b4b787c: SuccessfulCreate - Created pod: kserve-controller-manager-588b4b787c-4nm25 2026/07/05 17:35:39 17:33:21 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-588b4b787c from 0 to 1 2026/07/05 17:35:39 17:33:19 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Created - Created container: controller 2026/07/05 17:35:39 17:33:19 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Started - Started container webhook 2026/07/05 17:35:39 17:33:19 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: AddedInterface - Add eth0 [10.133.0.116/23] from ovn-kubernetes 2026/07/05 17:35:39 17:33:19 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Created - Created container: webhook 2026/07/05 17:35:39 17:33:19 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/05 17:35:39 17:33:19 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Started - Started container controller 2026/07/05 17:35:39 17:33:19 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/05 17:35:39 17:33:19 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: AddedInterface - Add eth0 [10.133.0.117/23] from ovn-kubernetes 2026/07/05 17:35:39 17:33:19 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-8648b6bb44-l65kd_20a05650-424e-43df-a54f-c97a806596e4 became leader 2026/07/05 17:35:39 17:33:18 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:33:18 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:33:18 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-rgktb to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:33:18 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-s2gzm to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:33:18 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-rgktb 2026/07/05 17:35:39 17:33:18 WARN ReplicaSet/spark-operator-controller-5dd7d48f87: FailedCreate - Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 2026/07/05 17:35:39 17:33:18 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: Killing - Stopping container manager 2026/07/05 17:35:39 17:33:18 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-s2gzm 2026/07/05 17:35:39 17:33:18 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 0 to 1 2026/07/05 17:35:39 17:33:18 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 0 to 1 2026/07/05 17:35:39 17:33:13 INFO Pod/feast-operator-controller-manager-7b78bd74b9-cpccm: Killing - Stopping container manager 2026/07/05 17:35:39 17:33:12 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd98nx4m: Scheduled - Successfully assigned redhat-ods-applications/data-science-pipelines-operator-controller-manager-595ffd98nx4m to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:33:12 INFO Lease/66483dbd.feast.dev: LeaderElection - feast-operator-controller-manager-7b78bd74b9-cpccm_715eee42-83b7-4cd9-a524-f0188710b2f6 became leader 2026/07/05 17:35:39 17:33:12 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd98nx4m: AddedInterface - Add eth0 [10.133.0.115/23] from ovn-kubernetes 2026/07/05 17:35:39 17:33:12 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd98nx4m: Pulled - Container image "quay.io/rhoai/odh-data-science-pipelines-operator-controller-rhel9@sha256:d9eaf8dab3521c7b97f1ed5d48481a4306122bb7bc4b751a965ebc9479dc2d93" already present on machine 2026/07/05 17:35:39 17:33:12 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd98nx4m: Created - Created container: manager 2026/07/05 17:35:39 17:33:12 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd98nx4m: Started - Started container manager 2026/07/05 17:35:39 17:33:12 INFO ReplicaSet/data-science-pipelines-operator-controller-manager-595ffd9458: SuccessfulCreate - Created pod: data-science-pipelines-operator-controller-manager-595ffd98nx4m 2026/07/05 17:35:39 17:33:12 INFO Deployment/data-science-pipelines-operator-controller-manager: ScalingReplicaSet - Scaled up replica set data-science-pipelines-operator-controller-manager-595ffd9458 from 0 to 1 2026/07/05 17:35:39 17:33:11 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: Killing - Stopping container manager 2026/07/05 17:35:39 17:33:11 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: Killing - Stopping container manager 2026/07/05 17:35:39 17:33:10 INFO Pod/odh-model-controller-566d8764f4-gmggg: Killing - Stopping container manager 2026/07/05 17:35:39 17:33:10 INFO ReplicaSet/odh-model-controller-566d8764f4: SuccessfulDelete - Deleted pod: odh-model-controller-566d8764f4-gmggg 2026/07/05 17:35:39 17:33:10 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled down replica set odh-model-controller-566d8764f4 from 1 to 0 2026/07/05 17:35:39 17:33:07 INFO Pod/spark-operator-webhook-5f8768dbfd-nfws7: Killing - Stopping container webhook 2026/07/05 17:35:39 17:33:07 INFO Pod/spark-operator-controller-5dd7d48f87-bv2wn: Killing - Stopping container controller 2026/07/05 17:35:39 17:33:06 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:33:06 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:33:03 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: Started - Started container manager 2026/07/05 17:35:39 17:33:03 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: Created - Created container: manager 2026/07/05 17:35:39 17:33:03 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: Pulled - Successfully pulled image "quay.io/rhoai/odh-trainer-rhel9@sha256:27961006c645f0800ecab6ec75c48d98de4aa1ea1d65b37652056f3cd5684ac5" in 2.418s (2.418s including waiting). Image size: 157432247 bytes. 2026/07/05 17:35:39 17:33:01 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: Pulling - Pulling image "quay.io/rhoai/odh-trainer-rhel9@sha256:27961006c645f0800ecab6ec75c48d98de4aa1ea1d65b37652056f3cd5684ac5" 2026/07/05 17:35:39 17:33:01 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: AddedInterface - Add eth0 [10.133.0.114/23] from ovn-kubernetes 2026/07/05 17:35:39 17:33:00 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-8648b6bb44-l65kd to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:33:00 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-8648b6bb44 from 0 to 1 2026/07/05 17:35:39 17:33:00 INFO ReplicaSet/kubeflow-trainer-controller-manager-8648b6bb44: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-8648b6bb44-l65kd 2026/07/05 17:35:39 17:32:59 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: Killing - Stopping container manager 2026/07/05 17:35:39 17:32:59 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l: Killing - Stopping container manager 2026/07/05 17:35:39 17:32:58 INFO Pod/odh-model-controller-5f7b7f7655-mf2jg: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5f7b7f7655-mf2jg to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:32:58 INFO Pod/odh-model-controller-5f7b7f7655-mf2jg: AddedInterface - Add eth0 [10.133.0.113/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:58 INFO ReplicaSet/odh-model-controller-5f7b7f7655: SuccessfulCreate - Created pod: odh-model-controller-5f7b7f7655-mf2jg 2026/07/05 17:35:39 17:32:58 INFO Pod/odh-model-controller-5f7b7f7655-mf2jg: Created - Created container: manager 2026/07/05 17:35:39 17:32:58 INFO Pod/odh-model-controller-5f7b7f7655-mf2jg: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" in 344ms (344ms including waiting). Image size: 238959021 bytes. 2026/07/05 17:35:39 17:32:58 INFO Pod/odh-model-controller-5f7b7f7655-mf2jg: Started - Started container manager 2026/07/05 17:35:39 17:32:58 INFO Pod/odh-model-controller-5f7b7f7655-mf2jg: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" 2026/07/05 17:35:39 17:32:57 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: Killing - Stopping container manager 2026/07/05 17:35:39 17:32:57 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5f7b7f7655 from 0 to 1 2026/07/05 17:35:39 17:32:56 INFO Pod/kubeflow-training-operator-7c59bdd544-qns74: AddedInterface - Add eth0 [10.133.0.112/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:56 INFO Pod/feast-operator-controller-manager-7b78bd74b9-cpccm: Pulled - Container image "quay.io/rhoai/odh-feast-operator-rhel9@sha256:fe1b375defb9ab9e79f523fd63103931941bfa0d7ff530ddd0d0db6fce412c8d" already present on machine 2026/07/05 17:35:39 17:32:56 INFO Pod/feast-operator-controller-manager-7b78bd74b9-cpccm: AddedInterface - Add eth0 [10.133.0.111/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:56 INFO Pod/kubeflow-training-operator-7c59bdd544-qns74: Started - Started container training-operator 2026/07/05 17:35:39 17:32:56 INFO Pod/feast-operator-controller-manager-7b78bd74b9-cpccm: Created - Created container: manager 2026/07/05 17:35:39 17:32:56 INFO Pod/feast-operator-controller-manager-7b78bd74b9-cpccm: Started - Started container manager 2026/07/05 17:35:39 17:32:56 INFO Pod/kubeflow-training-operator-7c59bdd544-qns74: Pulled - Container image "quay.io/rhoai/odh-training-operator-rhel9@sha256:752923f0362c0070dfdcdfb6fc0a10577cdda34007a88375c00a602ba96f2b66" already present on machine 2026/07/05 17:35:39 17:32:56 INFO Pod/kubeflow-training-operator-7c59bdd544-qns74: Created - Created container: training-operator 2026/07/05 17:35:39 17:32:55 INFO Pod/kubeflow-training-operator-7c59bdd544-qns74: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-training-operator-7c59bdd544-qns74 to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:32:55 INFO Pod/feast-operator-controller-manager-7b78bd74b9-cpccm: Scheduled - Successfully assigned redhat-ods-applications/feast-operator-controller-manager-7b78bd74b9-cpccm to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:32:55 INFO ReplicaSet/feast-operator-controller-manager-7b78bd74b9: SuccessfulCreate - Created pod: feast-operator-controller-manager-7b78bd74b9-cpccm 2026/07/05 17:35:39 17:32:55 INFO Pod/kubeflow-training-operator-7c59bdd544-r8lkq: Killing - Stopping container training-operator 2026/07/05 17:35:39 17:32:55 INFO ReplicaSet/kubeflow-training-operator-7c59bdd544: SuccessfulCreate - Created pod: kubeflow-training-operator-7c59bdd544-qns74 2026/07/05 17:35:39 17:32:55 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: Killing - Stopping container training-operator 2026/07/05 17:35:39 17:32:55 INFO Deployment/feast-operator-controller-manager: ScalingReplicaSet - Scaled up replica set feast-operator-controller-manager-7b78bd74b9 from 0 to 1 2026/07/05 17:35:39 17:32:55 INFO Deployment/kubeflow-training-operator: ScalingReplicaSet - Scaled up replica set kubeflow-training-operator-7c59bdd544 from 0 to 1 2026/07/05 17:35:39 17:32:54 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: Killing - Stopping container manager 2026/07/05 17:35:39 17:32:54 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: Killing - Stopping container manager 2026/07/05 17:35:39 17:32:48 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-b67847846-7vq2t_9fd9b1d9-cc9f-403f-b680-73e37b839fde became leader 2026/07/05 17:35:39 17:32:46 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: Created - Created container: manager 2026/07/05 17:35:39 17:32:46 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: Started - Started container manager 2026/07/05 17:35:39 17:32:45 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: AddedInterface - Add eth0 [10.133.0.110/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:45 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: Pulling - Pulling image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" 2026/07/05 17:35:39 17:32:45 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: Pulled - Successfully pulled image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" in 536ms (536ms including waiting). Image size: 173525457 bytes. 2026/07/05 17:35:39 17:32:44 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: Scheduled - Successfully assigned redhat-ods-applications/odh-notebook-controller-manager-6f96b94f4c-tmjpf to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:32:44 INFO Deployment/odh-notebook-controller-manager: ScalingReplicaSet - Scaled up replica set odh-notebook-controller-manager-6f96b94f4c from 1 to 2 2026/07/05 17:35:39 17:32:44 INFO ReplicaSet/odh-notebook-controller-manager-6f96b94f4c: SuccessfulCreate - Created pod: odh-notebook-controller-manager-6f96b94f4c-tmjpf 2026/07/05 17:35:39 17:32:39 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" 2026/07/05 17:35:39 17:32:39 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" in 515ms (515ms including waiting). Image size: 177813676 bytes. 2026/07/05 17:35:39 17:32:39 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" in 507ms (507ms including waiting). Image size: 185088364 bytes. 2026/07/05 17:35:39 17:32:39 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Started - Started container mlflow-ui 2026/07/05 17:35:39 17:32:39 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Created - Created container: mlflow-ui 2026/07/05 17:35:39 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" in 382ms (382ms including waiting). Image size: 177813676 bytes. 2026/07/05 17:35:39 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" in 483ms (483ms including waiting). Image size: 187798114 bytes. 2026/07/05 17:35:39 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Created - Created container: maas-ui 2026/07/05 17:35:39 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Started - Started container maas-ui 2026/07/05 17:35:39 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" 2026/07/05 17:35:39 17:32:38 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-588b4b787c-gk7pm_4323bc22-edb8-450f-a82c-231f54f5def9 became leader 2026/07/05 17:35:39 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Created - Created container: mlflow-ui 2026/07/05 17:35:39 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" in 453ms (453ms including waiting). Image size: 185088364 bytes. 2026/07/05 17:35:39 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" 2026/07/05 17:35:39 17:32:38 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-bc44d7c7b-p794f_7c1236ca-7649-45ed-902d-f628ca1c3adf became leader 2026/07/05 17:35:39 17:32:38 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Started - Started container mlflow-ui 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Created - Created container: gen-ai-ui 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Started - Started container maas-ui 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Created - Created container: maas-ui 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" in 558ms (558ms including waiting). Image size: 187798114 bytes. 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Started - Started container gen-ai-ui 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Created - Created container: gen-ai-ui 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" in 428ms (428ms including waiting). Image size: 241238855 bytes. 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" in 428ms (428ms including waiting). Image size: 182434600 bytes. 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Created - Created container: model-registry-ui 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Started - Started container model-registry-ui 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" in 412ms (412ms including waiting). Image size: 241238855 bytes. 2026/07/05 17:35:39 17:32:37 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Started - Started container gen-ai-ui 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Started - Started container rhods-dashboard 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Created - Created container: rhods-dashboard 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Started - Started container kube-rbac-proxy 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Created - Created container: kube-rbac-proxy 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:ac5537f459cd812a12e8f6ae13001150e29954913007b1790a44b15fb75f4926" already present on machine 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Started - Started container rhods-dashboard 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Created - Created container: rhods-dashboard 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" in 451ms (451ms including waiting). Image size: 799818131 bytes. 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: AddedInterface - Add eth0 [10.134.0.64/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Started - Started container model-registry-ui 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Created - Created container: model-registry-ui 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" in 475ms (475ms including waiting). Image size: 182434600 bytes. 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Started - Started container kube-rbac-proxy 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Created - Created container: kube-rbac-proxy 2026/07/05 17:35:39 17:32:36 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:ac5537f459cd812a12e8f6ae13001150e29954913007b1790a44b15fb75f4926" already present on machine 2026/07/05 17:35:39 17:32:35 INFO Pod/rhods-dashboard-6c65779f4b-5zgbz: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-6c65779f4b-5zgbz to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:32:35 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-6c65779f4b-xcrkd to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:39 17:32:35 INFO ReplicaSet/rhods-dashboard-6c65779f4b: SuccessfulCreate - Created pod: rhods-dashboard-6c65779f4b-xcrkd 2026/07/05 17:35:39 17:32:35 INFO ReplicaSet/rhods-dashboard-6c65779f4b: SuccessfulCreate - Created pod: rhods-dashboard-6c65779f4b-5zgbz 2026/07/05 17:35:39 17:32:35 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" in 431ms (431ms including waiting). Image size: 799818131 bytes. 2026/07/05 17:35:39 17:32:35 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" 2026/07/05 17:35:39 17:32:35 INFO Pod/rhods-dashboard-6c65779f4b-xcrkd: AddedInterface - Add eth0 [10.132.0.49/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:34 WARN ReplicaSet/rhods-dashboard-6c65779f4b: FailedCreate - Error creating: pods "rhods-dashboard-6c65779f4b-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 2026/07/05 17:35:39 17:32:33 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: Created - Created container: manager 2026/07/05 17:35:39 17:32:33 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: Pulled - Successfully pulled image "quay.io/rhoai/odh-data-science-pipelines-operator-controller-rhel9@sha256:d9eaf8dab3521c7b97f1ed5d48481a4306122bb7bc4b751a965ebc9479dc2d93" in 1.832s (1.832s including waiting). Image size: 176639426 bytes. 2026/07/05 17:35:39 17:32:33 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-6c65779f4b from 0 to 2 2026/07/05 17:35:39 17:32:33 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: Started - Started container manager 2026/07/05 17:35:39 17:32:32 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: Started - Started container manager 2026/07/05 17:35:39 17:32:32 INFO Pod/maas-api-557679ff4c-rn27g: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:86e3f628c409b1afa2773d3e0ed3ab7f977a4da82d6576d4b9cd3b3ddbadb447" in 2.09s (2.09s including waiting). Image size: 175014370 bytes. 2026/07/05 17:35:39 17:32:32 INFO Pod/maas-api-557679ff4c-rn27g: Started - Started container maas-api 2026/07/05 17:35:39 17:32:32 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: Created - Created container: manager 2026/07/05 17:35:39 17:32:32 INFO Pod/maas-api-557679ff4c-rn27g: Created - Created container: maas-api 2026/07/05 17:35:39 17:32:32 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:e1bd98206aee94e76215e2a9ea4987e9c6f4597bdba1358921496b275e91abc0" in 2.079s (2.079s including waiting). Image size: 211352054 bytes. 2026/07/05 17:35:39 17:32:31 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: Scheduled - Successfully assigned redhat-ods-applications/data-science-pipelines-operator-controller-manager-595ffd9dtr6j to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:32:31 INFO Deployment/data-science-pipelines-operator-controller-manager: ScalingReplicaSet - Scaled up replica set data-science-pipelines-operator-controller-manager-595ffd9458 from 1 to 2 2026/07/05 17:35:39 17:32:31 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: AddedInterface - Add eth0 [10.134.0.63/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:31 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9dtr6j: Pulling - Pulling image "quay.io/rhoai/odh-data-science-pipelines-operator-controller-rhel9@sha256:d9eaf8dab3521c7b97f1ed5d48481a4306122bb7bc4b751a965ebc9479dc2d93" 2026/07/05 17:35:39 17:32:31 INFO ReplicaSet/data-science-pipelines-operator-controller-manager-595ffd9458: SuccessfulCreate - Created pod: data-science-pipelines-operator-controller-manager-595ffd9dtr6j 2026/07/05 17:35:39 17:32:30 INFO Pod/maas-api-557679ff4c-rn27g: AddedInterface - Add eth0 [10.132.0.45/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:30 INFO Pod/dashboard-redirect-b564449f8-jxpdv: Pulled - Container image "registry.redhat.io/ubi9/nginx-126@sha256:f0a79ccf21b8780a7534d78ee0e49d1852654f5f0f52e17ebdbe185a71d93253" already present on machine 2026/07/05 17:35:39 17:32:30 INFO Pod/dashboard-redirect-b564449f8-jxpdv: Started - Started container nginx 2026/07/05 17:35:39 17:32:30 INFO Pod/dashboard-redirect-b564449f8-jxpdv: AddedInterface - Add eth0 [10.132.0.44/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:30 INFO Pod/maas-api-557679ff4c-rn27g: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:86e3f628c409b1afa2773d3e0ed3ab7f977a4da82d6576d4b9cd3b3ddbadb447" 2026/07/05 17:35:39 17:32:30 INFO Pod/dashboard-redirect-b564449f8-jxpdv: Created - Created container: nginx 2026/07/05 17:35:39 17:32:30 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: Pulling - Pulling image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:e1bd98206aee94e76215e2a9ea4987e9c6f4597bdba1358921496b275e91abc0" 2026/07/05 17:35:39 17:32:30 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: AddedInterface - Add eth0 [10.132.0.48/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:29 INFO Pod/model-registry-operator-controller-manager-b67847846-7vq2t: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-b67847846-7vq2t to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:39 17:32:29 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: Pulled - Successfully pulled image "quay.io/rhoai/odh-trainer-rhel9@sha256:27961006c645f0800ecab6ec75c48d98de4aa1ea1d65b37652056f3cd5684ac5" in 4.405s (4.405s including waiting). Image size: 157432247 bytes. 2026/07/05 17:35:39 17:32:29 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: Pulled - Successfully pulled image "quay.io/rhoai/odh-training-operator-rhel9@sha256:752923f0362c0070dfdcdfb6fc0a10577cdda34007a88375c00a602ba96f2b66" in 3.732s (3.732s including waiting). Image size: 203250147 bytes. 2026/07/05 17:35:39 17:32:29 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: Pulled - Successfully pulled image "quay.io/rhoai/odh-feast-operator-rhel9@sha256:fe1b375defb9ab9e79f523fd63103931941bfa0d7ff530ddd0d0db6fce412c8d" in 3.758s (3.758s including waiting). Image size: 185429423 bytes. 2026/07/05 17:35:39 17:32:29 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: Started - Started container training-operator 2026/07/05 17:35:39 17:32:29 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-b67847846 from 0 to 1 2026/07/05 17:35:39 17:32:29 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: Created - Created container: manager 2026/07/05 17:35:39 17:32:29 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Killing - Stopping container manager 2026/07/05 17:35:39 17:32:29 INFO Pod/model-registry-operator-controller-manager-b67847846-st27k: Killing - Stopping container manager 2026/07/05 17:35:39 17:32:29 INFO ReplicaSet/model-registry-operator-controller-manager-b67847846: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-b67847846-7vq2t 2026/07/05 17:35:39 17:32:29 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: Started - Started container manager 2026/07/05 17:35:39 17:32:29 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: Created - Created container: manager 2026/07/05 17:35:39 17:32:29 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: Created - Created container: training-operator 2026/07/05 17:35:39 17:32:29 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: Started - Started container manager 2026/07/05 17:35:39 17:32:28 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:32:28 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:32:28 INFO Pod/maas-api-557679ff4c-rn27g: Scheduled - Successfully assigned redhat-ods-applications/maas-api-557679ff4c-rn27g to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:39 17:32:28 INFO ReplicaSet/maas-api-557679ff4c: SuccessfulCreate - Created pod: maas-api-557679ff4c-rn27g 2026/07/05 17:35:39 17:32:27 INFO Pod/dashboard-redirect-b564449f8-vk6vt: Pulled - Container image "registry.redhat.io/ubi9/nginx-126@sha256:f0a79ccf21b8780a7534d78ee0e49d1852654f5f0f52e17ebdbe185a71d93253" already present on machine 2026/07/05 17:35:39 17:32:27 INFO Pod/dashboard-redirect-b564449f8-vk6vt: AddedInterface - Add eth0 [10.134.0.62/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:27 INFO Pod/dashboard-redirect-b564449f8-vk6vt: Created - Created container: nginx 2026/07/05 17:35:39 17:32:27 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-557679ff4c from 0 to 1 2026/07/05 17:35:39 17:32:27 INFO Pod/dashboard-redirect-b564449f8-vk6vt: Started - Started container nginx 2026/07/05 17:35:39 17:32:27 WARN ReplicaSet/maas-api-557679ff4c: FailedCreate - Error creating: pods "maas-api-557679ff4c-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/05 17:35:39 17:32:26 INFO Pod/dashboard-redirect-b564449f8-vk6vt: Scheduled - Successfully assigned redhat-ods-applications/dashboard-redirect-b564449f8-vk6vt to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:32:26 INFO Pod/dashboard-redirect-b564449f8-jxpdv: Scheduled - Successfully assigned redhat-ods-applications/dashboard-redirect-b564449f8-jxpdv to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:39 17:32:26 INFO ReplicaSet/dashboard-redirect-b564449f8: SuccessfulCreate - Created pod: dashboard-redirect-b564449f8-vk6vt 2026/07/05 17:35:39 17:32:26 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Pulled - Successfully pulled image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" in 1.459s (1.459s including waiting). Image size: 168349711 bytes. 2026/07/05 17:35:39 17:32:26 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Created - Created container: manager 2026/07/05 17:35:39 17:32:26 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Started - Started container manager 2026/07/05 17:35:39 17:32:26 INFO Deployment/dashboard-redirect: ScalingReplicaSet - Scaled up replica set dashboard-redirect-b564449f8 from 0 to 2 2026/07/05 17:35:39 17:32:26 INFO ReplicaSet/dashboard-redirect-b564449f8: SuccessfulCreate - Created pod: dashboard-redirect-b564449f8-jxpdv 2026/07/05 17:35:39 17:32:25 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-training-operator-7c59bdd544-t6h2g to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:39 17:32:25 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" in 1.628s (1.628s including waiting). Image size: 236362435 bytes. 2026/07/05 17:35:39 17:32:25 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Pulling - Pulling image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" 2026/07/05 17:35:39 17:32:25 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: Pulling - Pulling image "quay.io/rhoai/odh-training-operator-rhel9@sha256:752923f0362c0070dfdcdfb6fc0a10577cdda34007a88375c00a602ba96f2b66" 2026/07/05 17:35:39 17:32:25 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: AddedInterface - Add eth0 [10.132.0.42/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:25 INFO Deployment/kubeflow-training-operator: ScalingReplicaSet - Scaled up replica set kubeflow-training-operator-7c59bdd544 from 1 to 2 2026/07/05 17:35:39 17:32:25 INFO ReplicaSet/kubeflow-training-operator-7c59bdd544: SuccessfulCreate - Created pod: kubeflow-training-operator-7c59bdd544-t6h2g 2026/07/05 17:35:39 17:32:25 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: Pulling - Pulling image "quay.io/rhoai/odh-feast-operator-rhel9@sha256:fe1b375defb9ab9e79f523fd63103931941bfa0d7ff530ddd0d0db6fce412c8d" 2026/07/05 17:35:39 17:32:25 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: Pulling - Pulling image "quay.io/rhoai/odh-trainer-rhel9@sha256:27961006c645f0800ecab6ec75c48d98de4aa1ea1d65b37652056f3cd5684ac5" 2026/07/05 17:35:39 17:32:25 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: AddedInterface - Add eth0 [10.134.0.61/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:25 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: Created - Created container: manager 2026/07/05 17:35:39 17:32:25 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: AddedInterface - Add eth0 [10.132.0.41/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:25 INFO Pod/kubeflow-training-operator-7c59bdd544-t6h2g: AddedInterface - Add eth0 [10.132.0.43/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:25 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: Started - Started container manager 2026/07/05 17:35:39 17:32:24 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hn47x: Scheduled - Successfully assigned redhat-ods-applications/feast-operator-controller-manager-7b78bd74b9-hn47x to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:39 17:32:24 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Scheduled - Successfully assigned redhat-ods-applications/notebook-controller-deployment-6b68db7cbd-zzfr7 to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:32:24 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-8648b6bb44-bmxvv to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:39 17:32:24 INFO ReplicaSet/kubeflow-trainer-controller-manager-8648b6bb44: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-8648b6bb44-bmxvv 2026/07/05 17:35:39 17:32:24 INFO Deployment/notebook-controller-deployment: ScalingReplicaSet - Scaled up replica set notebook-controller-deployment-6b68db7cbd from 1 to 2 2026/07/05 17:35:39 17:32:24 INFO ReplicaSet/notebook-controller-deployment-6b68db7cbd: SuccessfulCreate - Created pod: notebook-controller-deployment-6b68db7cbd-zzfr7 2026/07/05 17:35:39 17:32:24 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" 2026/07/05 17:35:39 17:32:24 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: AddedInterface - Add eth0 [10.132.0.40/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:24 INFO ReplicaSet/feast-operator-controller-manager-7b78bd74b9: SuccessfulCreate - Created pod: feast-operator-controller-manager-7b78bd74b9-hn47x 2026/07/05 17:35:39 17:32:24 INFO Deployment/feast-operator-controller-manager: ScalingReplicaSet - Scaled up replica set feast-operator-controller-manager-7b78bd74b9 from 1 to 2 2026/07/05 17:35:39 17:32:24 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-8648b6bb44 from 1 to 2 2026/07/05 17:35:39 17:32:23 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-cr5pl to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:39 17:32:23 INFO ReplicaSet/llmisvc-controller-manager-bc44d7c7b: SuccessfulCreate - Created pod: llmisvc-controller-manager-bc44d7c7b-cr5pl 2026/07/05 17:35:39 17:32:23 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-bc44d7c7b from 1 to 2 2026/07/05 17:35:39 17:32:22 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Created - Created container: manager 2026/07/05 17:35:39 17:32:22 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" in 2.207s (2.207s including waiting). Image size: 236362435 bytes. 2026/07/05 17:35:39 17:32:22 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Started - Started container manager 2026/07/05 17:35:39 17:32:21 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: Created - Created container: manager 2026/07/05 17:35:39 17:32:21 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: Started - Started container manager 2026/07/05 17:35:39 17:32:20 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-p794f to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:32:20 INFO ReplicaSet/llmisvc-controller-manager-bc44d7c7b: SuccessfulCreate - Created pod: llmisvc-controller-manager-bc44d7c7b-p794f 2026/07/05 17:35:39 17:32:20 INFO Pod/dashboard-redirect-b564449f8-hhjwd: Killing - Stopping container nginx 2026/07/05 17:35:39 17:32:20 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" 2026/07/05 17:35:39 17:32:20 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: AddedInterface - Add eth0 [10.134.0.60/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:20 INFO Pod/dashboard-redirect-b564449f8-26p8f: Killing - Stopping container nginx 2026/07/05 17:35:39 17:32:20 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" in 374ms (374ms including waiting). Image size: 238427353 bytes. 2026/07/05 17:35:39 17:32:20 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" 2026/07/05 17:35:39 17:32:20 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: AddedInterface - Add eth0 [10.134.0.59/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:19 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-588b4b787c-gk7pm to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:32:19 INFO ReplicaSet/kserve-controller-manager-588b4b787c: SuccessfulCreate - Created pod: kserve-controller-manager-588b4b787c-gk7pm 2026/07/05 17:35:39 17:32:19 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-588b4b787c from 0 to 1 2026/07/05 17:35:39 17:32:19 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-bc44d7c7b from 0 to 1 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container autorag-ui 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container autorag-ui 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container model-registry-ui 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container kube-rbac-proxy 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container eval-hub-ui 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container automl-ui 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container automl-ui 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container rhods-dashboard 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container eval-hub-ui 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container mlflow-ui 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container mlflow-ui 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container gen-ai-ui 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container model-registry-ui 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container kube-rbac-proxy 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container maas-ui 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-mrm65: Killing - Stopping container gen-ai-ui 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container maas-ui 2026/07/05 17:35:39 17:32:18 INFO Lease/66483dbd.feast.dev: LeaderElection - feast-operator-controller-manager-7b78bd74b9-zv5kt_e58adde2-b1a6-4b05-8af0-fe2ec487a921 became leader 2026/07/05 17:35:39 17:32:18 INFO Pod/rhods-dashboard-6c65779f4b-rnt2q: Killing - Stopping container rhods-dashboard 2026/07/05 17:35:39 17:32:15 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-8648b6bb44-zfg9l_a10b4c02-6c38-4e68-86bd-ae5cfcbc32e8 became leader 2026/07/05 17:35:39 17:32:14 INFO Pod/ai-gateway-operator-64fcd4bb69-csb97: Created - Created container: copy-manifests 2026/07/05 17:35:39 17:32:14 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: Started - Started container manager 2026/07/05 17:35:39 17:32:14 INFO Pod/ai-gateway-operator-64fcd4bb69-csb97: Pulled - Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-operator:odh-stable" in 11.043s (11.043s including waiting). Image size: 204996419 bytes. 2026/07/05 17:35:39 17:32:14 INFO Pod/ai-gateway-operator-64fcd4bb69-csb97: Started - Started container copy-manifests 2026/07/05 17:35:39 17:32:14 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: Created - Created container: manager 2026/07/05 17:35:39 17:32:12 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: Pulled - Successfully pulled image "quay.io/rhoai/odh-data-science-pipelines-operator-controller-rhel9@sha256:d9eaf8dab3521c7b97f1ed5d48481a4306122bb7bc4b751a965ebc9479dc2d93" in 10.867s (10.867s including waiting). Image size: 176639426 bytes. 2026/07/05 17:35:39 17:32:05 WARN Pod/ai-gateway-operator-f68c76b74-dd7cn: Unhealthy - Readiness probe failed: Get "http://10.133.0.93:8081/readyz": dial tcp 10.133.0.93:8081: connect: connection refused 2026/07/05 17:35:39 17:32:04 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Killing - Stopping container manager 2026/07/05 17:35:39 17:32:03 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Started - Started container manager 2026/07/05 17:35:39 17:32:03 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Created - Created container: manager 2026/07/05 17:35:39 17:32:03 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Pulled - Successfully pulled image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" in 2.94s (2.94s including waiting). Image size: 173525457 bytes. 2026/07/05 17:35:39 17:32:03 INFO Pod/ai-gateway-operator-64fcd4bb69-csb97: Pulling - Pulling image "quay.io/opendatahub/odh-ai-gateway-operator:odh-stable" 2026/07/05 17:35:39 17:32:02 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: Created - Created container: manager 2026/07/05 17:35:39 17:32:02 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: Started - Started container manager 2026/07/05 17:35:39 17:32:02 INFO Job/selector-migration: Completed - Job completed 2026/07/05 17:35:39 17:32:02 INFO Pod/ai-gateway-operator-64fcd4bb69-csb97: AddedInterface - Add eth0 [10.133.0.109/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:02 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: Pulled - Successfully pulled image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" in 1.768s (1.768s including waiting). Image size: 168349711 bytes. 2026/07/05 17:35:39 17:32:01 INFO Pod/ai-gateway-operator-64fcd4bb69-csb97: Scheduled - Successfully assigned redhat-ods-applications/ai-gateway-operator-64fcd4bb69-csb97 to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:32:01 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: Pulling - Pulling image "quay.io/rhoai/odh-data-science-pipelines-operator-controller-rhel9@sha256:d9eaf8dab3521c7b97f1ed5d48481a4306122bb7bc4b751a965ebc9479dc2d93" 2026/07/05 17:35:39 17:32:01 INFO Deployment/ai-gateway-operator: ScalingReplicaSet - Scaled up replica set ai-gateway-operator-64fcd4bb69 from 0 to 1 2026/07/05 17:35:39 17:32:01 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: AddedInterface - Add eth0 [10.133.0.106/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:01 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: Pulling - Pulling image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" 2026/07/05 17:35:39 17:32:01 INFO ReplicaSet/ai-gateway-operator-64fcd4bb69: SuccessfulCreate - Created pod: ai-gateway-operator-64fcd4bb69-csb97 2026/07/05 17:35:39 17:32:00 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: Scheduled - Successfully assigned redhat-ods-applications/notebook-controller-deployment-6b68db7cbd-6wpjx to ip-10-0-137-37.ec2.internal 2026/07/05 17:35:39 17:32:00 INFO ReplicaSet/notebook-controller-deployment-6b68db7cbd: SuccessfulCreate - Created pod: notebook-controller-deployment-6b68db7cbd-6wpjx 2026/07/05 17:35:39 17:32:00 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: AddedInterface - Add eth0 [10.133.0.104/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:00 INFO Pod/kubeflow-training-operator-7c59bdd544-r8lkq: AddedInterface - Add eth0 [10.133.0.105/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:00 INFO Deployment/notebook-controller-deployment: ScalingReplicaSet - Scaled up replica set notebook-controller-deployment-6b68db7cbd from 0 to 1 2026/07/05 17:35:39 17:32:00 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: AddedInterface - Add eth0 [10.132.0.39/23] from ovn-kubernetes 2026/07/05 17:35:39 17:32:00 INFO Pod/kubeflow-training-operator-7c59bdd544-r8lkq: Started - Started container training-operator 2026/07/05 17:35:39 17:32:00 INFO Pod/kubeflow-training-operator-7c59bdd544-r8lkq: Created - Created container: training-operator 2026/07/05 17:35:39 17:32:00 WARN ReplicaSet/notebook-controller-deployment-6b68db7cbd: FailedCreate - Error creating: pods "notebook-controller-deployment-6b68db7cbd-" is forbidden: error looking up service account redhat-ods-applications/notebook-controller-service-account: serviceaccount "notebook-controller-service-account" not found 2026/07/05 17:35:39 17:32:00 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Pulling - Pulling image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" 2026/07/05 17:35:39 17:32:00 INFO Pod/kubeflow-training-operator-7c59bdd544-r8lkq: Pulled - Container image "quay.io/rhoai/odh-training-operator-rhel9@sha256:752923f0362c0070dfdcdfb6fc0a10577cdda34007a88375c00a602ba96f2b66" already present on machine 2026/07/05 17:35:39 17:31:59 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:59 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:59 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: Scheduled - Successfully assigned redhat-ods-applications/data-science-pipelines-operator-controller-manager-595ffd92phgg to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:59 WARN Pod/data-science-pipelines-operator-controller-manager-595ffd92phgg: FailedMount - MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 2026/07/05 17:35:39 17:31:59 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: Pulled - Successfully pulled image "quay.io/rhoai/odh-feast-operator-rhel9@sha256:fe1b375defb9ab9e79f523fd63103931941bfa0d7ff530ddd0d0db6fce412c8d" in 3.501s (3.501s including waiting). Image size: 185429423 bytes. 2026/07/05 17:35:39 17:31:59 INFO Deployment/data-science-pipelines-operator-controller-manager: ScalingReplicaSet - Scaled up replica set data-science-pipelines-operator-controller-manager-595ffd9458 from 0 to 1 2026/07/05 17:35:39 17:31:59 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: Created - Created container: manager 2026/07/05 17:35:39 17:31:59 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: Started - Started container manager 2026/07/05 17:35:39 17:31:59 INFO ReplicaSet/data-science-pipelines-operator-controller-manager-595ffd9458: SuccessfulCreate - Created pod: data-science-pipelines-operator-controller-manager-595ffd92phgg 2026/07/05 17:35:39 17:31:58 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:58 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:58 INFO Pod/kubeflow-training-operator-7c59bdd544-r8lkq: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-training-operator-7c59bdd544-r8lkq to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:58 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:31:58 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-566d8764f4-gmggg_ae5c89f4-1a6a-4fb5-a95b-dd524613ed05 became leader 2026/07/05 17:35:39 17:31:58 INFO ReplicaSet/kubeflow-training-operator-7c59bdd544: SuccessfulCreate - Created pod: kubeflow-training-operator-7c59bdd544-r8lkq 2026/07/05 17:35:39 17:31:58 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-8648b6bb44 from 0 to 1 2026/07/05 17:35:39 17:31:58 INFO ReplicaSet/kubeflow-trainer-controller-manager-8648b6bb44: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-8648b6bb44-zfg9l 2026/07/05 17:35:39 17:31:58 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l: Created - Created container: manager 2026/07/05 17:35:39 17:31:58 INFO Deployment/kubeflow-training-operator: ScalingReplicaSet - Scaled up replica set kubeflow-training-operator-7c59bdd544 from 0 to 1 2026/07/05 17:35:39 17:31:58 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:27961006c645f0800ecab6ec75c48d98de4aa1ea1d65b37652056f3cd5684ac5" already present on machine 2026/07/05 17:35:39 17:31:58 WARN Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: FailedMount - MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 2026/07/05 17:35:39 17:31:58 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l: Started - Started container manager 2026/07/05 17:35:39 17:31:58 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-zfg9l: AddedInterface - Add eth0 [10.134.0.57/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:57 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Scheduled - Successfully assigned redhat-ods-applications/odh-notebook-controller-manager-6f96b94f4c-kswrx to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:57 INFO Deployment/odh-notebook-controller-manager: ScalingReplicaSet - Scaled up replica set odh-notebook-controller-manager-6f96b94f4c from 0 to 1 2026/07/05 17:35:39 17:31:57 INFO ReplicaSet/odh-notebook-controller-manager-6f96b94f4c: SuccessfulCreate - Created pod: odh-notebook-controller-manager-6f96b94f4c-kswrx 2026/07/05 17:35:39 17:31:56 INFO Pod/selector-migration-v7sw6: Scheduled - Successfully assigned redhat-ods-applications/selector-migration-v7sw6 to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:56 INFO Pod/selector-migration-v7sw6: Started - Started container migrate 2026/07/05 17:35:39 17:31:56 INFO Job/selector-migration: SuccessfulCreate - Created pod: selector-migration-v7sw6 2026/07/05 17:35:39 17:31:56 INFO Pod/selector-migration-v7sw6: AddedInterface - Add eth0 [10.133.0.102/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:56 INFO Pod/selector-migration-v7sw6: Pulled - Container image "registry.redhat.io/openshift4/ose-cli@sha256:bc35a9fc663baf0d6493cc57e89e77a240a36c43cf38fb78d8e61d3b87cf5cc5" already present on machine 2026/07/05 17:35:39 17:31:56 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-5b858f66fc-xrp5h_91470823-f2fd-46f8-a3c9-0c8162123242 became leader 2026/07/05 17:35:39 17:31:56 INFO Pod/selector-migration-v7sw6: Created - Created container: migrate 2026/07/05 17:35:39 17:31:56 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: Pulling - Pulling image "quay.io/rhoai/odh-feast-operator-rhel9@sha256:fe1b375defb9ab9e79f523fd63103931941bfa0d7ff530ddd0d0db6fce412c8d" 2026/07/05 17:35:39 17:31:56 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: AddedInterface - Add eth0 [10.133.0.101/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:55 INFO Pod/feast-operator-controller-manager-7b78bd74b9-zv5kt: Scheduled - Successfully assigned redhat-ods-applications/feast-operator-controller-manager-7b78bd74b9-zv5kt to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:55 INFO Lease/54e06e98.ogx.io: LeaderElection - ogx-k8s-operator-controller-manager-94b957b46-57fdc_fdba6bf6-3748-4914-b7ad-62048d56cc40 became leader 2026/07/05 17:35:39 17:31:55 INFO ReplicaSet/feast-operator-controller-manager-7b78bd74b9: SuccessfulCreate - Created pod: feast-operator-controller-manager-7b78bd74b9-zv5kt 2026/07/05 17:35:39 17:31:55 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-b67847846-jjt62_6dedcfe4-2e7a-469c-ab16-194787f85493 became leader 2026/07/05 17:35:39 17:31:55 INFO Deployment/feast-operator-controller-manager: ScalingReplicaSet - Scaled up replica set feast-operator-controller-manager-7b78bd74b9 from 0 to 1 2026/07/05 17:35:39 17:31:54 INFO Pod/model-registry-operator-controller-manager-b67847846-st27k: Started - Started container manager 2026/07/05 17:35:39 17:31:54 INFO Pod/model-registry-operator-controller-manager-b67847846-st27k: Created - Created container: manager 2026/07/05 17:35:39 17:31:54 INFO Pod/model-registry-operator-controller-manager-b67847846-st27k: AddedInterface - Add eth0 [10.134.0.56/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:54 INFO Pod/model-registry-operator-controller-manager-b67847846-st27k: Pulled - Container image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:e1bd98206aee94e76215e2a9ea4987e9c6f4597bdba1358921496b275e91abc0" already present on machine 2026/07/05 17:35:39 17:31:53 INFO Pod/model-registry-operator-controller-manager-b67847846-st27k: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-b67847846-st27k to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:31:53 INFO ReplicaSet/model-registry-operator-controller-manager-b67847846: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-b67847846-st27k 2026/07/05 17:35:39 17:31:53 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-b67847846 from 1 to 2 2026/07/05 17:35:39 17:31:48 WARN Job/selector-migration: FailedCreate - Error creating: pods "selector-migration-" is forbidden: error looking up service account redhat-ods-applications/feast-operator-controller-manager: serviceaccount "feast-operator-controller-manager" not found 2026/07/05 17:35:39 17:31:48 WARN ReplicaSet/kserve-controller-manager-588b4b787c: FailedCreate - Error creating: pods "kserve-controller-manager-588b4b787c-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 2026/07/05 17:35:39 17:31:48 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-588b4b787c from 0 to 2 2026/07/05 17:35:39 17:31:43 WARN ReplicaSet/kserve-controller-manager-588b4b787c: FailedCreate - Error creating: pods "kserve-controller-manager-588b4b787c-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 2026/07/05 17:35:39 17:31:43 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-588b4b787c from 0 to 2 2026/07/05 17:35:39 17:31:42 INFO Pod/model-serving-api-5c897df776-4mkx7: Created - Created container: server 2026/07/05 17:35:39 17:31:42 INFO Pod/model-serving-api-5c897df776-4mkx7: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:c690422f9b5d3fc40fe327208b7af310035080730395bb9c4e393965dce7b53f" in 1.597s (1.597s including waiting). Image size: 172852455 bytes. 2026/07/05 17:35:39 17:31:42 INFO Pod/model-serving-api-5c897df776-4mkx7: Started - Started container server 2026/07/05 17:35:39 17:31:41 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:41 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:41 INFO Pod/kserve-controller-manager-588b4b787c-w4zf4: Created - Created container: manager 2026/07/05 17:35:39 17:31:41 INFO Pod/kserve-controller-manager-588b4b787c-w4zf4: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" in 2.116s (2.116s including waiting). Image size: 238427353 bytes. 2026/07/05 17:35:39 17:31:41 INFO Pod/kserve-controller-manager-588b4b787c-w4zf4: Started - Started container manager 2026/07/05 17:35:39 17:31:41 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd9m5w6l: Killing - Stopping container manager 2026/07/05 17:35:39 17:31:40 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: Created - Created container: manager 2026/07/05 17:35:39 17:31:40 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: Killing - Stopping container manager 2026/07/05 17:35:39 17:31:40 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: Started - Started container manager 2026/07/05 17:35:39 17:31:40 INFO Pod/model-serving-api-5c897df776-4mkx7: Pulling - Pulling image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:c690422f9b5d3fc40fe327208b7af310035080730395bb9c4e393965dce7b53f" 2026/07/05 17:35:39 17:31:40 INFO Pod/model-serving-api-5c897df776-4mkx7: AddedInterface - Add eth0 [10.133.0.100/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:40 INFO Pod/odh-model-controller-566d8764f4-gmggg: Created - Created container: manager 2026/07/05 17:35:39 17:31:40 INFO Pod/odh-model-controller-566d8764f4-gmggg: Started - Started container manager 2026/07/05 17:35:39 17:31:39 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:39 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:39 INFO Pod/model-serving-api-5c897df776-4mkx7: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5c897df776-4mkx7 to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:39 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:e1bd98206aee94e76215e2a9ea4987e9c6f4597bdba1358921496b275e91abc0" in 1.993s (1.993s including waiting). Image size: 211352054 bytes. 2026/07/05 17:35:39 17:31:39 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: AddedInterface - Add eth0 [10.133.0.99/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:39 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: Created - Created container: manager 2026/07/05 17:35:39 17:31:39 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: Pulled - Successfully pulled image "quay.io/rhoai/odh-ogx-k8s-operator-rhel9@sha256:f7b82e87ac7ed8b10126a45828a11e8c3a77e0e867783b5ef49e896a5c37d810" in 421ms (421ms including waiting). Image size: 195397925 bytes. 2026/07/05 17:35:39 17:31:39 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: Pulling - Pulling image "quay.io/rhoai/odh-ogx-k8s-operator-rhel9@sha256:f7b82e87ac7ed8b10126a45828a11e8c3a77e0e867783b5ef49e896a5c37d810" 2026/07/05 17:35:39 17:31:39 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: AddedInterface - Add eth0 [10.133.0.95/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-8lkpf: AddedInterface - Add eth0 [10.133.0.98/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:39 INFO Pod/maas-controller-5b858f66fc-xrp5h: AddedInterface - Add eth0 [10.133.0.97/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:39 INFO Pod/maas-controller-5b858f66fc-xrp5h: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5ad153372250a40333b74ab5ef8a66b84d831b9c9ec8e9bf0c313802c85ffae1" 2026/07/05 17:35:39 17:31:39 INFO Pod/maas-controller-5b858f66fc-xrp5h: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5ad153372250a40333b74ab5ef8a66b84d831b9c9ec8e9bf0c313802c85ffae1" in 567ms (567ms including waiting). Image size: 212862053 bytes. 2026/07/05 17:35:39 17:31:39 INFO Pod/maas-controller-5b858f66fc-xrp5h: Created - Created container: manager 2026/07/05 17:35:39 17:31:39 INFO Pod/maas-controller-5b858f66fc-xrp5h: Started - Started container manager 2026/07/05 17:35:39 17:31:39 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Created - Created container: manager 2026/07/05 17:35:39 17:31:39 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Started - Started container manager 2026/07/05 17:35:39 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-8lkpf: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" 2026/07/05 17:35:39 17:31:39 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5c897df776 from 0 to 1 2026/07/05 17:35:39 17:31:39 WARN Pod/model-serving-api-5c897df776-4mkx7: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/05 17:35:39 17:31:39 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" 2026/07/05 17:35:39 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-8lkpf: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" in 309ms (309ms including waiting). Image size: 238427353 bytes. 2026/07/05 17:35:39 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-8lkpf: Created - Created container: manager 2026/07/05 17:35:39 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-8lkpf: Started - Started container manager 2026/07/05 17:35:39 17:31:39 INFO Pod/odh-model-controller-566d8764f4-gmggg: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" in 487ms (487ms including waiting). Image size: 238959021 bytes. 2026/07/05 17:35:39 17:31:39 INFO Pod/odh-model-controller-566d8764f4-gmggg: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" 2026/07/05 17:35:39 17:31:39 INFO Pod/odh-model-controller-566d8764f4-gmggg: AddedInterface - Add eth0 [10.133.0.96/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:39 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: Started - Started container manager 2026/07/05 17:35:39 17:31:39 WARN ReplicaSet/model-serving-api-5c897df776: FailedCreate - Error creating: pods "model-serving-api-5c897df776-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/05 17:35:39 17:31:39 INFO ReplicaSet/model-serving-api-5c897df776: SuccessfulCreate - Created pod: model-serving-api-5c897df776-4mkx7 2026/07/05 17:35:39 17:31:39 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" in 377ms (377ms including waiting). Image size: 236362435 bytes. 2026/07/05 17:35:39 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-w4zf4: AddedInterface - Add eth0 [10.134.0.55/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:39 INFO Pod/kserve-controller-manager-588b4b787c-w4zf4: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" 2026/07/05 17:35:39 17:31:38 INFO Pod/llmisvc-controller-manager-bc44d7c7b-tqjqw: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-tqjqw to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:38 INFO Pod/kserve-controller-manager-588b4b787c-w4zf4: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-588b4b787c-w4zf4 to ip-10-0-131-75.ec2.internal 2026/07/05 17:35:39 17:31:38 INFO Pod/kserve-controller-manager-588b4b787c-8lkpf: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-588b4b787c-8lkpf to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:38 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:38 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:38 INFO Pod/maas-controller-5b858f66fc-xrp5h: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-5b858f66fc-xrp5h to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:38 INFO Pod/odh-model-controller-566d8764f4-gmggg: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-566d8764f4-gmggg to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:38 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-5b858f66fc from 0 to 1 2026/07/05 17:35:39 17:31:38 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-588b4b787c from 0 to 2 2026/07/05 17:35:39 17:31:38 INFO ReplicaSet/kserve-controller-manager-588b4b787c: SuccessfulCreate - Created pod: kserve-controller-manager-588b4b787c-8lkpf 2026/07/05 17:35:39 17:31:38 INFO ReplicaSet/llmisvc-controller-manager-bc44d7c7b: SuccessfulCreate - Created pod: llmisvc-controller-manager-bc44d7c7b-tqjqw 2026/07/05 17:35:39 17:31:38 INFO ReplicaSet/odh-model-controller-566d8764f4: SuccessfulCreate - Created pod: odh-model-controller-566d8764f4-gmggg 2026/07/05 17:35:39 17:31:38 WARN Pod/odh-model-controller-566d8764f4-gmggg: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/05 17:35:39 17:31:38 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-bc44d7c7b from 0 to 1 2026/07/05 17:35:39 17:31:38 INFO ReplicaSet/maas-controller-5b858f66fc: SuccessfulCreate - Created pod: maas-controller-5b858f66fc-xrp5h 2026/07/05 17:35:39 17:31:38 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-566d8764f4 from 0 to 1 2026/07/05 17:35:39 17:31:38 INFO ReplicaSet/kserve-controller-manager-588b4b787c: SuccessfulCreate - Created pod: kserve-controller-manager-588b4b787c-w4zf4 2026/07/05 17:35:39 17:31:37 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Pulling - Pulling image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:e1bd98206aee94e76215e2a9ea4987e9c6f4597bdba1358921496b275e91abc0" 2026/07/05 17:35:39 17:31:37 WARN Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 2026/07/05 17:35:39 17:31:36 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: Scheduled - Successfully assigned redhat-ods-applications/ogx-k8s-operator-controller-manager-94b957b46-57fdc to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:36 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-b67847846-jjt62 to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:36 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:36 INFO Deployment/ogx-k8s-operator-controller-manager: ScalingReplicaSet - Scaled up replica set ogx-k8s-operator-controller-manager-94b957b46 from 0 to 1 2026/07/05 17:35:39 17:31:36 INFO ReplicaSet/ogx-k8s-operator-controller-manager-94b957b46: SuccessfulCreate - Created pod: ogx-k8s-operator-controller-manager-94b957b46-57fdc 2026/07/05 17:35:39 17:31:36 INFO ReplicaSet/model-registry-operator-controller-manager-b67847846: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-b67847846-jjt62 2026/07/05 17:35:39 17:31:36 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-b67847846 from 0 to 1 2026/07/05 17:35:39 17:31:36 INFO Pod/model-registry-operator-controller-manager-b67847846-jjt62: AddedInterface - Add eth0 [10.133.0.94/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:34 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Pulled - Successfully pulled image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" in 349ms (349ms including waiting). Image size: 205008753 bytes. 2026/07/05 17:35:39 17:31:34 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Created - Created container: manager 2026/07/05 17:35:39 17:31:34 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Started - Started container manager 2026/07/05 17:35:39 17:31:34 INFO Lease/ai-gateway-lock: LeaderElection - ai-gateway-operator-f68c76b74-dd7cn_0b026c3b-2f14-4ee1-bdd0-1f325834b9cc became leader 2026/07/05 17:35:39 17:31:33 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Created - Created container: copy-manifests 2026/07/05 17:35:39 17:31:33 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Pulling - Pulling image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" 2026/07/05 17:35:39 17:31:33 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Started - Started container copy-manifests 2026/07/05 17:35:39 17:31:32 INFO Pod/model-registry-operator-controller-manager-b67847846-kljq2: Killing - Stopping container manager 2026/07/05 17:35:39 17:31:32 INFO Pod/notebook-controller-deployment-6b68db7cbd-5wjj4: Killing - Stopping container manager 2026/07/05 17:35:39 17:31:32 INFO Pod/kubeflow-training-operator-7c59bdd544-cdgsb: Killing - Stopping container training-operator 2026/07/05 17:35:39 17:31:32 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-rlpwh: Killing - Stopping container manager 2026/07/05 17:35:39 17:31:32 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Pulled - Successfully pulled image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" in 3.678s (3.678s including waiting). Image size: 205008753 bytes. 2026/07/05 17:35:39 17:31:32 INFO Pod/maas-controller-5b858f66fc-mxjzp: Killing - Stopping container manager 2026/07/05 17:35:39 17:31:32 INFO Pod/maas-api-955664fbd-fv5z6: Killing - Stopping container maas-api 2026/07/05 17:35:39 17:31:32 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-hwdjl: Killing - Stopping container manager 2026/07/05 17:35:39 17:31:32 INFO Pod/ogx-k8s-operator-controller-manager-94b957b46-8ltwl: Killing - Stopping container manager 2026/07/05 17:35:39 17:31:31 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:31 INFO Pod/model-serving-api-5c897df776-gvttw: Killing - Stopping container server 2026/07/05 17:35:39 17:31:31 INFO Pod/kserve-controller-manager-588b4b787c-xpwqz: Killing - Stopping container manager 2026/07/05 17:35:39 17:31:31 INFO Pod/odh-model-controller-566d8764f4-zxdr4: Killing - Stopping container manager 2026/07/05 17:35:39 17:31:31 INFO Pod/llmisvc-controller-manager-bc44d7c7b-zffh2: Killing - Stopping container manager 2026/07/05 17:35:39 17:31:30 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:30 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:30 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:35:39 17:31:29 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Pulling - Pulling image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" 2026/07/05 17:35:39 17:31:29 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: AddedInterface - Add eth0 [10.133.0.93/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:29 INFO Pod/feast-operator-controller-manager-7b78bd74b9-hsl6h: Killing - Stopping container manager 2026/07/05 17:35:39 17:31:28 INFO Pod/ai-gateway-operator-f68c76b74-dd7cn: Scheduled - Successfully assigned redhat-ods-applications/ai-gateway-operator-f68c76b74-dd7cn to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:28 INFO Pod/selector-migration-sgh2h: Started - Started container migrate 2026/07/05 17:35:39 17:31:28 INFO Pod/selector-migration-sgh2h: Created - Created container: migrate 2026/07/05 17:35:39 17:31:28 INFO Pod/selector-migration-sgh2h: Pulled - Container image "registry.redhat.io/openshift4/ose-cli@sha256:bc35a9fc663baf0d6493cc57e89e77a240a36c43cf38fb78d8e61d3b87cf5cc5" already present on machine 2026/07/05 17:35:39 17:31:28 INFO Pod/selector-migration-sgh2h: AddedInterface - Add eth0 [10.133.0.92/23] from ovn-kubernetes 2026/07/05 17:35:39 17:31:28 INFO ReplicaSet/ai-gateway-operator-f68c76b74: SuccessfulCreate - Created pod: ai-gateway-operator-f68c76b74-dd7cn 2026/07/05 17:35:39 17:31:28 INFO Deployment/ai-gateway-operator: ScalingReplicaSet - Scaled up replica set ai-gateway-operator-f68c76b74 from 0 to 1 2026/07/05 17:35:39 17:31:27 INFO Pod/selector-migration-sgh2h: Scheduled - Successfully assigned redhat-ods-applications/selector-migration-sgh2h to ip-10-0-143-231.ec2.internal 2026/07/05 17:35:39 17:31:27 INFO Job/selector-migration: SuccessfulCreate - Created pod: selector-migration-sgh2h 2026/07/05 17:35:39 === RESOURCE QUOTAS === 2026/07/05 17:35:39 No resource quotas found in monitored namespaces 2026/07/05 17:35:39 Classification for Validate resource deletion recovery: infrastructure/probe-failure (code=1014, confidence=medium) 2026/07/05 17:35:39 Evidence: event Pod/kuberay-operator-74d759f546-g4x5g: Unhealthy - Readiness probe failed: Get "http://10.134.0.66:8080/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"probe-failure","error_code":1014,"evidence":["event Pod/kuberay-operator-74d759f546-g4x5g: Unhealthy - Readiness probe failed: Get \"http://10.134.0.66:8080/metrics\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"],"confidence":"medium"} 2026/07/05 17:35:39 Diagnostics complete for failed test: Validate resource deletion recovery === RUN TestOdhOperator/components/group_1/kserve/Validate_resource_deletion_recovery testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/kserve/Validate_resource_deletion_recovery (175.51s) === RUN TestOdhOperator/components/group_1/kserve === PAUSE TestOdhOperator/components/group_1/kserve === CONT TestOdhOperator/components/group_1/kserve test_context_test.go:223: Timed out after 90.003s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:121 with: Recreated deployment should have Available condition Expected <string>: map[apiVersion:apps/v1 kind:Deployment metadata:map[annotations:map[deployment.kubernetes.io/revision:1 internal.config.kubernetes.io/previousKinds:Deployment internal.config.kubernetes.io/previousNames:llmisvc-controller-manager internal.config.kubernetes.io/previousNamespaces:opendatahub platform.opendatahub.io/instance.generation:1 platform.opendatahub.io/instance.name:default-kserve platform.opendatahub.io/instance.uid:734f51e4-59c0-46fb-b29a-07f4068145d0 platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-05T17:34:05Z generation:1 labels:map[app.kubernetes.io/component:controller app.kubernetes.io/name:llmisvc-controller-manager app.kubernetes.io/part-of:kserve app.opendatahub.io/kserve:true control-plane:llmisvc-controller-manager controller-tools.k8s.io:1.0 platform.opendatahub.io/part-of:kserve] managedFields:[map[apiVersion:apps/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:internal.config.kubernetes.io/previousKinds:map[] f:internal.config.kubernetes.io/previousNames:map[] f:internal.config.kubernetes.io/previousNamespaces:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:app.kubernetes.io/component:map[] f:app.kubernetes.io/name:map[] f:app.kubernetes.io/part-of:map[] f:app.opendatahub.io/kserve:map[] f:control-plane:map[] f:controller-tools.k8s.io:map[] f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"734f51e4-59c0-46fb-b29a-07f4068145d0"}:map[]]] f:spec:map[f:replicas:map[] f:selector:map[] f:strategy:map[] f:template:map[f:metadata:map[f:annotations:map[f:kubectl.kubernetes.io/default-container:map[]] f:labels:map[f:app.kubernetes.io/component:map[] f:app.kubernetes.io/name:map[] f:app.kubernetes.io/part-of:map[] f:app.opendatahub.io/kserve:map[] f:control-plane:map[] f:controller-tools.k8s.io:map[]]] f:spec:map[f:containers:map[k:{"name":"manager"}:map[.:map[] f:args:map[] f:command:map[] f:env:map[k:{"name":"LLM_INFERENCE_SERVICE_CONFIG_PREFIX"}:map[.:map[] f:name:map[] f:value:map[]] k:{"name":"POD_NAMESPACE"}:map[.:map[] f:name:map[] f:valueFrom:map[f:fieldRef:map[]]]] f:image:map[] f:imagePullPolicy:map[] f:livenessProbe:map[f:failureThreshold:map[] f:httpGet:map[f:path:map[] f:port:map[]] f:initialDelaySeconds:map[] f:periodSeconds:map[] f:timeoutSeconds:map[]] f:name:map[] f:ports:map[k:{"containerPort":8443,"protocol":"TCP"}:map[.:map[] f:containerPort:map[] f:name:map[] f:protocol:map[]] k:{"containerPort":9443,"protocol":"TCP"}:map[.:map[] f:containerPort:map[] f:name:map[] f:protocol:map[]]] f:readinessProbe:map[f:failureThreshold:map[] f:httpGet:map[f:path:map[] f:port:map[]] f:initialDelaySeconds:map[] f:periodSeconds:map[] f:timeoutSeconds:map[]] f:resources:map[f:limits:map[f:cpu:map[] f:memory:map[]] f:requests:map[f:cpu:map[] f:memory:map[]]] f:securityContext:map[f:allowPrivilegeEscalation:map[] f:capabilities:map[f:drop:map[]] f:privileged:map[] f:readOnlyRootFilesystem:map[] f:runAsNonRoot:map[] f:seccompProfile:map[f:type:map[]]] f:volumeMounts:map[k:{"mountPath":"/tmp/k8s-webhook-server/serving-certs"}:map[.:map[] f:mountPath:map[] f:name:map[] f:readOnly:map[]]]]] f:securityContext:map[f:runAsNonRoot:map[] f:seccompProfile:map[f:type:map[]]] f:serviceAccountName:map[] f:terminationGracePeriodSeconds:map[] f:volumes:map[k:{"name":"cert"}:map[.:map[] f:name:map[] f:secret:map[f:defaultMode:map[] f:secretName:map[]]]]]]]] manager:kserve operation:Apply time:2026-07-05T17:34:06Z] map[apiVersion:apps/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:deployment.kubernetes.io/revision:map[]]] f:status:map[f:conditions:map[.:map[] k:{"type":"Available"}:map[.:map[] f:lastTransitionTime:map[] f:lastUpdateTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Progressing"}:map[.:map[] f:lastTransitionTime:map[] f:lastUpdateTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]]] f:observedGeneration:map[] f:replicas:map[] f:unavailableReplicas:map[] f:updatedReplicas:map[]]] manager:kube-controller-manager operation:Update subresource:status time:2026-07-05T17:34:05Z]] name:llmisvc-controller-manager namespace:redhat-ods-applications ownerReferences:[map[apiVersion:components.platform.opendatahub.io/v1alpha1 blockOwnerDeletion:true controller:true kind:Kserve name:default-kserve uid:734f51e4-59c0-46fb-b29a-07f4068145d0]] resourceVersion:128475 uid:ae3c924a-bdb1-47b9-99e9-52d55bd12808] spec:map[progressDeadlineSeconds:600 replicas:1 revisionHistoryLimit:10 selector:map[matchLabels:map[app.kubernetes.io/part-of:kserve app.opendatahub.io/kserve:true control-plane:llmisvc-controller-manager controller-tools.k8s.io:1.0]] strategy:map[rollingUpdate:map[maxSurge:25% maxUnavailable:25%] type:RollingUpdate] template:map[metadata:map[annotations:map[kubectl.kubernetes.io/default-container:manager] labels:map[app.kubernetes.io/component:controller app.kubernetes.io/name:llmisvc-controller-manager app.kubernetes.io/part-of:kserve app.opendatahub.io/kserve:true control-plane:llmisvc-controller-manager controller-tools.k8s.io:1.0]] spec:map[containers:[map[args:[--metrics-addr=127.0.0.1:8443 --leader-elect] command:[/manager] env:[map[name:POD_NAMESPACE valueFrom:map[fieldRef:map[apiVersion:v1 fieldPath:metadata.namespace]]] map[name:LLM_INFERENCE_SERVICE_CONFIG_PREFIX value:v3-5-0-ea-2-kserve-]] image:registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b imagePullPolicy:Always livenessProbe:map[failureThreshold:5 httpGet:map[path:/healthz port:8081 scheme:HTTP] initialDelaySeconds:30 periodSeconds:10 successThreshold:1 timeoutSeconds:5] name:manager ports:[map[containerPort:9443 name:webhook-server protocol:TCP] map[containerPort:8443 name:metrics protocol:TCP]] readinessProbe:map[failureThreshold:5 httpGet:map[path:/readyz port:8081 scheme:HTTP] initialDelaySeconds:30 periodSeconds:5 successThreshold:1 timeoutSeconds:5] resources:map[limits:map[cpu:100m memory:300Mi] requests:map[cpu:100m memory:300Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] privileged:false readOnlyRootFilesystem:true runAsNonRoot:true seccompProfile:map[type:RuntimeDefault]] terminationMessagePath:/dev/termination-log terminationMessagePolicy:File volumeMounts:[map[mountPath:/tmp/k8s-webhook-server/serving-certs name:cert readOnly:true]]]] dnsPolicy:ClusterFirst restartPolicy:Always schedulerName:default-scheduler securityContext:map[runAsNonRoot:true seccompProfile:map[type:RuntimeDefault]] serviceAccount:llmisvc-controller-manager serviceAccountName:llmisvc-controller-manager terminationGracePeriodSeconds:30 volumes:[map[name:cert secret:map[defaultMode:420 secretName:llmisvc-webhook-server-cert]]]]]] status:map[conditions:[map[lastTransitionTime:2026-07-05T17:34:05Z lastUpdateTime:2026-07-05T17:34:05Z message:Deployment does not have minimum availability. reason:MinimumReplicasUnavailable status:False type:Available] map[lastTransitionTime:2026-07-05T17:34:05Z lastUpdateTime:2026-07-05T17:34:05Z message:ReplicaSet "llmisvc-controller-manager-bc44d7c7b" is progressing. reason:ReplicaSetUpdated status:True type:Progressing]] observedGeneration:1 replicas:1 unavailableReplicas:1 updatedReplicas:1]] to match expression <string>: .status.conditions[] | select(.type == "Available") | .status == "True" --- FAIL: TestOdhOperator/components/group_1/kserve (256.10s) === RUN TestOdhOperator/components/group_1 --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components/group_2/modelcontroller/Validate_resource_deletion_recovery/ConfigMap_deletion_recovery/ConfigMap_workload-variant-autoscaler-saturation-scaling-config testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/modelcontroller/Validate_resource_deletion_recovery/ConfigMap_deletion_recovery/ConfigMap_workload-variant-autoscaler-saturation-scaling-config (95.01s) 2026/07/05 17:41:56 TEST FAILURE DETECTED: ConfigMap deletion recovery 2026/07/05 17:41:56 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:ConfigMap deletion recovery) === 2026/07/05 17:41:56 clusterhealth: node metrics unavailable (Metrics Server may not be installed): the server is currently unable to handle the request (get nodes.metrics.k8s.io) 2026/07/05 17:41:56 === CLUSTER STATE === 2026/07/05 17:41:56 Failed to collect node data: unhealthy nodes: ip-10-0-131-75.ec2.internal (Ready=Unknown); ip-10-0-137-37.ec2.internal (Ready=Unknown) 2026/07/05 17:41:56 === DEPLOYMENTS === 2026/07/05 17:41:56 Failed to collect deployment data: deployments not ready: redhat-ods-applications/kserve-controller-manager (0/1), redhat-ods-applications/llmisvc-controller-manager (0/1), redhat-ods-applications/maas-api (0/1), redhat-ods-applications/maas-controller (0/1), redhat-ods-applications/mlflow-operator-controller-manager (0/1), redhat-ods-applications/model-registry-operator-controller-manager (0/1), redhat-ods-applications/model-serving-api (0/2), redhat-ods-applications/odh-model-controller (0/2), redhat-ods-applications/ogx-k8s-operator-controller-manager (0/1), redhat-ods-applications/postgres (0/1), redhat-ods-applications/trustyai-service-operator-controller-manager (0/1), redhat-ods-applications/workload-variant-autoscaler-controller-manager (0/2) 2026/07/05 17:41:56 === PODS === 2026/07/05 17:41:56 Namespace redhat-ods-applications: no problematic pods found 2026/07/05 17:41:56 Namespace kube-system: no problematic pods found 2026/07/05 17:41:56 === OPERATOR STATUS === 2026/07/05 17:41:56 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; dependent kueue: deployment 0/2 ready; dependent jobset: deployment 0/1 ready; dependent leader-worker-set: deployment 0/1 ready; dependent cert-manager: deployment 0/1 ready; dependent kuadrant: deployment 0/1 ready 2026/07/05 17:41:56 === DSCI STATUS === 2026/07/05 17:41:56 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/05 17:41:56 === DSC STATUS === 2026/07/05 17:41:56 DSC default-dsc: 2026/07/05 17:41:56 Ready: True - 2026/07/05 17:41:56 ProvisioningSucceeded: True - 2026/07/05 17:41:56 ComponentsReady: True - 2026/07/05 17:41:56 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/05 17:41:56 DashboardReady: False - Component ManagementState is set to Removed 2026/07/05 17:41:56 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/05 17:41:56 KserveLLMInferenceServiceDependencies: True - 2026/07/05 17:41:56 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/05 17:41:56 KserveReady: True - 2026/07/05 17:41:56 KueueReady: True - 2026/07/05 17:41:56 LLM-D-WVADependencies: True - 2026/07/05 17:41:56 MLflowOperatorReady: True - 2026/07/05 17:41:56 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/07/05 17:41:56 ModelControllerReady: True - 2026/07/05 17:41:56 ModelRegistryReady: True - 2026/07/05 17:41:56 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/05 17:41:56 OGXReady: True - 2026/07/05 17:41:56 RayReady: False - Component ManagementState is set to Removed 2026/07/05 17:41:56 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/05 17:41:56 TrainerReady: False - Component ManagementState is set to Removed 2026/07/05 17:41:56 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/05 17:41:56 TrustyAIReady: True - 2026/07/05 17:41:56 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/05 17:41:56 === RECENT EVENTS === 2026/07/05 17:41:56 17:41:18 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/kserve-controller-manager-588b4b787c-4nm25 2026/07/05 17:41:56 17:41:18 INFO Pod/odh-model-controller-566d8764f4-ddc4h: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/odh-model-controller-566d8764f4-ddc4h 2026/07/05 17:41:56 17:41:18 INFO Pod/model-serving-api-5c897df776-2pwnq: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/model-serving-api-5c897df776-2pwnq 2026/07/05 17:41:56 17:41:18 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q 2026/07/05 17:41:56 17:41:18 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-h9kc7 2026/07/05 17:41:56 17:41:17 WARN Pod/trustyai-service-operator-controller-manager-7bcdbcb4fd-n65d4: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:17 WARN Pod/maas-controller-5b858f66fc-xrp5h: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:17 WARN Pod/model-serving-api-5c897df776-4mkx7: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:17 WARN Pod/global-pull-secret-syncer-5l728: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:17 WARN Pod/kube-apiserver-proxy-ip-10-0-143-231.ec2.internal: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:17 WARN Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:17 WARN Pod/postgres-69fb4594cf-mz8gk: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:17 WARN Pod/konnectivity-agent-mff9p: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:17 WARN Pod/mlflow-operator-controller-manager-57f4cd4d95-jlw55: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:12 WARN Pod/model-serving-api-5c897df776-2pwnq: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/kube-apiserver-proxy-ip-10-0-131-75.ec2.internal: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/kserve-controller-manager-588b4b787c-4nm25: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/workload-variant-autoscaler-controller-manager-67777f585d-dck2s: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/konnectivity-agent-tcxd2: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/global-pull-secret-syncer-gfzgr: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/global-pull-secret-syncer-8vvhd: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/odh-model-controller-566d8764f4-ddc4h: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/model-registry-operator-controller-manager-b67847846-s94zx: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/konnectivity-agent-n2m85: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/odh-model-controller-566d8764f4-rw474: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:41:11 WARN Pod/maas-api-557679ff4c-rn27g: NodeNotReady - Node is not ready 2026/07/05 17:41:56 17:39:53 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Pulled - Successfully pulled image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:79fa44b245109a0c1d214c8b313dc8172b367dedf75c4691ed1faefa46f41775" in 1.498s (1.498s including waiting). Image size: 195240092 bytes. 2026/07/05 17:41:56 17:39:53 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Created - Created container: manager 2026/07/05 17:41:56 17:39:53 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Started - Started container manager 2026/07/05 17:41:56 17:39:51 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q to ip-10-0-131-75.ec2.internal 2026/07/05 17:41:56 17:39:51 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-67777f585d: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-67777f585d-5kz8q 2026/07/05 17:41:56 17:39:51 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: AddedInterface - Add eth0 [10.134.0.78/23] from ovn-kubernetes 2026/07/05 17:41:56 17:39:51 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Pulling - Pulling image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:79fa44b245109a0c1d214c8b313dc8172b367dedf75c4691ed1faefa46f41775" 2026/07/05 17:41:56 17:39:50 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-67777f585d from 1 to 2 2026/07/05 17:41:56 17:39:33 INFO Pod/odh-model-controller-566d8764f4-ddc4h: Created - Created container: manager 2026/07/05 17:41:56 17:39:33 INFO Pod/odh-model-controller-566d8764f4-ddc4h: Started - Started container manager 2026/07/05 17:41:56 17:39:33 INFO Pod/odh-model-controller-566d8764f4-ddc4h: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" in 2.242s (2.242s including waiting). Image size: 238959021 bytes. 2026/07/05 17:41:56 17:39:31 INFO Pod/odh-model-controller-566d8764f4-ddc4h: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-566d8764f4-ddc4h to ip-10-0-131-75.ec2.internal 2026/07/05 17:41:56 17:39:31 INFO Pod/odh-model-controller-566d8764f4-ddc4h: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" 2026/07/05 17:41:56 17:39:31 INFO Pod/odh-model-controller-566d8764f4-ddc4h: AddedInterface - Add eth0 [10.134.0.77/23] from ovn-kubernetes 2026/07/05 17:41:56 17:39:31 INFO ReplicaSet/odh-model-controller-566d8764f4: SuccessfulCreate - Created pod: odh-model-controller-566d8764f4-ddc4h 2026/07/05 17:41:56 17:39:30 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-566d8764f4 from 1 to 2 2026/07/05 17:41:56 17:39:17 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:39:17 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:39:11 INFO Pod/model-serving-api-5c897df776-2pwnq: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5c897df776-2pwnq to ip-10-0-131-75.ec2.internal 2026/07/05 17:41:56 17:39:11 INFO Pod/model-serving-api-5c897df776-2pwnq: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:c690422f9b5d3fc40fe327208b7af310035080730395bb9c4e393965dce7b53f" already present on machine 2026/07/05 17:41:56 17:39:11 INFO Pod/model-serving-api-5c897df776-2pwnq: Started - Started container server 2026/07/05 17:41:56 17:39:11 INFO Pod/model-serving-api-5c897df776-2pwnq: AddedInterface - Add eth0 [10.134.0.76/23] from ovn-kubernetes 2026/07/05 17:41:56 17:39:11 INFO ReplicaSet/model-serving-api-5c897df776: SuccessfulCreate - Created pod: model-serving-api-5c897df776-2pwnq 2026/07/05 17:41:56 17:39:11 INFO Pod/model-serving-api-5c897df776-2pwnq: Created - Created container: server 2026/07/05 17:41:56 17:39:10 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5c897df776 from 1 to 2 2026/07/05 17:41:56 17:39:09 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:39:09 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:38:48 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-566d8764f4-rw474_fc972b41-94a0-444f-acb6-01f6b7be2bc2 became leader 2026/07/05 17:41:56 17:38:45 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:38:45 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:38:34 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-b67847846-s94zx_97acdd28-4627-4d57-aac0-5d608a3d4989 became leader 2026/07/05 17:41:56 17:38:32 INFO ReplicaSet/odh-model-controller-5f7b7f7655: SuccessfulDelete - Deleted pod: odh-model-controller-5f7b7f7655-mf2jg 2026/07/05 17:41:56 17:38:32 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled down replica set odh-model-controller-5f7b7f7655 from 1 to 0 2026/07/05 17:41:56 17:38:32 INFO Pod/odh-model-controller-5f7b7f7655-mf2jg: Killing - Stopping container manager 2026/07/05 17:41:56 17:38:31 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-dck2s: Pulled - Successfully pulled image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:79fa44b245109a0c1d214c8b313dc8172b367dedf75c4691ed1faefa46f41775" in 1.767s (1.767s including waiting). Image size: 195240092 bytes. 2026/07/05 17:41:56 17:38:31 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-dck2s: Created - Created container: manager 2026/07/05 17:41:56 17:38:31 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-dck2s: Started - Started container manager 2026/07/05 17:41:56 17:38:31 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-67777f585d-dck2s_f71ae809-9bfb-4ca3-895d-1462f946fb47 became leader 2026/07/05 17:41:56 17:38:29 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:38:29 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/workload-variant-autoscaler-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: bearerTokenSecret: [REDACTED] to get secret "workload-variant-autoscaler-controller-manager-token": secrets "workload-variant-autoscaler-controller-manager-token" not found 2026/07/05 17:41:56 17:38:29 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:38:29 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-dck2s: AddedInterface - Add eth0 [10.132.0.53/23] from ovn-kubernetes 2026/07/05 17:41:56 17:38:29 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-dck2s: Pulling - Pulling image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:79fa44b245109a0c1d214c8b313dc8172b367dedf75c4691ed1faefa46f41775" 2026/07/05 17:41:56 17:38:28 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-dck2s: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-67777f585d-dck2s to ip-10-0-137-37.ec2.internal 2026/07/05 17:41:56 17:38:28 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-67777f585d: FailedCreate - Error creating: pods "workload-variant-autoscaler-controller-manager-67777f585d-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 2026/07/05 17:41:56 17:38:28 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-67777f585d from 0 to 1 2026/07/05 17:41:56 17:38:28 WARN Pod/workload-variant-autoscaler-controller-manager-67777f585d-dck2s: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 2026/07/05 17:41:56 17:38:28 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-67777f585d: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-67777f585d-dck2s 2026/07/05 17:41:56 17:38:20 INFO Pod/odh-model-controller-566d8764f4-rw474: Started - Started container manager 2026/07/05 17:41:56 17:38:20 INFO Pod/odh-model-controller-566d8764f4-rw474: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" in 2.602s (2.602s including waiting). Image size: 238959021 bytes. 2026/07/05 17:41:56 17:38:20 INFO Pod/odh-model-controller-566d8764f4-rw474: Created - Created container: manager 2026/07/05 17:41:56 17:38:18 INFO Pod/model-registry-operator-controller-manager-b67847846-s94zx: AddedInterface - Add eth0 [10.132.0.52/23] from ovn-kubernetes 2026/07/05 17:41:56 17:38:18 INFO Pod/model-registry-operator-controller-manager-b67847846-s94zx: Pulled - Container image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:e1bd98206aee94e76215e2a9ea4987e9c6f4597bdba1358921496b275e91abc0" already present on machine 2026/07/05 17:41:56 17:38:18 INFO Pod/model-registry-operator-controller-manager-b67847846-s94zx: Created - Created container: manager 2026/07/05 17:41:56 17:38:18 INFO Pod/model-registry-operator-controller-manager-b67847846-s94zx: Started - Started container manager 2026/07/05 17:41:56 17:38:18 INFO Pod/odh-model-controller-566d8764f4-rw474: AddedInterface - Add eth0 [10.132.0.51/23] from ovn-kubernetes 2026/07/05 17:41:56 17:38:18 INFO Pod/odh-model-controller-566d8764f4-rw474: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:49111a3777043a2d16e6599bdc99cacb4839a64949097bf9bdfa798d3bba2c43" 2026/07/05 17:41:56 17:38:18 WARN Pod/model-registry-operator-controller-manager-b67847846-s94zx: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/07/05 17:41:56 17:38:17 INFO Pod/model-registry-operator-controller-manager-b67847846-s94zx: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-b67847846-s94zx to ip-10-0-137-37.ec2.internal 2026/07/05 17:41:56 17:38:17 INFO Pod/odh-model-controller-566d8764f4-rw474: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-566d8764f4-rw474 to ip-10-0-137-37.ec2.internal 2026/07/05 17:41:56 17:38:17 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-566d8764f4 from 0 to 1 2026/07/05 17:41:56 17:38:17 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-b67847846 from 0 to 1 2026/07/05 17:41:56 17:38:17 INFO ReplicaSet/model-registry-operator-controller-manager-b67847846: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-b67847846-s94zx 2026/07/05 17:41:56 17:38:17 INFO ReplicaSet/odh-model-controller-566d8764f4: SuccessfulCreate - Created pod: odh-model-controller-566d8764f4-rw474 2026/07/05 17:41:56 17:38:16 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:38:16 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:38:07 INFO Pod/spark-operator-webhook-5f8768dbfd-rn22f: Killing - Stopping container webhook 2026/07/05 17:41:56 17:38:07 INFO Pod/spark-operator-controller-5dd7d48f87-w2f6t: Killing - Stopping container controller 2026/07/05 17:41:56 17:38:06 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:38:06 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:37:53 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-5dd7d48f87-w2f6t_7a3bf462-1ef6-487d-a25c-a80b773a6445 became leader 2026/07/05 17:41:56 17:37:51 INFO Pod/spark-operator-webhook-5f8768dbfd-rn22f: Started - Started container webhook 2026/07/05 17:41:56 17:37:51 INFO Pod/spark-operator-webhook-5f8768dbfd-rn22f: Created - Created container: webhook 2026/07/05 17:41:56 17:37:51 INFO Pod/spark-operator-webhook-5f8768dbfd-rn22f: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/05 17:41:56 17:37:51 INFO Pod/spark-operator-webhook-5f8768dbfd-rn22f: AddedInterface - Add eth0 [10.134.0.74/23] from ovn-kubernetes 2026/07/05 17:41:56 17:37:50 INFO Pod/spark-operator-webhook-5f8768dbfd-rn22f: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-rn22f to ip-10-0-131-75.ec2.internal 2026/07/05 17:41:56 17:37:50 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Killing - Stopping container webhook 2026/07/05 17:41:56 17:37:50 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 0 to 1 2026/07/05 17:41:56 17:37:50 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-rn22f 2026/07/05 17:41:56 17:37:50 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Killing - Stopping container webhook 2026/07/05 17:41:56 17:37:36 INFO Pod/spark-operator-controller-5dd7d48f87-w2f6t: Started - Started container controller 2026/07/05 17:41:56 17:37:36 INFO Pod/spark-operator-controller-5dd7d48f87-w2f6t: Created - Created container: controller 2026/07/05 17:41:56 17:37:35 INFO Pod/spark-operator-controller-5dd7d48f87-w2f6t: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-w2f6t to ip-10-0-131-75.ec2.internal 2026/07/05 17:41:56 17:37:35 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Killing - Stopping container controller 2026/07/05 17:41:56 17:37:35 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-w2f6t 2026/07/05 17:41:56 17:37:35 INFO Pod/spark-operator-controller-5dd7d48f87-w2f6t: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/05 17:41:56 17:37:35 INFO Pod/spark-operator-controller-5dd7d48f87-w2f6t: AddedInterface - Add eth0 [10.134.0.73/23] from ovn-kubernetes 2026/07/05 17:41:56 17:37:35 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 0 to 1 2026/07/05 17:41:56 17:37:35 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Killing - Stopping container controller 2026/07/05 17:41:56 17:37:06 INFO SparkApplication/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607: SparkDriverCompleted - Driver scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607-driver completed 2026/07/05 17:41:56 17:37:04 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:37:04 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:37:04 INFO Pod/spark-pi-c73af09f335aa692-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/05 17:41:56 17:37:04 INFO SparkApplication/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607: SparkExecutorCompleted - Executor spark-pi-c73af09f335aa692-exec-1 completed 2026/07/05 17:41:56 17:37:03 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 17:37:03 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:41:56 === RESOURCE QUOTAS === 2026/07/05 17:41:56 No resource quotas found in monitored namespaces 2026/07/05 17:41:56 Classification for ConfigMap deletion recovery: infrastructure/rbac (code=1011, confidence=medium) 2026/07/05 17:41:56 Evidence: event ReplicaSet/workload-variant-autoscaler-controller-manager-67777f585d: FailedCreate - Error creating: pods "workload-variant-autoscaler-controller-manager-67777f585d-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"rbac","error_code":1011,"evidence":["event ReplicaSet/workload-variant-autoscaler-controller-manager-67777f585d: FailedCreate - Error creating: pods \"workload-variant-autoscaler-controller-manager-67777f585d-\" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount \"workload-variant-autoscaler-controller-manager\" not found"],"confidence":"medium"} 2026/07/05 17:41:56 Diagnostics complete for failed test: ConfigMap deletion recovery === RUN TestOdhOperator/components/group_2/modelcontroller/Validate_resource_deletion_recovery/ConfigMap_deletion_recovery testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/modelcontroller/Validate_resource_deletion_recovery/ConfigMap_deletion_recovery (105.18s) 2026/07/05 17:41:56 TEST FAILURE DETECTED: Validate resource deletion recovery 2026/07/05 17:41:56 Diagnostics already ran for key "test:Validate resource deletion recovery" 2026/07/05 17:41:56 Diagnostics complete for failed test: Validate resource deletion recovery === RUN TestOdhOperator/components/group_2/modelcontroller/Validate_resource_deletion_recovery testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/modelcontroller/Validate_resource_deletion_recovery (105.18s) === RUN TestOdhOperator/components/group_2/modelcontroller === PAUSE TestOdhOperator/components/group_2/modelcontroller === CONT TestOdhOperator/components/group_2/modelcontroller 2026-07-05T17:38:15Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.23"}} test_context_test.go:1321: Timed out after 90.000s. Resource ConfigMap redhat-ods-applications/workload-variant-autoscaler-saturation-scaling-config was not properly recreated with new identity The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1302 with: Expected ConfigMap redhat-ods-applications/workload-variant-autoscaler-saturation-scaling-config to be recreated Expected <*unstructured.Unstructured | 0x0>: nil not to be nil --- FAIL: TestOdhOperator/components/group_2/modelcontroller (220.46s) === RUN TestOdhOperator/components/group_2/kueue/Validate_autoCreateQueues_disabled_skips_queue_creation helper_test.go:156: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. helper_test.go:156: Cleaning up any existing Kueue test resources. helper_test.go:156: Cleaning up Kueue cluster-scoped resources helper_test.go:156: Will attempt to delete 3 Kueue resources helper_test.go:156: Attempting to delete ClusterQueue /default helper_test.go:156: Successfully processed deletion of ClusterQueue /default helper_test.go:156: Attempting to delete ResourceFlavor /default-flavor helper_test.go:156: Successfully processed deletion of ResourceFlavor /default-flavor helper_test.go:156: Attempting to delete Kueue /cluster helper_test.go:156: Successfully processed deletion of Kueue /cluster helper_test.go:156: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:156: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:156: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:156: Creating test namespace (test-kueue-no-queues-d959dhe96nfs74re5dig) with Kueue management annotation. helper_test.go:156: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:156: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:156: Updating DSC to set Kueue to Unmanaged with autoCreateQueues=false (default). 2026-07-05T17:39:27Z INFO unknown field "spec.components.kueue.autoCreateQueues" helper_test.go:156: Verifying Kueue configuration (cluster) is still created. helper_test.go:156: Verifying no ClusterQueue, ResourceFlavor or LocalQueue is created (autoCreateQueues=false). testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_autoCreateQueues_disabled_skips_queue_creation (320.26s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' test_context_test.go:615: Timed out after 300.009s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'default' of kind 'ClusterQueue' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "apiVersion": <string>"kueue.x-k8s.io/v1beta2", "kind": <string>"ClusterQueue", "metadata": <map[string]interface {} | len:8>{ "creationTimestamp": <string>"2026-07-05T17:39:27Z", "generation": <int64>1, "labels": <map[string]interface {} | len:1>{ "platform.opendatahub.io/part-of": <string>"kueue", }, "managedFields": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:6>{ "apiVersion": <string>"kueue.x-k8s.io/v1beta2", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:2>{ "f:annotations": <map[string]interface {} | len:6>{ "f:platform.opendatahub.io/instance.generation": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/instance.name": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/instance.uid": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/type": <map[string]interface {} | len:0>{}, "f:platform.opendatahub.io/version": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, }, "f:labels": <map[string]interface {} | len:2>{ "f:platform.opendatahub.io/part-of": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:7>{ "f:resourceGroups": <map[string]interface {} | len:0>{}, "f:stopPolicy": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, "f:flavorFungibility": <map[string]interface {} | len:3>{ "f:whenCanPreempt": <map[string]interface {} | len:0>{}, ".": <map[string]interface {} | len:0>{}, "f:whenCanBorrow": <map[string]interface {} | len:0>{}, }, "f:namespaceSelector": <map[string]interface {} | len:0>{}, "f:preemption": <map[string]interface {} | len:4>{ ".": <map[string]interface {} | len:0>{}, "f:borrowWithinCohort": <map[string]interface {} | len:2>{...: ..., ...: ...}, "f:reclaimWithinCohort": <map[string]interface {} | len:0>{}, "f:withinClusterQueue": <map[string]interface {} | len:0>{}, }, "f:queueingStrategy": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-05T17:39:27Z", }, ], "name": <string>"default", "resourceVersion": <string>"136980", "uid": <string>"b299f975-dd34-4de7-bc3c-595f0a471b96", "annotations": <map[string]interface {} | len:5>{ "platform.opendatahub.io/instance.generation": <string>"1", "platform.opendatahub.io/instance.name": <string>"default-kueue", "platform.opendatahub.io/instance.uid": <string>"e7a05f90-ffd5-4820-9fc8-30e19557e095", "platform.opendatahub.io/type": <string>"OpenShift AI Self-Managed", "platform.opendatahub.io/version": <string>"3.5.0-ea.2", }, }, "spec": <map[string]interface {} | len:6>{ "flavorFungibility": <map[string]interface {} | len:2>{ "whenCanBorrow": <string>"MayStopSearch", "whenCanPreempt": <string>"TryNextFlavor", }, "namespaceSelector": <map[string]interface {} | len:1>{ "matchLabels": <map[string]interface {} | len:1>{ "kueue.openshift.io/managed": <string>"true", }, }, "preemption": <map[string]interface {} | len:3>{ "borrowWithinCohort": <map[string]interface {} | len:1>{ "policy": <string>"Never", }, "reclaimWithinCohort": <string>"Never", "withinClusterQueue": <string>"Never", }, "queueingStrategy": <string>"BestEffortFIFO", "resourceGroups": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:2>{ "coveredResources": <[]interface {} | len:2, cap:2>[<string>"cpu", <string>"memory"], "flavors": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:2>{ "name": <string>"default-flavor", "resources": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:2>{...: ..., ...: ...}, <map[string]interface {} | len:2>{...: ..., ...: ...}, ], }, ], }, ], "stopPolicy": <string>"None", }, }, }, ] to be empty test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (381.55s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components 2026/07/05 17:38:15 TEST FAILURE DETECTED: group 1 2026/07/05 17:38:15 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/05 17:38:15 === CLUSTER STATE === 2026/07/05 17:38:15 Node: ip-10-0-131-75.ec2.internal 2026/07/05 17:38:15 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:38:15 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:38:15 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:38:15 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:38:15 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/05 17:38:15 Node: ip-10-0-137-37.ec2.internal 2026/07/05 17:38:15 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:38:15 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:38:15 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:38:15 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:38:15 Resources: 7500m CPU, 31236220Ki memory allocatable (of 8000m CPU, 32387196Ki memory capacity) 2026/07/05 17:38:15 Node: ip-10-0-143-231.ec2.internal 2026/07/05 17:38:15 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:38:15 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:38:15 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:38:15 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:38:15 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/05 17:38:15 === DEPLOYMENTS === 2026/07/05 17:38:15 Namespace: redhat-ods-applications 2026/07/05 17:38:15 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/05 17:38:15 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/05 17:38:15 Deployment maas-api: 1/1 (READY) 2026/07/05 17:38:15 Deployment maas-controller: 1/1 (READY) 2026/07/05 17:38:15 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/05 17:38:15 Deployment model-serving-api: 1/1 (READY) 2026/07/05 17:38:15 Deployment odh-model-controller: 1/1 (READY) 2026/07/05 17:38:15 Deployment ogx-k8s-operator-controller-manager: 1/1 (READY) 2026/07/05 17:38:15 Deployment postgres: 1/1 (READY) 2026/07/05 17:38:15 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/05 17:38:15 Namespace: kube-system 2026/07/05 17:38:15 === PODS === 2026/07/05 17:38:15 Namespace redhat-ods-applications: no problematic pods found 2026/07/05 17:38:15 Namespace kube-system: no problematic pods found 2026/07/05 17:38:15 === OPERATOR STATUS === 2026/07/05 17:38:15 Operator deployment: 3/3 ready 2026/07/05 17:38:15 === DSCI STATUS === 2026/07/05 17:38:15 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/05 17:38:15 === DSC STATUS === 2026/07/05 17:38:15 DSC default-dsc: 2026/07/05 17:38:15 Ready: True - 2026/07/05 17:38:15 ProvisioningSucceeded: True - 2026/07/05 17:38:15 ComponentsReady: True - 2026/07/05 17:38:15 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/05 17:38:15 DashboardReady: False - Component ManagementState is set to Removed 2026/07/05 17:38:15 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/05 17:38:15 KserveLLMInferenceServiceDependencies: True - 2026/07/05 17:38:15 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/05 17:38:15 KserveReady: True - 2026/07/05 17:38:15 KueueReady: False - Component ManagementState is set to Removed 2026/07/05 17:38:15 LLM-D-WVADependencies: True - 2026/07/05 17:38:15 MLflowOperatorReady: True - 2026/07/05 17:38:15 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/07/05 17:38:15 ModelControllerReady: True - 2026/07/05 17:38:15 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/05 17:38:15 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/05 17:38:15 OGXReady: True - 2026/07/05 17:38:15 RayReady: False - Component ManagementState is set to Removed 2026/07/05 17:38:15 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/05 17:38:15 TrainerReady: False - Component ManagementState is set to Removed 2026/07/05 17:38:15 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/05 17:38:15 TrustyAIReady: True - 2026/07/05 17:38:15 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/05 17:38:15 === RECENT EVENTS === 2026/07/05 17:38:15 17:38:07 INFO Pod/spark-operator-webhook-5f8768dbfd-rn22f: Killing - Stopping container webhook 2026/07/05 17:38:15 17:38:07 INFO Pod/spark-operator-controller-5dd7d48f87-w2f6t: Killing - Stopping container controller 2026/07/05 17:38:15 17:38:06 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:38:06 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:37:53 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-5dd7d48f87-w2f6t_7a3bf462-1ef6-487d-a25c-a80b773a6445 became leader 2026/07/05 17:38:15 17:37:51 INFO Pod/spark-operator-webhook-5f8768dbfd-rn22f: AddedInterface - Add eth0 [10.134.0.74/23] from ovn-kubernetes 2026/07/05 17:38:15 17:37:51 INFO Pod/spark-operator-webhook-5f8768dbfd-rn22f: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/05 17:38:15 17:37:51 INFO Pod/spark-operator-webhook-5f8768dbfd-rn22f: Created - Created container: webhook 2026/07/05 17:38:15 17:37:51 INFO Pod/spark-operator-webhook-5f8768dbfd-rn22f: Started - Started container webhook 2026/07/05 17:38:15 17:37:50 INFO Pod/spark-operator-webhook-5f8768dbfd-rn22f: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-rn22f to ip-10-0-131-75.ec2.internal 2026/07/05 17:38:15 17:37:50 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 0 to 1 2026/07/05 17:38:15 17:37:50 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Killing - Stopping container webhook 2026/07/05 17:38:15 17:37:50 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-rn22f 2026/07/05 17:38:15 17:37:50 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Killing - Stopping container webhook 2026/07/05 17:38:15 17:37:36 INFO Pod/spark-operator-controller-5dd7d48f87-w2f6t: Started - Started container controller 2026/07/05 17:38:15 17:37:36 INFO Pod/spark-operator-controller-5dd7d48f87-w2f6t: Created - Created container: controller 2026/07/05 17:38:15 17:37:35 INFO Pod/spark-operator-controller-5dd7d48f87-w2f6t: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-w2f6t to ip-10-0-131-75.ec2.internal 2026/07/05 17:38:15 17:37:35 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-w2f6t 2026/07/05 17:38:15 17:37:35 INFO Pod/spark-operator-controller-5dd7d48f87-w2f6t: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/05 17:38:15 17:37:35 INFO Pod/spark-operator-controller-5dd7d48f87-w2f6t: AddedInterface - Add eth0 [10.134.0.73/23] from ovn-kubernetes 2026/07/05 17:38:15 17:37:35 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Killing - Stopping container controller 2026/07/05 17:38:15 17:37:35 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Killing - Stopping container controller 2026/07/05 17:38:15 17:37:35 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 0 to 1 2026/07/05 17:38:15 17:37:06 INFO SparkApplication/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607: SparkDriverCompleted - Driver scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607-driver completed 2026/07/05 17:38:15 17:37:04 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:37:04 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:37:04 INFO Pod/spark-pi-c73af09f335aa692-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/05 17:38:15 17:37:04 INFO SparkApplication/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607: SparkExecutorCompleted - Executor spark-pi-c73af09f335aa692-exec-1 completed 2026/07/05 17:38:15 17:37:03 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:37:03 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:36:50 INFO Pod/spark-pi-c73af09f335aa692-exec-1: AddedInterface - Add eth0 [10.133.0.123/23] from ovn-kubernetes 2026/07/05 17:38:15 17:36:50 INFO Pod/spark-pi-c73af09f335aa692-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/05 17:38:15 17:36:50 INFO Pod/spark-pi-c73af09f335aa692-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/05 17:38:15 17:36:50 INFO Pod/spark-pi-c73af09f335aa692-exec-1: Started - Started container spark-kubernetes-executor 2026/07/05 17:38:15 17:36:50 INFO SparkApplication/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607: SparkExecutorRunning - Executor spark-pi-c73af09f335aa692-exec-1 is running 2026/07/05 17:38:15 17:36:49 INFO Pod/spark-pi-c73af09f335aa692-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-c73af09f335aa692-exec-1 to ip-10-0-143-231.ec2.internal 2026/07/05 17:38:15 17:36:49 INFO SparkApplication/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607: SparkExecutorPending - Executor spark-pi-c73af09f335aa692-exec-1 is pending 2026/07/05 17:38:15 17:36:41 INFO SparkApplication/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607: SparkDriverRunning - Driver scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607-driver is running 2026/07/05 17:38:15 17:36:40 INFO Pod/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607-driver: AddedInterface - Add eth0 [10.133.0.122/23] from ovn-kubernetes 2026/07/05 17:38:15 17:36:40 INFO SparkApplication/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607: SparkApplicationSubmitted - SparkApplication scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607 was submitted successfully 2026/07/05 17:38:15 17:36:40 INFO Pod/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607-driver: Started - Started container spark-kubernetes-driver 2026/07/05 17:38:15 17:36:40 INFO Pod/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607-driver: Created - Created container: spark-kubernetes-driver 2026/07/05 17:38:15 17:36:40 INFO Pod/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607-driver: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/05 17:38:15 17:36:39 INFO Pod/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607-driver: Scheduled - Successfully assigned redhat-ods-applications/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607-driver to ip-10-0-143-231.ec2.internal 2026/07/05 17:38:15 17:36:39 WARN Pod/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-b7739d9f335a844a-conf-map" not found 2026/07/05 17:38:15 17:35:59 WARN ReplicaSet/rhods-dashboard-6c65779f4b: FailedCreate - Error creating: pods "rhods-dashboard-6c65779f4b-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 2026/07/05 17:38:15 17:35:59 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-6c65779f4b from 0 to 2 2026/07/05 17:38:15 17:35:58 INFO Pod/dashboard-redirect-b564449f8-vk6vt: Killing - Stopping container nginx 2026/07/05 17:38:15 17:35:58 INFO Pod/dashboard-redirect-b564449f8-jxpdv: Killing - Stopping container nginx 2026/07/05 17:38:15 17:35:46 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkDriverCompleted - Driver spark-pi-d959b8e96nfs74re5dgg-driver completed 2026/07/05 17:38:15 17:35:43 INFO Pod/spark-pi-8b56089f3359686c-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/05 17:38:15 17:35:43 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkExecutorCompleted - Executor spark-pi-8b56089f3359686c-exec-1 completed 2026/07/05 17:38:15 17:35:31 INFO Pod/kuberay-operator-74d759f546-ljsv9: Killing - Stopping container kuberay-operator 2026/07/05 17:38:15 17:35:30 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkExecutorRunning - Executor spark-pi-8b56089f3359686c-exec-1 is running 2026/07/05 17:38:15 17:35:29 INFO Pod/spark-pi-8b56089f3359686c-exec-1: AddedInterface - Add eth0 [10.133.0.121/23] from ovn-kubernetes 2026/07/05 17:38:15 17:35:29 INFO Pod/spark-pi-8b56089f3359686c-exec-1: Started - Started container spark-kubernetes-executor 2026/07/05 17:38:15 17:35:29 INFO Pod/spark-pi-8b56089f3359686c-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/05 17:38:15 17:35:29 INFO Pod/spark-pi-8b56089f3359686c-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/05 17:38:15 17:35:28 INFO Pod/spark-pi-8b56089f3359686c-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-8b56089f3359686c-exec-1 to ip-10-0-143-231.ec2.internal 2026/07/05 17:38:15 17:35:28 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkExecutorPending - Executor spark-pi-8b56089f3359686c-exec-1 is pending 2026/07/05 17:38:15 17:35:28 INFO Lease/ray-operator-leader: LeaderElection - kuberay-operator-74d759f546-ljsv9_13210562-1da5-4cca-8090-9b58f50ca77f became leader 2026/07/05 17:38:15 17:35:28 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-bc44d7c7b-h9kc7_3d60e20d-76d4-403f-839c-9b4c308da487 became leader 2026/07/05 17:38:15 17:35:19 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkDriverRunning - Driver spark-pi-d959b8e96nfs74re5dgg-driver is running 2026/07/05 17:38:15 17:35:19 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Pulled - Successfully pulled image "quay.io/opendatahub/data-processing:Spark-v4.0.1" in 20.724s (20.724s including waiting). Image size: 1778928657 bytes. 2026/07/05 17:38:15 17:35:19 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Created - Created container: spark-kubernetes-driver 2026/07/05 17:38:15 17:35:19 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Started - Started container spark-kubernetes-driver 2026/07/05 17:38:15 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container mlflow-ui 2026/07/05 17:38:15 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" in 600ms (600ms including waiting). Image size: 177813676 bytes. 2026/07/05 17:38:15 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: mlflow-ui 2026/07/05 17:38:15 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" 2026/07/05 17:38:15 17:35:15 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" in 490ms (490ms including waiting). Image size: 185088364 bytes. 2026/07/05 17:38:15 17:35:14 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container maas-ui 2026/07/05 17:38:15 17:35:14 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: maas-ui 2026/07/05 17:38:15 17:35:14 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" in 526ms (526ms including waiting). Image size: 187798114 bytes. 2026/07/05 17:38:15 17:35:14 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" 2026/07/05 17:38:15 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" 2026/07/05 17:38:15 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" 2026/07/05 17:38:15 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: model-registry-ui 2026/07/05 17:38:15 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container model-registry-ui 2026/07/05 17:38:15 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" in 529ms (529ms including waiting). Image size: 182434600 bytes. 2026/07/05 17:38:15 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" in 488ms (488ms including waiting). Image size: 241238855 bytes. 2026/07/05 17:38:15 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: gen-ai-ui 2026/07/05 17:38:15 17:35:13 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container gen-ai-ui 2026/07/05 17:38:15 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: kube-rbac-proxy 2026/07/05 17:38:15 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" in 616ms (616ms including waiting). Image size: 799818131 bytes. 2026/07/05 17:38:15 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container kube-rbac-proxy 2026/07/05 17:38:15 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" 2026/07/05 17:38:15 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Created - Created container: rhods-dashboard 2026/07/05 17:38:15 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Started - Started container rhods-dashboard 2026/07/05 17:38:15 17:35:12 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:ac5537f459cd812a12e8f6ae13001150e29954913007b1790a44b15fb75f4926" already present on machine 2026/07/05 17:38:15 17:35:11 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" 2026/07/05 17:38:15 17:35:11 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: AddedInterface - Add eth0 [10.134.0.72/23] from ovn-kubernetes 2026/07/05 17:38:15 17:35:10 INFO Pod/rhods-dashboard-6c65779f4b-fmwmv: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-6c65779f4b-fmwmv to ip-10-0-131-75.ec2.internal 2026/07/05 17:38:15 17:35:10 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Started - Started container manager 2026/07/05 17:38:15 17:35:10 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Created - Created container: manager 2026/07/05 17:38:15 17:35:10 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Pulled - Successfully pulled image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" in 613ms (613ms including waiting). Image size: 173525457 bytes. 2026/07/05 17:38:15 17:35:10 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Started - Started container manager 2026/07/05 17:38:15 17:35:10 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Created - Created container: manager 2026/07/05 17:38:15 17:35:10 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" in 500ms (500ms including waiting). Image size: 236362435 bytes. 2026/07/05 17:38:15 17:35:09 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Created - Created container: controller 2026/07/05 17:38:15 17:35:09 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Pulling - Pulling image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:e5fffe3d04f1e4ed913bf7b66d5e98bd8582752ea19663268342efd23d662a47" 2026/07/05 17:38:15 17:35:09 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: AddedInterface - Add eth0 [10.134.0.70/23] from ovn-kubernetes 2026/07/05 17:38:15 17:35:09 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:bfa09b5146c891592ee70e04dbca84c2336206e13bb5aa10e9c56703b504bd1b" 2026/07/05 17:38:15 17:35:09 INFO Pod/notebook-controller-deployment-6b68db7cbd-dp87m: Pulling - Pulling image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" 2026/07/05 17:38:15 17:35:09 INFO Pod/notebook-controller-deployment-6b68db7cbd-dp87m: Pulled - Successfully pulled image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:2001b58be2fd5aa4114c87c379960e1bc93703d762a6ed165799781104488e58" in 469ms (469ms including waiting). Image size: 168349711 bytes. 2026/07/05 17:38:15 17:35:09 INFO Pod/kuberay-operator-74d759f546-ljsv9: Started - Started container kuberay-operator 2026/07/05 17:38:15 17:35:09 INFO Pod/kuberay-operator-74d759f546-ljsv9: AddedInterface - Add eth0 [10.134.0.71/23] from ovn-kubernetes 2026/07/05 17:38:15 17:35:09 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: AddedInterface - Add eth0 [10.134.0.69/23] from ovn-kubernetes 2026/07/05 17:38:15 17:35:09 INFO Pod/kuberay-operator-74d759f546-ljsv9: Created - Created container: kuberay-operator 2026/07/05 17:38:15 17:35:09 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Pulled - Successfully pulled image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" in 1m23.925s (1m23.925s including waiting). Image size: 2736477781 bytes. 2026/07/05 17:38:15 17:35:09 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Started - Started container controller 2026/07/05 17:38:15 17:35:09 INFO Pod/kuberay-operator-74d759f546-ljsv9: Pulled - Container image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" already present on machine 2026/07/05 17:38:15 17:35:09 INFO Pod/notebook-controller-deployment-6b68db7cbd-dp87m: AddedInterface - Add eth0 [10.134.0.68/23] from ovn-kubernetes 2026/07/05 17:38:15 17:35:09 WARN Pod/notebook-controller-deployment-6b68db7cbd-dp87m: Failed - Error: configmap "notebook-controller-config" not found 2026/07/05 17:38:15 17:35:07 WARN Pod/kuberay-operator-74d759f546-g4x5g: Unhealthy - Readiness probe failed: Get "http://10.134.0.66:8080/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/05 17:38:15 17:34:59 WARN Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Unhealthy - Readiness probe failed: Get "http://10.134.0.61:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/05 17:38:15 17:34:59 WARN Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Unhealthy - Readiness probe failed: Get "http://10.134.0.60:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/05 17:38:15 17:34:58 WARN Pod/rhods-dashboard-6c65779f4b-fmwmv: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/07/05 17:38:15 17:34:58 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/05 17:38:15 17:34:58 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: AddedInterface - Add eth0 [10.133.0.120/23] from ovn-kubernetes 2026/07/05 17:38:15 17:34:58 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Pulling - Pulling image "quay.io/opendatahub/data-processing:Spark-v4.0.1" 2026/07/05 17:38:15 17:34:58 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Created - Created container: webhook 2026/07/05 17:38:15 17:34:58 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Started - Started container webhook 2026/07/05 17:38:15 17:34:58 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: AddedInterface - Add eth0 [10.133.0.119/23] from ovn-kubernetes 2026/07/05 17:38:15 17:34:56 WARN Pod/kuberay-operator-74d759f546-g4x5g: Unhealthy - Readiness probe failed: Get "http://10.134.0.66:8080/metrics": dial tcp 10.134.0.66:8080: connect: connection refused 2026/07/05 17:38:15 17:34:54 WARN Pod/kuberay-operator-74d759f546-llk9f: Unhealthy - Readiness probe failed: Get "http://10.133.0.118:8080/metrics": dial tcp 10.133.0.118:8080: connect: connection refused 2026/07/05 17:38:15 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" in 462ms (462ms including waiting). Image size: 185088364 bytes. 2026/07/05 17:38:15 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:223cf7fb6040bbf57b2445d0bc6a1b5043d4bcf54c3bd272700791594a6c2623" 2026/07/05 17:38:15 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container mlflow-ui 2026/07/05 17:38:15 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: mlflow-ui 2026/07/05 17:38:15 17:34:44 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" in 394ms (394ms including waiting). Image size: 177813676 bytes. 2026/07/05 17:38:15 17:34:43 INFO Pod/spark-pi-d959b8e96nfs74re5dgg-driver: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-d959b8e96nfs74re5dgg-driver to ip-10-0-143-231.ec2.internal 2026/07/05 17:38:15 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container maas-ui 2026/07/05 17:38:15 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" 2026/07/05 17:38:15 17:34:43 WARN Pod/spark-pi-d959b8e96nfs74re5dgg-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d959b8e96nfs74re5dgg-3380d59f3358a115-driver-podspec-conf-map" not found 2026/07/05 17:38:15 17:34:43 WARN Pod/spark-pi-d959b8e96nfs74re5dgg-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-89627a9f3358b879-conf-map" not found 2026/07/05 17:38:15 17:34:43 INFO SparkApplication/spark-pi-d959b8e96nfs74re5dgg: SparkApplicationSubmitted - SparkApplication spark-pi-d959b8e96nfs74re5dgg was submitted successfully 2026/07/05 17:38:15 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: model-registry-ui 2026/07/05 17:38:15 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container model-registry-ui 2026/07/05 17:38:15 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" 2026/07/05 17:38:15 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:b3572eaf1dcda4027027901e592df26307a6135b9c5b4778d15a711bef52c280" in 381ms (381ms including waiting). Image size: 241238855 bytes. 2026/07/05 17:38:15 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: gen-ai-ui 2026/07/05 17:38:15 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container gen-ai-ui 2026/07/05 17:38:15 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:feee294f8bab936cc5684e267a4a065c01ebb5d554ca88cfe870fc61b9e0a01d" 2026/07/05 17:38:15 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:7052d754cf9ab54cf8e9b09bfbc58b888aa0e5c3f60bbd49abd46a97f2fad9b3" in 366ms (366ms including waiting). Image size: 187798114 bytes. 2026/07/05 17:38:15 17:34:43 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: maas-ui 2026/07/05 17:38:15 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:ac5537f459cd812a12e8f6ae13001150e29954913007b1790a44b15fb75f4926" already present on machine 2026/07/05 17:38:15 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: kube-rbac-proxy 2026/07/05 17:38:15 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container rhods-dashboard 2026/07/05 17:38:15 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Created - Created container: rhods-dashboard 2026/07/05 17:38:15 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" in 456ms (456ms including waiting). Image size: 799818131 bytes. 2026/07/05 17:38:15 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" in 447ms (447ms including waiting). Image size: 182434600 bytes. 2026/07/05 17:38:15 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:a87537db8e31c592fff6c05e0a823e0c37f6f6426d19d69207e15f515476c928" 2026/07/05 17:38:15 17:34:42 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Started - Started container kube-rbac-proxy 2026/07/05 17:38:15 17:34:41 WARN Pod/rhods-dashboard-6c65779f4b-fmwmv: FailedScheduling - 0/3 nodes are available: 3 Insufficient cpu. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. 2026/07/05 17:38:15 17:34:41 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-6c65779f4b-lstrb to ip-10-0-137-37.ec2.internal 2026/07/05 17:38:15 17:34:41 INFO ReplicaSet/rhods-dashboard-6c65779f4b: SuccessfulCreate - Created pod: rhods-dashboard-6c65779f4b-fmwmv 2026/07/05 17:38:15 17:34:41 INFO ReplicaSet/rhods-dashboard-6c65779f4b: SuccessfulCreate - Created pod: rhods-dashboard-6c65779f4b-lstrb 2026/07/05 17:38:15 17:34:41 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:29d00e57fc50806281cb490815e8dda6a58845704d5d8ff4ec5444fee55e0a67" 2026/07/05 17:38:15 17:34:41 INFO Pod/rhods-dashboard-6c65779f4b-lstrb: AddedInterface - Add eth0 [10.132.0.50/23] from ovn-kubernetes 2026/07/05 17:38:15 17:34:40 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-6c65779f4b from 0 to 2 2026/07/05 17:38:15 17:34:36 INFO Pod/kuberay-operator-74d759f546-ljsv9: Scheduled - Successfully assigned redhat-ods-applications/kuberay-operator-74d759f546-ljsv9 to ip-10-0-131-75.ec2.internal 2026/07/05 17:38:15 17:34:36 INFO ReplicaSet/kuberay-operator-74d759f546: SuccessfulCreate - Created pod: kuberay-operator-74d759f546-ljsv9 2026/07/05 17:38:15 17:34:36 INFO Deployment/kuberay-operator: ScalingReplicaSet - Scaled up replica set kuberay-operator-74d759f546 from 0 to 1 2026/07/05 17:38:15 17:34:35 INFO Pod/kuberay-operator-74d759f546-g4x5g: Killing - Stopping container kuberay-operator 2026/07/05 17:38:15 17:34:35 INFO Pod/kuberay-operator-74d759f546-llk9f: Killing - Stopping container kuberay-operator 2026/07/05 17:38:15 17:34:06 WARN Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Unhealthy - Readiness probe failed: Get "http://10.133.0.104:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/05 17:38:15 17:34:05 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-h9kc7 to ip-10-0-131-75.ec2.internal 2026/07/05 17:38:15 17:34:05 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-7jxl6: Scheduled - Successfully assigned redhat-ods-applications/odh-notebook-controller-manager-6f96b94f4c-7jxl6 to ip-10-0-131-75.ec2.internal 2026/07/05 17:38:15 17:34:05 INFO Pod/spark-operator-webhook-5f8768dbfd-wx56c: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-wx56c to ip-10-0-143-231.ec2.internal 2026/07/05 17:38:15 17:34:05 INFO Deployment/odh-notebook-controller-manager: ScalingReplicaSet - Scaled up replica set odh-notebook-controller-manager-6f96b94f4c from 0 to 1 2026/07/05 17:38:15 17:34:05 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 1 to 2 2026/07/05 17:38:15 17:34:05 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-wx56c 2026/07/05 17:38:15 17:34:05 INFO ReplicaSet/llmisvc-controller-manager-bc44d7c7b: SuccessfulCreate - Created pod: llmisvc-controller-manager-bc44d7c7b-h9kc7 2026/07/05 17:38:15 17:34:05 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-bc44d7c7b from 0 to 1 2026/07/05 17:38:15 17:34:05 INFO ReplicaSet/odh-notebook-controller-manager-6f96b94f4c: SuccessfulCreate - Created pod: odh-notebook-controller-manager-6f96b94f4c-7jxl6 2026/07/05 17:38:15 17:34:04 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-tmjpf: Killing - Stopping container manager 2026/07/05 17:38:15 17:34:04 WARN Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Unhealthy - Readiness probe failed: Get "http://10.134.0.60:8081/readyz": dial tcp 10.134.0.60:8081: connect: connection refused 2026/07/05 17:38:15 17:34:04 INFO Pod/odh-notebook-controller-manager-6f96b94f4c-kswrx: Killing - Stopping container manager 2026/07/05 17:38:15 17:34:03 INFO Pod/llmisvc-controller-manager-bc44d7c7b-cr5pl: Killing - Stopping container manager 2026/07/05 17:38:15 17:34:03 INFO Pod/llmisvc-controller-manager-bc44d7c7b-p794f: Killing - Stopping container manager 2026/07/05 17:38:15 17:34:02 INFO Pod/kuberay-operator-74d759f546-llk9f: Pulled - Successfully pulled image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" in 1.407s (1.407s including waiting). Image size: 179205940 bytes. 2026/07/05 17:38:15 17:34:02 INFO Pod/kuberay-operator-74d759f546-llk9f: Created - Created container: kuberay-operator 2026/07/05 17:38:15 17:34:02 INFO Pod/kuberay-operator-74d759f546-llk9f: Started - Started container kuberay-operator 2026/07/05 17:38:15 17:34:02 INFO Lease/ray-operator-leader: LeaderElection - kuberay-operator-74d759f546-g4x5g_34d876db-58de-4ab5-ab74-052534dfbdba became leader 2026/07/05 17:38:15 17:34:01 INFO Pod/kuberay-operator-74d759f546-llk9f: AddedInterface - Add eth0 [10.133.0.118/23] from ovn-kubernetes 2026/07/05 17:38:15 17:34:01 INFO Pod/kuberay-operator-74d759f546-llk9f: Pulling - Pulling image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" 2026/07/05 17:38:15 17:34:00 INFO Pod/kuberay-operator-74d759f546-llk9f: Scheduled - Successfully assigned redhat-ods-applications/kuberay-operator-74d759f546-llk9f to ip-10-0-143-231.ec2.internal 2026/07/05 17:38:15 17:34:00 INFO Pod/notebook-controller-deployment-6b68db7cbd-dp87m: Scheduled - Successfully assigned redhat-ods-applications/notebook-controller-deployment-6b68db7cbd-dp87m to ip-10-0-131-75.ec2.internal 2026/07/05 17:38:15 17:34:00 INFO Deployment/kuberay-operator: ScalingReplicaSet - Scaled up replica set kuberay-operator-74d759f546 from 1 to 2 2026/07/05 17:38:15 17:34:00 INFO ReplicaSet/kuberay-operator-74d759f546: SuccessfulCreate - Created pod: kuberay-operator-74d759f546-llk9f 2026/07/05 17:38:15 17:34:00 INFO ReplicaSet/notebook-controller-deployment-6b68db7cbd: SuccessfulCreate - Created pod: notebook-controller-deployment-6b68db7cbd-dp87m 2026/07/05 17:38:15 17:34:00 INFO Deployment/notebook-controller-deployment: ScalingReplicaSet - Scaled up replica set notebook-controller-deployment-6b68db7cbd from 0 to 1 2026/07/05 17:38:15 17:33:59 INFO Pod/notebook-controller-deployment-6b68db7cbd-6wpjx: Killing - Stopping container manager 2026/07/05 17:38:15 17:33:59 INFO Pod/notebook-controller-deployment-6b68db7cbd-zzfr7: Killing - Stopping container manager 2026/07/05 17:38:15 17:33:45 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-jdvzx to ip-10-0-131-75.ec2.internal 2026/07/05 17:38:15 17:33:45 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-jdvzx 2026/07/05 17:38:15 17:33:45 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 1 to 2 2026/07/05 17:38:15 17:33:45 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: AddedInterface - Add eth0 [10.134.0.67/23] from ovn-kubernetes 2026/07/05 17:38:15 17:33:45 INFO Pod/spark-operator-controller-5dd7d48f87-jdvzx: Pulling - Pulling image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" 2026/07/05 17:38:15 17:33:44 INFO Pod/kuberay-operator-74d759f546-g4x5g: Pulled - Successfully pulled image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" in 1.631s (1.631s including waiting). Image size: 179205940 bytes. 2026/07/05 17:38:15 17:33:44 INFO Pod/kuberay-operator-74d759f546-g4x5g: Started - Started container kuberay-operator 2026/07/05 17:38:15 17:33:44 INFO Pod/kuberay-operator-74d759f546-g4x5g: Created - Created container: kuberay-operator 2026/07/05 17:38:15 17:33:42 INFO Pod/kuberay-operator-74d759f546-g4x5g: Pulling - Pulling image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:4d440d0d87798a72522688424f6a4d24009906abbd798c76b4838e19e22eefde" 2026/07/05 17:38:15 17:33:42 INFO Pod/kuberay-operator-74d759f546-g4x5g: AddedInterface - Add eth0 [10.134.0.66/23] from ovn-kubernetes 2026/07/05 17:38:15 17:33:41 INFO Pod/kuberay-operator-74d759f546-g4x5g: Scheduled - Successfully assigned redhat-ods-applications/kuberay-operator-74d759f546-g4x5g to ip-10-0-131-75.ec2.internal 2026/07/05 17:38:15 17:33:41 INFO Deployment/kuberay-operator: ScalingReplicaSet - Scaled up replica set kuberay-operator-74d759f546 from 0 to 1 2026/07/05 17:38:15 17:33:41 INFO ReplicaSet/kuberay-operator-74d759f546: SuccessfulCreate - Created pod: kuberay-operator-74d759f546-g4x5g 2026/07/05 17:38:15 17:33:41 WARN Pod/kuberay-operator-74d759f546-g4x5g: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 2026/07/05 17:38:15 17:33:38 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-588b4b787c-4nm25_e7ea5d10-a5b5-4cb5-b915-1e85d8cc1b75 became leader 2026/07/05 17:38:15 17:33:38 INFO Lease/spark-operator-webhook-lock: LeaderElection - spark-operator-webhook-5f8768dbfd-rgktb_b0ba836a-be37-4a30-a983-1f197f6c49e2 became leader 2026/07/05 17:38:15 17:33:36 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-5dd7d48f87-s2gzm_70a06366-6c28-43ad-8a34-79526b6f18e7 became leader 2026/07/05 17:38:15 17:33:30 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5f7b7f7655-mf2jg_d4d6fa96-e67f-491c-ba63-6964158add9e became leader 2026/07/05 17:38:15 17:33:27 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:33:27 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:33:27 INFO Pod/data-science-pipelines-operator-controller-manager-595ffd98nx4m: Killing - Stopping container manager 2026/07/05 17:38:15 17:33:27 INFO Pod/kubeflow-trainer-controller-manager-8648b6bb44-l65kd: Killing - Stopping container manager 2026/07/05 17:38:15 17:33:26 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:33:26 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:33:25 INFO Pod/kubeflow-training-operator-7c59bdd544-qns74: Killing - Stopping container training-operator 2026/07/05 17:38:15 17:33:25 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled down replica set rhods-dashboard-6c65779f4b from 2 to 1 2026/07/05 17:38:15 17:33:25 INFO ReplicaSet/rhods-dashboard-6c65779f4b: SuccessfulDelete - Deleted pod: rhods-dashboard-6c65779f4b-5zgbz 2026/07/05 17:38:15 17:33:24 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:33:24 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Created - Created container: manager 2026/07/05 17:38:15 17:33:22 INFO Pod/kuberay-operator-74d759f546-b82w5: Killing - Stopping container kuberay-operator 2026/07/05 17:38:15 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: AddedInterface - Add eth0 [10.134.0.65/23] from ovn-kubernetes 2026/07/05 17:38:15 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" 2026/07/05 17:38:15 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:224bc3f088056de1c870b3ce193f1ceb26811a167430ba9a2151b5e8987150be" in 385ms (385ms including waiting). Image size: 238427353 bytes. 2026/07/05 17:38:15 17:33:22 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Started - Started container manager 2026/07/05 17:38:15 17:33:21 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-588b4b787c-4nm25 to ip-10-0-131-75.ec2.internal 2026/07/05 17:38:15 17:33:21 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-588b4b787c from 0 to 1 2026/07/05 17:38:15 17:33:21 INFO ReplicaSet/kserve-controller-manager-588b4b787c: SuccessfulCreate - Created pod: kserve-controller-manager-588b4b787c-4nm25 2026/07/05 17:38:15 17:33:19 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/05 17:38:15 17:33:19 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Created - Created container: controller 2026/07/05 17:38:15 17:33:19 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-8648b6bb44-l65kd_20a05650-424e-43df-a54f-c97a806596e4 became leader 2026/07/05 17:38:15 17:33:19 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Started - Started container webhook 2026/07/05 17:38:15 17:33:19 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Created - Created container: webhook 2026/07/05 17:38:15 17:33:19 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/05 17:38:15 17:33:19 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: AddedInterface - Add eth0 [10.133.0.117/23] from ovn-kubernetes 2026/07/05 17:38:15 17:33:19 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: AddedInterface - Add eth0 [10.133.0.116/23] from ovn-kubernetes 2026/07/05 17:38:15 17:33:19 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Started - Started container controller 2026/07/05 17:38:15 17:33:18 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:33:18 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/05 17:38:15 17:33:18 INFO Pod/spark-operator-webhook-5f8768dbfd-rgktb: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-rgktb to ip-10-0-143-231.ec2.internal 2026/07/05 17:38:15 17:33:18 INFO Pod/spark-operator-controller-5dd7d48f87-s2gzm: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-s2gzm to ip-10-0-143-231.ec2.internal 2026/07/05 17:38:15 17:33:18 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 0 to 1 2026/07/05 17:38:15 17:33:18 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-s2gzm 2026/07/05 17:38:15 17:33:18 WARN ReplicaSet/spark-operator-controller-5dd7d48f87: FailedCreate - Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 2026/07/05 17:38:15 17:33:18 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 0 to 1 2026/07/05 17:38:15 17:33:18 INFO Pod/kserve-controller-manager-588b4b787c-gk7pm: Killing - Stopping container manager 2026/07/05 17:38:15 17:33:18 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-rgktb 2026/07/05 17:38:15 === RESOURCE QUOTAS === 2026/07/05 17:38:15 No resource quotas found in monitored namespaces 2026/07/05 17:38:15 Classification for group 1: infrastructure/storage (code=1006, confidence=medium) 2026/07/05 17:38:15 Evidence: event Pod/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-b7739d9f335a844a-conf-map" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/scheduled-spark-pi-d959buu96nfs74re5dh0-1783272985000248607-driver: FailedMount - MountVolume.SetUp failed for volume \"spark-conf-volume-driver\" : configmap \"spark-drv-b7739d9f335a844a-conf-map\" not found"],"confidence":"medium"} 2026/07/05 17:38:15 Diagnostics complete for failed test: group 1 controller_test.go:763: Stopping: group 1 test failed. 2026/07/05 17:38:15 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/05 17:38:15 Circuit breaker: infrastructure/unknown failure recorded (2/3) 2026/07/05 17:44:37 TEST FAILURE DETECTED: group 2 2026/07/05 17:44:37 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/05 17:44:37 === CLUSTER STATE === 2026/07/05 17:44:37 No nodes found in cluster 2026/07/05 17:44:37 === DEPLOYMENTS === 2026/07/05 17:44:37 Failed to collect deployment data: deployments not ready: redhat-ods-applications/kserve-controller-manager (0/1), redhat-ods-applications/llmisvc-controller-manager (0/1), redhat-ods-applications/maas-api (0/1), redhat-ods-applications/maas-controller (0/1), redhat-ods-applications/mlflow-operator-controller-manager (0/1), redhat-ods-applications/model-registry-operator-controller-manager (0/1), redhat-ods-applications/model-serving-api (0/2), redhat-ods-applications/odh-model-controller (0/2), redhat-ods-applications/ogx-k8s-operator-controller-manager (0/1), redhat-ods-applications/postgres (0/1), redhat-ods-applications/trustyai-service-operator-controller-manager (0/1), redhat-ods-applications/workload-variant-autoscaler-controller-manager (0/2) 2026/07/05 17:44:37 === PODS === 2026/07/05 17:44:37 Failed to collect pod data: redhat-ods-applications/maas-api-557679ff4c-47gbq: phase=Pending; redhat-ods-applications/maas-controller-5b858f66fc-gl9nd: phase=Pending; redhat-ods-applications/mlflow-operator-controller-manager-57f4cd4d95-kfl6d: phase=Pending; redhat-ods-applications/model-registry-operator-controller-manager-b67847846-txw9d: phase=Pending; redhat-ods-applications/model-serving-api-5c897df776-h8bvz: phase=Pending; redhat-ods-applications/odh-model-controller-566d8764f4-hs7g4: phase=Pending; redhat-ods-applications/ogx-k8s-operator-controller-manager-94b957b46-8xj29: phase=Pending; redhat-ods-applications/postgres-69fb4594cf-2cq2f: phase=Pending; redhat-ods-applications/trustyai-service-operator-controller-manager-7bcdbcb4fd-pxc8f: phase=Pending; redhat-ods-applications/workload-variant-autoscaler-controller-manager-67777f585d-x2zbx: phase=Pending 2026/07/05 17:44:37 === OPERATOR STATUS === 2026/07/05 17:44:37 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-654b676654-mwcrr: Pending; operator pod rhods-operator-654b676654-q27p4: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-b289v: Pending; dependent jobset: deployment 0/1 ready; pod jobset-operator-56d489b4b8-g9zzk: Pending; dependent leader-worker-set: deployment 0/1 ready; pod openshift-lws-operator-697fbb9648-kmfqb: Pending; dependent cert-manager: deployment 0/1 ready; pod cert-manager-operator-controller-manager-55c444bf86-5l8dr: Pending; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-dk8gv: Pending 2026/07/05 17:44:37 === DSCI STATUS === 2026/07/05 17:44:37 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/05 17:44:37 === DSC STATUS === 2026/07/05 17:44:37 DSC default-dsc: 2026/07/05 17:44:37 Ready: True - 2026/07/05 17:44:37 ProvisioningSucceeded: True - 2026/07/05 17:44:37 ComponentsReady: True - 2026/07/05 17:44:37 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 DashboardReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 KserveLLMInferenceServiceDependencies: True - 2026/07/05 17:44:37 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/05 17:44:37 KserveReady: True - 2026/07/05 17:44:37 KueueReady: True - 2026/07/05 17:44:37 LLM-D-WVADependencies: True - 2026/07/05 17:44:37 MLflowOperatorReady: True - 2026/07/05 17:44:37 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/07/05 17:44:37 ModelControllerReady: True - 2026/07/05 17:44:37 ModelRegistryReady: True - 2026/07/05 17:44:37 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/05 17:44:37 OGXReady: True - 2026/07/05 17:44:37 RayReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 TrainerReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 TrustyAIReady: True - 2026/07/05 17:44:37 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 === RECENT EVENTS === 2026/07/05 17:44:37 17:44:36 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-67777f585d: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:44:37 17:44:35 WARN ReplicaSet/odh-model-controller-566d8764f4: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:44:37 17:44:35 WARN ReplicaSet/kserve-controller-manager-588b4b787c: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:44:37 17:44:35 WARN ReplicaSet/model-serving-api-5c897df776: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:44:37 17:44:35 WARN ReplicaSet/llmisvc-controller-manager-bc44d7c7b: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:44:37 17:44:11 WARN Service/llmisvc-webhook-server-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/llmisvc-webhook-server-service: skipping Pod llmisvc-controller-manager-bc44d7c7b-h9kc7 for Service redhat-ods-applications/llmisvc-webhook-server-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/kserve-webhook-server-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/kserve-webhook-server-service: skipping Pod kserve-controller-manager-588b4b787c-4nm25 for Service redhat-ods-applications/kserve-webhook-server-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/model-serving-api: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/model-serving-api: skipping Pod model-serving-api-5c897df776-2pwnq for Service redhat-ods-applications/model-serving-api: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/kserve-controller-manager-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/kserve-controller-manager-service: skipping Pod kserve-controller-manager-588b4b787c-4nm25 for Service redhat-ods-applications/kserve-controller-manager-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/llmisvc-controller-manager-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/llmisvc-controller-manager-service: skipping Pod llmisvc-controller-manager-bc44d7c7b-h9kc7 for Service redhat-ods-applications/llmisvc-controller-manager-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/odh-model-controller-metrics-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/odh-model-controller-metrics-service: skipping Pod odh-model-controller-566d8764f4-ddc4h for Service redhat-ods-applications/odh-model-controller-metrics-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/odh-model-controller-webhook-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/odh-model-controller-webhook-service: skipping Pod odh-model-controller-566d8764f4-ddc4h for Service redhat-ods-applications/odh-model-controller-webhook-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/workload-variant-autoscaler-controller-manager-metrics-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/workload-variant-autoscaler-controller-manager-metrics-service: skipping Pod workload-variant-autoscaler-controller-manager-67777f585d-5kz8q for Service redhat-ods-applications/workload-variant-autoscaler-controller-manager-metrics-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/kserve-controller-manager-metrics-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/kserve-controller-manager-metrics-service: skipping Pod kserve-controller-manager-588b4b787c-4nm25 for Service redhat-ods-applications/kserve-controller-manager-metrics-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:43:03 WARN Pod/maas-api-557679ff4c-47gbq: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:43:03 WARN Pod/odh-model-controller-566d8764f4-hs7g4: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:43:03 WARN Pod/model-registry-operator-controller-manager-b67847846-txw9d: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:43:03 INFO ReplicaSet/odh-model-controller-566d8764f4: SuccessfulCreate - Created pod: odh-model-controller-566d8764f4-hs7g4 2026/07/05 17:44:37 17:43:03 INFO ReplicaSet/maas-api-557679ff4c: SuccessfulCreate - Created pod: maas-api-557679ff4c-47gbq 2026/07/05 17:44:37 17:43:03 INFO ReplicaSet/model-registry-operator-controller-manager-b67847846: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-b67847846-txw9d 2026/07/05 17:44:37 17:43:02 WARN Pod/workload-variant-autoscaler-controller-manager-67777f585d-x2zbx: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:43:02 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-67777f585d: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-67777f585d-x2zbx 2026/07/05 17:44:37 17:42:22 WARN Pod/postgres-69fb4594cf-2cq2f: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:42:22 WARN Pod/model-serving-api-5c897df776-h8bvz: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:42:22 INFO ReplicaSet/postgres-69fb4594cf: SuccessfulCreate - Created pod: postgres-69fb4594cf-2cq2f 2026/07/05 17:44:37 17:42:22 INFO ReplicaSet/model-serving-api-5c897df776: SuccessfulCreate - Created pod: model-serving-api-5c897df776-h8bvz 2026/07/05 17:44:37 17:42:21 WARN Pod/mlflow-operator-controller-manager-57f4cd4d95-kfl6d: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:42:21 WARN Pod/ogx-k8s-operator-controller-manager-94b957b46-8xj29: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:42:21 WARN Pod/maas-controller-5b858f66fc-gl9nd: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:42:21 INFO ReplicaSet/maas-controller-5b858f66fc: SuccessfulCreate - Created pod: maas-controller-5b858f66fc-gl9nd 2026/07/05 17:44:37 17:42:21 INFO ReplicaSet/mlflow-operator-controller-manager-57f4cd4d95: SuccessfulCreate - Created pod: mlflow-operator-controller-manager-57f4cd4d95-kfl6d 2026/07/05 17:44:37 17:42:21 INFO ReplicaSet/ogx-k8s-operator-controller-manager-94b957b46: SuccessfulCreate - Created pod: ogx-k8s-operator-controller-manager-94b957b46-8xj29 2026/07/05 17:44:37 17:42:20 WARN Pod/trustyai-service-operator-controller-manager-7bcdbcb4fd-pxc8f: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:42:20 INFO ReplicaSet/trustyai-service-operator-controller-manager-7bcdbcb4fd: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-7bcdbcb4fd-pxc8f 2026/07/05 17:44:37 17:41:18 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/kserve-controller-manager-588b4b787c-4nm25 2026/07/05 17:44:37 17:41:18 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q 2026/07/05 17:44:37 17:41:18 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-h9kc7 2026/07/05 17:44:37 17:41:18 INFO Pod/model-serving-api-5c897df776-2pwnq: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/model-serving-api-5c897df776-2pwnq 2026/07/05 17:44:37 17:41:18 INFO Pod/odh-model-controller-566d8764f4-ddc4h: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/odh-model-controller-566d8764f4-ddc4h 2026/07/05 17:44:37 17:41:17 WARN Pod/global-pull-secret-syncer-5l728: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/kube-apiserver-proxy-ip-10-0-143-231.ec2.internal: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/konnectivity-agent-mff9p: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/postgres-69fb4594cf-mz8gk: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/maas-controller-5b858f66fc-xrp5h: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/trustyai-service-operator-controller-manager-7bcdbcb4fd-n65d4: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/model-serving-api-5c897df776-4mkx7: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/mlflow-operator-controller-manager-57f4cd4d95-jlw55: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:12 WARN Pod/model-serving-api-5c897df776-2pwnq: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/kube-apiserver-proxy-ip-10-0-131-75.ec2.internal: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/model-registry-operator-controller-manager-b67847846-s94zx: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/kserve-controller-manager-588b4b787c-4nm25: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/konnectivity-agent-n2m85: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/odh-model-controller-566d8764f4-rw474: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/workload-variant-autoscaler-controller-manager-67777f585d-dck2s: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/maas-api-557679ff4c-rn27g: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/global-pull-secret-syncer-gfzgr: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/odh-model-controller-566d8764f4-ddc4h: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/konnectivity-agent-tcxd2: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/global-pull-secret-syncer-8vvhd: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:39:53 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Started - Started container manager 2026/07/05 17:44:37 17:39:53 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Pulled - Successfully pulled image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:79fa44b245109a0c1d214c8b313dc8172b367dedf75c4691ed1faefa46f41775" in 1.498s (1.498s including waiting). Image size: 195240092 bytes. 2026/07/05 17:44:37 17:39:53 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Created - Created container: manager 2026/07/05 17:44:37 17:39:51 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q to ip-10-0-131-75.ec2.internal 2026/07/05 17:44:37 17:39:51 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-67777f585d: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-67777f585d-5kz8q 2026/07/05 17:44:37 17:39:51 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Pulling - Pulling image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:79fa44b245109a0c1d214c8b313dc8172b367dedf75c4691ed1faefa46f41775" 2026/07/05 17:44:37 17:39:51 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: AddedInterface - Add eth0 [10.134.0.78/23] from ovn-kubernetes 2026/07/05 17:44:37 17:39:50 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-67777f585d from 1 to 2 2026/07/05 17:44:37 === RESOURCE QUOTAS === 2026/07/05 17:44:37 No resource quotas found in monitored namespaces 2026/07/05 17:44:37 Classification for group 2: infrastructure/pod-startup (code=1002, confidence=high) 2026/07/05 17:44:37 Evidence: pod maas-api-557679ff4c-47gbq stuck in Pending for 1m34s FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["pod maas-api-557679ff4c-47gbq stuck in Pending for 1m34s"],"confidence":"high"} 2026/07/05 17:44:37 Diagnostics complete for failed test: group 2 controller_test.go:763: Stopping: group 2 test failed. 2026/07/05 17:44:37 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/05 17:44:37 Circuit breaker: infrastructure/unknown failure recorded (3/3) 2026/07/05 17:44:37 Circuit breaker: threshold reached (3 failures), running confirming health check... 2026/07/05 17:44:37 CIRCUIT BREAKER TRIPPED: Infrastructure failure detected after 3 infrastructure/unknown failures. Cluster health issues: [operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-654b676654-mwcrr: Pending; operator pod rhods-operator-654b676654-q27p4: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-b289v: Pending; dependent jobset: deployment 0/1 ready; pod jobset-operator-56d489b4b8-g9zzk: Pending; dependent leader-worker-set: deployment 0/1 ready; pod openshift-lws-operator-697fbb9648-kmfqb: Pending; dependent cert-manager: deployment 0/1 ready; pod cert-manager-operator-controller-manager-55c444bf86-5l8dr: Pending; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-dk8gv: Pending] --- FAIL: TestOdhOperator/components (794.45s) === RUN TestOdhOperator 2026/07/05 17:26:12 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/05 17:31:22 TEST FAILURE DETECTED: DSCInitialization and DataScienceCluster validation E2E Tests 2026/07/05 17:31:22 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:DSCInitialization and DataScienceCluster validation E2E Tests) === 2026/07/05 17:31:22 === CLUSTER STATE === 2026/07/05 17:31:22 Node: ip-10-0-131-75.ec2.internal 2026/07/05 17:31:22 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:31:22 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:31:22 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:31:22 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:31:22 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/05 17:31:22 Node: ip-10-0-137-37.ec2.internal 2026/07/05 17:31:22 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:31:22 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:31:22 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:31:22 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:31:22 Resources: 7500m CPU, 31236220Ki memory allocatable (of 8000m CPU, 32387196Ki memory capacity) 2026/07/05 17:31:22 Node: ip-10-0-143-231.ec2.internal 2026/07/05 17:31:22 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/05 17:31:22 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/05 17:31:22 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/05 17:31:22 Condition Ready: True - kubelet is posting ready status 2026/07/05 17:31:22 Resources: 7500m CPU, 31236228Ki memory allocatable (of 8000m CPU, 32387204Ki memory capacity) 2026/07/05 17:31:22 === DEPLOYMENTS === 2026/07/05 17:31:22 Namespace: redhat-ods-applications 2026/07/05 17:31:22 Deployment dashboard-redirect: 2/2 (READY) 2026/07/05 17:31:22 Deployment data-science-pipelines-operator-controller-manager: 1/1 (READY) 2026/07/05 17:31:22 Deployment feast-operator-controller-manager: 1/1 (READY) 2026/07/05 17:31:22 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/05 17:31:22 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/05 17:31:22 Deployment kubeflow-training-operator: 1/1 (READY) 2026/07/05 17:31:22 Deployment kuberay-operator: 1/1 (READY) 2026/07/05 17:31:22 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/05 17:31:22 Deployment maas-api: 1/1 (READY) 2026/07/05 17:31:22 Deployment maas-controller: 1/1 (READY) 2026/07/05 17:31:22 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/05 17:31:22 Deployment model-registry-operator-controller-manager: 1/1 (READY) 2026/07/05 17:31:22 Deployment model-serving-api: 1/1 (READY) 2026/07/05 17:31:22 Deployment notebook-controller-deployment: 1/1 (READY) 2026/07/05 17:31:22 Deployment odh-model-controller: 1/1 (READY) 2026/07/05 17:31:22 Deployment odh-notebook-controller-manager: 1/1 (READY) 2026/07/05 17:31:22 Deployment ogx-k8s-operator-controller-manager: 1/1 (READY) 2026/07/05 17:31:22 Deployment postgres: 1/1 (READY) 2026/07/05 17:31:22 Deployment rhods-dashboard: 2/2 (READY) 2026/07/05 17:31:22 Deployment spark-operator-controller: 1/1 (READY) 2026/07/05 17:31:22 Deployment spark-operator-webhook: 1/1 (READY) 2026/07/05 17:31:22 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/05 17:31:22 Namespace: kube-system 2026/07/05 17:31:22 === PODS === 2026/07/05 17:31:22 Namespace kube-system: no problematic pods found 2026/07/05 17:31:22 Namespace: redhat-ods-applications 2026/07/05 17:31:22 Pod maas-api-key-cleanup-29721180-p5qrl: Succeeded 2026/07/05 17:31:22 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/07/05 17:31:22 Pod maas-api-key-cleanup-29721195-9g7pb: Succeeded 2026/07/05 17:31:22 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/07/05 17:31:22 Pod maas-api-key-cleanup-29721210-r94rz: Succeeded 2026/07/05 17:31:22 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/07/05 17:31:22 === LOGS (CURRENT) for container cleanup === 2026/07/05 17:31:22 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/07/05 17:31:22 === END LOGS === 2026/07/05 17:31:22 === LOGS (CURRENT) for container cleanup === 2026/07/05 17:31:22 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/07/05 17:31:22 === END LOGS === 2026/07/05 17:31:22 === LOGS (CURRENT) for container cleanup === 2026/07/05 17:31:22 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/07/05 17:31:22 === END LOGS === 2026/07/05 17:31:22 === OPERATOR STATUS === 2026/07/05 17:31:22 Operator deployment: 3/3 ready 2026/07/05 17:31:22 === DSCI STATUS === 2026/07/05 17:31:22 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/05 17:31:22 === DSC STATUS === 2026/07/05 17:31:22 DSC default-dsc: 2026/07/05 17:31:22 Ready: True - 2026/07/05 17:31:22 ProvisioningSucceeded: True - 2026/07/05 17:31:22 ModulesReady: True - All registered modules have ManagementState Removed or are not configured 2026/07/05 17:31:22 AIPipelinesReady: True - 2026/07/05 17:31:22 ComponentsReady: True - 2026/07/05 17:31:22 DashboardReady: True - 2026/07/05 17:31:22 FeastOperatorReady: True - 2026/07/05 17:31:22 KserveLLMInferenceServiceDependencies: True - 2026/07/05 17:31:22 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/05 17:31:22 KserveReady: True - 2026/07/05 17:31:22 KueueReady: False - Component ManagementState is set to Removed 2026/07/05 17:31:22 LLM-D-WVADependencies: True - 2026/07/05 17:31:22 MLflowOperatorReady: True - 2026/07/05 17:31:22 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/07/05 17:31:22 ModelControllerReady: True - 2026/07/05 17:31:22 ModelRegistryReady: True - 2026/07/05 17:31:22 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/05 17:31:22 OGXReady: True - 2026/07/05 17:31:22 RayReady: True - 2026/07/05 17:31:22 SparkOperatorReady: True - 2026/07/05 17:31:22 TrainerReady: True - 2026/07/05 17:31:22 TrainingOperatorReady: True - 2026/07/05 17:31:22 TrustyAIReady: True - 2026/07/05 17:31:22 WorkbenchesReady: True - 2026/07/05 17:31:22 === RECENT EVENTS === 2026/07/05 17:31:22 17:30:04 INFO Job/maas-api-key-cleanup-29721210: Completed - Job completed 2026/07/05 17:31:22 17:30:04 INFO CronJob/maas-api-key-cleanup: SuccessfulDelete - Deleted job maas-api-key-cleanup-29721165 2026/07/05 17:31:22 17:30:04 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29721210, condition: Complete 2026/07/05 17:31:22 17:30:01 INFO Pod/maas-api-key-cleanup-29721210-r94rz: Created - Created container: cleanup 2026/07/05 17:31:22 17:30:01 INFO Pod/maas-api-key-cleanup-29721210-r94rz: Started - Started container cleanup 2026/07/05 17:31:22 17:30:00 INFO Pod/maas-api-key-cleanup-29721210-r94rz: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29721210-r94rz to ip-10-0-143-231.ec2.internal 2026/07/05 17:31:22 17:30:00 INFO Pod/maas-api-key-cleanup-29721210-r94rz: AddedInterface - Add eth0 [10.133.0.91/23] from ovn-kubernetes 2026/07/05 17:31:22 17:30:00 INFO Pod/maas-api-key-cleanup-29721210-r94rz: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/05 17:31:22 17:30:00 INFO Job/maas-api-key-cleanup-29721210: SuccessfulCreate - Created pod: maas-api-key-cleanup-29721210-r94rz 2026/07/05 17:31:22 17:30:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29721210 2026/07/05 17:31:22 === RESOURCE QUOTAS === 2026/07/05 17:31:22 No resource quotas found in monitored namespaces 2026/07/05 17:31:22 Classification for DSCInitialization and DataScienceCluster validation E2E Tests: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/05 17:31:22 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/05 17:31:22 Diagnostics complete for failed test: DSCInitialization and DataScienceCluster validation E2E Tests controller_test.go:478: Stopping: DSCInitialization and DataScienceCluster validation E2E Tests test failed. 2026/07/05 17:31:22 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/05 17:31:22 Circuit breaker: infrastructure/unknown failure recorded (1/3) 2026/07/05 17:44:37 TEST FAILURE DETECTED: components 2026/07/05 17:44:37 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/05 17:44:37 === CLUSTER STATE === 2026/07/05 17:44:37 No nodes found in cluster 2026/07/05 17:44:37 === DEPLOYMENTS === 2026/07/05 17:44:37 Failed to collect deployment data: deployments not ready: redhat-ods-applications/kserve-controller-manager (0/1), redhat-ods-applications/llmisvc-controller-manager (0/1), redhat-ods-applications/maas-api (0/1), redhat-ods-applications/maas-controller (0/1), redhat-ods-applications/mlflow-operator-controller-manager (0/1), redhat-ods-applications/model-registry-operator-controller-manager (0/1), redhat-ods-applications/model-serving-api (0/2), redhat-ods-applications/odh-model-controller (0/2), redhat-ods-applications/ogx-k8s-operator-controller-manager (0/1), redhat-ods-applications/postgres (0/1), redhat-ods-applications/trustyai-service-operator-controller-manager (0/1), redhat-ods-applications/workload-variant-autoscaler-controller-manager (0/2) 2026/07/05 17:44:37 === PODS === 2026/07/05 17:44:37 Failed to collect pod data: redhat-ods-applications/maas-api-557679ff4c-47gbq: phase=Pending; redhat-ods-applications/maas-controller-5b858f66fc-gl9nd: phase=Pending; redhat-ods-applications/mlflow-operator-controller-manager-57f4cd4d95-kfl6d: phase=Pending; redhat-ods-applications/model-registry-operator-controller-manager-b67847846-txw9d: phase=Pending; redhat-ods-applications/model-serving-api-5c897df776-h8bvz: phase=Pending; redhat-ods-applications/odh-model-controller-566d8764f4-hs7g4: phase=Pending; redhat-ods-applications/ogx-k8s-operator-controller-manager-94b957b46-8xj29: phase=Pending; redhat-ods-applications/postgres-69fb4594cf-2cq2f: phase=Pending; redhat-ods-applications/trustyai-service-operator-controller-manager-7bcdbcb4fd-pxc8f: phase=Pending; redhat-ods-applications/workload-variant-autoscaler-controller-manager-67777f585d-x2zbx: phase=Pending 2026/07/05 17:44:37 === OPERATOR STATUS === 2026/07/05 17:44:37 Failed to collect operator data: operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-654b676654-mwcrr: Pending; operator pod rhods-operator-654b676654-q27p4: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-b289v: Pending; dependent jobset: deployment 0/1 ready; pod jobset-operator-56d489b4b8-g9zzk: Pending; dependent leader-worker-set: deployment 0/1 ready; pod openshift-lws-operator-697fbb9648-kmfqb: Pending; dependent cert-manager: deployment 0/1 ready; pod cert-manager-operator-controller-manager-55c444bf86-5l8dr: Pending; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-dk8gv: Pending 2026/07/05 17:44:37 === DSCI STATUS === 2026/07/05 17:44:37 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/05 17:44:37 === DSC STATUS === 2026/07/05 17:44:37 DSC default-dsc: 2026/07/05 17:44:37 Ready: True - 2026/07/05 17:44:37 ProvisioningSucceeded: True - 2026/07/05 17:44:37 ComponentsReady: True - 2026/07/05 17:44:37 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 DashboardReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 KserveLLMInferenceServiceDependencies: True - 2026/07/05 17:44:37 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/05 17:44:37 KserveReady: True - 2026/07/05 17:44:37 KueueReady: True - 2026/07/05 17:44:37 LLM-D-WVADependencies: True - 2026/07/05 17:44:37 MLflowOperatorReady: True - 2026/07/05 17:44:37 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/07/05 17:44:37 ModelControllerReady: True - 2026/07/05 17:44:37 ModelRegistryReady: True - 2026/07/05 17:44:37 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/05 17:44:37 OGXReady: True - 2026/07/05 17:44:37 RayReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 TrainerReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 TrustyAIReady: True - 2026/07/05 17:44:37 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/05 17:44:37 === RECENT EVENTS === 2026/07/05 17:44:37 17:44:36 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-67777f585d: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:44:37 17:44:35 WARN ReplicaSet/odh-model-controller-566d8764f4: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:44:37 17:44:35 WARN ReplicaSet/kserve-controller-manager-588b4b787c: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:44:37 17:44:35 WARN ReplicaSet/model-serving-api-5c897df776: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:44:37 17:44:35 WARN ReplicaSet/llmisvc-controller-manager-bc44d7c7b: FailedCreate - Error creating: Internal error occurred: failed calling webhook "block-resources.hypershift.openshift.io": failed to call webhook: Post "https://xxx-invalid-service-xxx.default.svc:443/validate?timeout=30s": service "xxx-invalid-service-xxx" not found 2026/07/05 17:44:37 17:44:11 WARN Service/llmisvc-webhook-server-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/llmisvc-webhook-server-service: skipping Pod llmisvc-controller-manager-bc44d7c7b-h9kc7 for Service redhat-ods-applications/llmisvc-webhook-server-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/kserve-webhook-server-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/kserve-webhook-server-service: skipping Pod kserve-controller-manager-588b4b787c-4nm25 for Service redhat-ods-applications/kserve-webhook-server-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/model-serving-api: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/model-serving-api: skipping Pod model-serving-api-5c897df776-2pwnq for Service redhat-ods-applications/model-serving-api: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/kserve-controller-manager-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/kserve-controller-manager-service: skipping Pod kserve-controller-manager-588b4b787c-4nm25 for Service redhat-ods-applications/kserve-controller-manager-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/llmisvc-controller-manager-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/llmisvc-controller-manager-service: skipping Pod llmisvc-controller-manager-bc44d7c7b-h9kc7 for Service redhat-ods-applications/llmisvc-controller-manager-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/odh-model-controller-metrics-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/odh-model-controller-metrics-service: skipping Pod odh-model-controller-566d8764f4-ddc4h for Service redhat-ods-applications/odh-model-controller-metrics-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/odh-model-controller-webhook-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/odh-model-controller-webhook-service: skipping Pod odh-model-controller-566d8764f4-ddc4h for Service redhat-ods-applications/odh-model-controller-webhook-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/workload-variant-autoscaler-controller-manager-metrics-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/workload-variant-autoscaler-controller-manager-metrics-service: skipping Pod workload-variant-autoscaler-controller-manager-67777f585d-5kz8q for Service redhat-ods-applications/workload-variant-autoscaler-controller-manager-metrics-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:44:11 WARN Service/kserve-controller-manager-metrics-service: FailedToUpdateEndpointSlices - Error updating Endpoint Slices for Service redhat-ods-applications/kserve-controller-manager-metrics-service: skipping Pod kserve-controller-manager-588b4b787c-4nm25 for Service redhat-ods-applications/kserve-controller-manager-metrics-service: Node ip-10-0-131-75.ec2.internal Not Found 2026/07/05 17:44:37 17:43:03 WARN Pod/maas-api-557679ff4c-47gbq: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:43:03 WARN Pod/odh-model-controller-566d8764f4-hs7g4: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:43:03 WARN Pod/model-registry-operator-controller-manager-b67847846-txw9d: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:43:03 INFO ReplicaSet/odh-model-controller-566d8764f4: SuccessfulCreate - Created pod: odh-model-controller-566d8764f4-hs7g4 2026/07/05 17:44:37 17:43:03 INFO ReplicaSet/maas-api-557679ff4c: SuccessfulCreate - Created pod: maas-api-557679ff4c-47gbq 2026/07/05 17:44:37 17:43:03 INFO ReplicaSet/model-registry-operator-controller-manager-b67847846: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-b67847846-txw9d 2026/07/05 17:44:37 17:43:02 WARN Pod/workload-variant-autoscaler-controller-manager-67777f585d-x2zbx: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:43:02 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-67777f585d: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-67777f585d-x2zbx 2026/07/05 17:44:37 17:42:22 WARN Pod/postgres-69fb4594cf-2cq2f: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:42:22 WARN Pod/model-serving-api-5c897df776-h8bvz: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:42:22 INFO ReplicaSet/postgres-69fb4594cf: SuccessfulCreate - Created pod: postgres-69fb4594cf-2cq2f 2026/07/05 17:44:37 17:42:22 INFO ReplicaSet/model-serving-api-5c897df776: SuccessfulCreate - Created pod: model-serving-api-5c897df776-h8bvz 2026/07/05 17:44:37 17:42:21 WARN Pod/mlflow-operator-controller-manager-57f4cd4d95-kfl6d: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:42:21 WARN Pod/ogx-k8s-operator-controller-manager-94b957b46-8xj29: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:42:21 WARN Pod/maas-controller-5b858f66fc-gl9nd: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:42:21 INFO ReplicaSet/maas-controller-5b858f66fc: SuccessfulCreate - Created pod: maas-controller-5b858f66fc-gl9nd 2026/07/05 17:44:37 17:42:21 INFO ReplicaSet/mlflow-operator-controller-manager-57f4cd4d95: SuccessfulCreate - Created pod: mlflow-operator-controller-manager-57f4cd4d95-kfl6d 2026/07/05 17:44:37 17:42:21 INFO ReplicaSet/ogx-k8s-operator-controller-manager-94b957b46: SuccessfulCreate - Created pod: ogx-k8s-operator-controller-manager-94b957b46-8xj29 2026/07/05 17:44:37 17:42:20 WARN Pod/trustyai-service-operator-controller-manager-7bcdbcb4fd-pxc8f: FailedScheduling - 0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. 2026/07/05 17:44:37 17:42:20 INFO ReplicaSet/trustyai-service-operator-controller-manager-7bcdbcb4fd: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-7bcdbcb4fd-pxc8f 2026/07/05 17:44:37 17:41:18 INFO Pod/kserve-controller-manager-588b4b787c-4nm25: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/kserve-controller-manager-588b4b787c-4nm25 2026/07/05 17:44:37 17:41:18 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q 2026/07/05 17:44:37 17:41:18 INFO Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/llmisvc-controller-manager-bc44d7c7b-h9kc7 2026/07/05 17:44:37 17:41:18 INFO Pod/model-serving-api-5c897df776-2pwnq: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/model-serving-api-5c897df776-2pwnq 2026/07/05 17:44:37 17:41:18 INFO Pod/odh-model-controller-566d8764f4-ddc4h: TaintManagerEviction - Cancelling deletion of Pod redhat-ods-applications/odh-model-controller-566d8764f4-ddc4h 2026/07/05 17:44:37 17:41:17 WARN Pod/global-pull-secret-syncer-5l728: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/kube-apiserver-proxy-ip-10-0-143-231.ec2.internal: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/konnectivity-agent-mff9p: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/postgres-69fb4594cf-mz8gk: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/maas-controller-5b858f66fc-xrp5h: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/trustyai-service-operator-controller-manager-7bcdbcb4fd-n65d4: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/model-serving-api-5c897df776-4mkx7: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/ogx-k8s-operator-controller-manager-94b957b46-57fdc: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:17 WARN Pod/mlflow-operator-controller-manager-57f4cd4d95-jlw55: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:12 WARN Pod/model-serving-api-5c897df776-2pwnq: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/kube-apiserver-proxy-ip-10-0-131-75.ec2.internal: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/model-registry-operator-controller-manager-b67847846-s94zx: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/kserve-controller-manager-588b4b787c-4nm25: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/konnectivity-agent-n2m85: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/odh-model-controller-566d8764f4-rw474: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/workload-variant-autoscaler-controller-manager-67777f585d-dck2s: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/maas-api-557679ff4c-rn27g: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/global-pull-secret-syncer-gfzgr: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/llmisvc-controller-manager-bc44d7c7b-h9kc7: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/kube-apiserver-proxy-ip-10-0-137-37.ec2.internal: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/odh-model-controller-566d8764f4-ddc4h: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/konnectivity-agent-tcxd2: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:41:11 WARN Pod/global-pull-secret-syncer-8vvhd: NodeNotReady - Node is not ready 2026/07/05 17:44:37 17:39:53 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Started - Started container manager 2026/07/05 17:44:37 17:39:53 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Pulled - Successfully pulled image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:79fa44b245109a0c1d214c8b313dc8172b367dedf75c4691ed1faefa46f41775" in 1.498s (1.498s including waiting). Image size: 195240092 bytes. 2026/07/05 17:44:37 17:39:53 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Created - Created container: manager 2026/07/05 17:44:37 17:39:51 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q to ip-10-0-131-75.ec2.internal 2026/07/05 17:44:37 17:39:51 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-67777f585d: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-67777f585d-5kz8q 2026/07/05 17:44:37 17:39:51 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: Pulling - Pulling image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:79fa44b245109a0c1d214c8b313dc8172b367dedf75c4691ed1faefa46f41775" 2026/07/05 17:44:37 17:39:51 INFO Pod/workload-variant-autoscaler-controller-manager-67777f585d-5kz8q: AddedInterface - Add eth0 [10.134.0.78/23] from ovn-kubernetes 2026/07/05 17:44:37 17:39:50 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-67777f585d from 1 to 2 2026/07/05 17:44:37 === RESOURCE QUOTAS === 2026/07/05 17:44:37 No resource quotas found in monitored namespaces 2026/07/05 17:44:37 Classification for components: infrastructure/pod-startup (code=1002, confidence=high) 2026/07/05 17:44:37 Evidence: pod maas-api-557679ff4c-47gbq stuck in Pending for 1m34s FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["pod maas-api-557679ff4c-47gbq stuck in Pending for 1m34s"],"confidence":"high"} 2026/07/05 17:44:37 Diagnostics complete for failed test: components controller_test.go:491: Stopping: components test failed. Deletion Policy: Never. Skipping deletion tests. 2026/07/05 17:44:37 === CIRCUIT BREAKER SUMMARY === 2026/07/05 17:44:37 Tripped 1 time(s) 2026/07/05 17:44:37 Reason: Infrastructure failure detected after 3 infrastructure/unknown failures. Cluster health issues: [operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-654b676654-mwcrr: Pending; operator pod rhods-operator-654b676654-q27p4: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-b289v: Pending; dependent jobset: deployment 0/1 ready; pod jobset-operator-56d489b4b8-g9zzk: Pending; dependent leader-worker-set: deployment 0/1 ready; pod openshift-lws-operator-697fbb9648-kmfqb: Pending; dependent cert-manager: deployment 0/1 ready; pod cert-manager-operator-controller-manager-55c444bf86-5l8dr: Pending; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-dk8gv: Pending] 2026/07/05 17:44:37 =============================== controller_test.go:433: Circuit breaker tripped: Infrastructure failure detected after 3 infrastructure/unknown failures. Cluster health issues: [operator deployment rhods-operator: 0/3 ready; operator pod rhods-operator-654b676654-mwcrr: Pending; operator pod rhods-operator-654b676654-q27p4: Pending; dependent kueue: deployment 0/2 ready; pod openshift-kueue-operator-8568d5f8-b289v: Pending; dependent jobset: deployment 0/1 ready; pod jobset-operator-56d489b4b8-g9zzk: Pending; dependent leader-worker-set: deployment 0/1 ready; pod openshift-lws-operator-697fbb9648-kmfqb: Pending; dependent cert-manager: deployment 0/1 ready; pod cert-manager-operator-controller-manager-55c444bf86-5l8dr: Pending; dependent kuadrant: deployment 0/1 ready; pod authorino-operator-6d85f6564-dk8gv: Pending] --- FAIL: TestOdhOperator (1105.01s) === RUN TestRunExcluding/runs_all_except_excluded testgroup_test.go:119: Error Trace: /workspace/tests/e2e/testgroup_test.go:119 Error: []string{} does not contain "auth" Test: TestRunExcluding/runs_all_except_excluded --- FAIL: TestRunExcluding/runs_all_except_excluded (0.00s) === RUN TestRunExcluding --- FAIL: TestRunExcluding (0.00s)