=== RUN TestOdhOperator/components/group_1/feastoperator/Validate_update_operand_resources/deployment_feast-operator-controller-manager testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/feastoperator/Validate_update_operand_resources/deployment_feast-operator-controller-manager (5.33s) 2026/07/16 22:53:38 TEST FAILURE DETECTED: Validate update operand resources 2026/07/16 22:53:38 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:Validate update operand resources) === 2026/07/16 22:53:39 === CLUSTER STATE === 2026/07/16 22:53:39 Node: ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 22:53:39 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 22:53:39 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 22:53:39 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 22:53:39 Condition Ready: True - kubelet is posting ready status 2026/07/16 22:53:39 Resources: 15500m CPU, 63871020Ki memory allocatable (of 16000m CPU, 65021996Ki memory capacity) 2026/07/16 22:53:39 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:53:39 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 22:53:39 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 22:53:39 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 22:53:39 Condition Ready: True - kubelet is posting ready status 2026/07/16 22:53:39 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/16 22:53:39 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:53:39 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 22:53:39 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 22:53:39 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 22:53:39 Condition Ready: True - kubelet is posting ready status 2026/07/16 22:53:39 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/16 22:53:39 === DEPLOYMENTS === 2026/07/16 22:53:39 Failed to collect deployment data: deployments not ready: redhat-ods-applications/ai-gateway-operator (0/1) 2026/07/16 22:53:39 === PODS === 2026/07/16 22:53:39 Failed to collect pod data: redhat-ods-applications/ai-gateway-operator-86947886bc-5tssj: container manager not ready 2026/07/16 22:53:39 === OPERATOR STATUS === 2026/07/16 22:53:39 Operator deployment: 3/3 ready 2026/07/16 22:53:39 === DSCI STATUS === 2026/07/16 22:53:39 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/16 22:53:39 === DSC STATUS === 2026/07/16 22:53:39 Failed to collect DSC data: default-dsc unhealthy conditions: Ready=False; ComponentsReady=False; AIPipelinesReady=False; FeastOperatorReady=False; KserveReady=False; ModelControllerReady=False; ModelRegistryReady=False; ModelsAsServiceReady=False; OGXReady=False; TrainingOperatorReady=False; WorkbenchesReady=False 2026/07/16 22:53:39 === RECENT EVENTS === 2026/07/16 22:53:39 22:53:35 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:53:39 22:53:35 WARN Pod/maas-api-8f8fd6cd4-kwpjv: Unhealthy - Readiness probe failed: Get "https://10.129.1.219:8443/health": dial tcp 10.129.1.219:8443: connect: connection refused 2026/07/16 22:53:39 22:53:35 INFO Pod/kubeflow-training-operator-54786944d6-mhhx4: Killing - Stopping container training-operator 2026/07/16 22:53:39 22:53:35 INFO Pod/odh-model-controller-65fdb9c5c-qsmv4: Killing - Stopping container manager 2026/07/16 22:53:39 22:53:35 INFO Pod/notebook-controller-deployment-647b7d4d45-fs4zz: Killing - Stopping container manager 2026/07/16 22:53:39 22:53:35 INFO Pod/model-serving-api-5679d6b4f-d594b: Killing - Stopping container server 2026/07/16 22:53:39 22:53:35 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-lcft7: Killing - Stopping container manager 2026/07/16 22:53:39 22:53:35 INFO Pod/odh-notebook-controller-manager-7784c757df-hs92w: Killing - Stopping container manager 2026/07/16 22:53:39 22:53:35 INFO Pod/maas-api-8f8fd6cd4-kwpjv: Killing - Stopping container maas-api 2026/07/16 22:53:39 22:53:35 INFO Pod/maas-controller-75bbfd978-p8gbx: Killing - Stopping container manager 2026/07/16 22:53:39 22:53:35 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-4tqnw: Killing - Stopping container manager 2026/07/16 22:53:39 22:53:35 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-j89gx: Killing - Stopping container manager 2026/07/16 22:53:39 22:53:35 INFO Pod/kserve-controller-manager-dd9f6cf9-qnjks: Killing - Stopping container manager 2026/07/16 22:53:39 22:53:34 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:53:39 22:53:34 INFO Pod/feast-operator-controller-manager-7b7cd769f7-m8h4v: Killing - Stopping container manager 2026/07/16 22:53:39 22:53:34 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988w7km9: Killing - Stopping container manager 2026/07/16 22:53:39 22:53:33 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:53:39 22:53:32 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:53:39 22:53:31 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:53:39 22:53:30 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:53:39 22:53:30 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:53:39 22:53:29 INFO Pod/ai-gateway-operator-86947886bc-5tssj: Started - Started container manager 2026/07/16 22:53:39 22:53:29 INFO Pod/ai-gateway-operator-86947886bc-5tssj: Pulled - Successfully pulled image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" in 717ms (717ms including waiting). Image size: 205008753 bytes. 2026/07/16 22:53:39 22:53:29 INFO Pod/ai-gateway-operator-86947886bc-5tssj: Created - Created container: manager 2026/07/16 22:53:39 22:53:28 INFO Pod/ai-gateway-operator-86947886bc-5tssj: Pulled - Successfully pulled image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" in 509ms (509ms including waiting). Image size: 205008753 bytes. 2026/07/16 22:53:39 22:53:28 INFO Pod/ai-gateway-operator-86947886bc-5tssj: Pulling - Pulling image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" 2026/07/16 22:53:39 22:53:28 INFO Pod/ai-gateway-operator-86947886bc-5tssj: Started - Started container copy-manifests 2026/07/16 22:53:39 22:53:28 INFO Pod/ai-gateway-operator-86947886bc-5tssj: Created - Created container: copy-manifests 2026/07/16 22:53:39 22:53:27 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:53:39 22:53:27 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:53:39 22:53:27 INFO Pod/ai-gateway-operator-86947886bc-5tssj: AddedInterface - Add eth0 [10.129.0.9/23] from ovn-kubernetes 2026/07/16 22:53:39 22:53:27 INFO Pod/ai-gateway-operator-86947886bc-5tssj: Pulling - Pulling image "quay.io/rhoai/odh-ai-gateway-operator-rhel9@sha256:0385c7265f7014a6e2b4b98b2e900e6b1b986dc75c37b7ff4cac9387ad8a4a71" 2026/07/16 22:53:39 22:53:26 INFO Pod/ai-gateway-operator-86947886bc-5tssj: Scheduled - Successfully assigned redhat-ods-applications/ai-gateway-operator-86947886bc-5tssj to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:53:39 22:53:26 INFO Deployment/ai-gateway-operator: ScalingReplicaSet - Scaled up replica set ai-gateway-operator-86947886bc from 0 to 1 2026/07/16 22:53:39 22:53:26 INFO ReplicaSet/ai-gateway-operator-86947886bc: SuccessfulCreate - Created pod: ai-gateway-operator-86947886bc-5tssj 2026/07/16 22:53:39 22:49:48 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:53:39 22:49:48 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:53:39 22:49:47 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:53:39 22:49:47 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:53:39 === RESOURCE QUOTAS === 2026/07/16 22:53:39 No resource quotas found in monitored namespaces 2026/07/16 22:53:39 Classification for Validate update operand resources: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/16 22:53:39 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/16 22:53:39 Diagnostics complete for failed test: Validate update operand resources === RUN TestOdhOperator/components/group_1/feastoperator/Validate_update_operand_resources testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/feastoperator/Validate_update_operand_resources (6.13s) === RUN TestOdhOperator/components/group_1/feastoperator === PAUSE TestOdhOperator/components/group_1/feastoperator === CONT TestOdhOperator/components/group_1/feastoperator resource_applier_test.go:245: Failed after 5.102s. The function passed to Consistently failed at /workspace/tests/e2e/resource_applier_test.go:278 with: Error occurred while applying the resource 'redhat-ods-applications/feast-operator-controller-manager' of kind 'Deployment': resourceVersion should not be set on objects to be created Unexpected error: <*errors.StatusError | 0x12c2c059c0a0>: resourceVersion should not be set on objects to be created { ErrStatus: { TypeMeta: {Kind: "Status", APIVersion: "v1"}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: "resourceVersion should not be set on objects to be created", Reason: "", Details: nil, Code: 500, }, } occurred --- FAIL: TestOdhOperator/components/group_1/feastoperator (18.47s) === RUN TestOdhOperator/components/group_1 --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:150: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:150: Disabling MaaS subcomponent (setting to Removed) helper_test.go:150: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.19s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-16T23:06:24Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.19"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com, certSecret=2r2qdb90ltvhjc43lp6iqc6aqb7bgjda-primary-cert-bundle-secret controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.009s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "name": <string>"default", "uid": <string>"9da604c0-a799-475d-aec2-5af3e651f338", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", }, ], "resourceVersion": <string>"35941596", "uid": <string>"041a42e2-4673-4f4e-8dc3-3f92bfde5b7c", "creationTimestamp": <string>"2026-07-16T23:07:23Z", "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"9da604c0-a799-475d-aec2-5af3e651f338\"}": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-16T23:07:23Z", }, <map[string]interface {} | len:7>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"manager", "operation": <string>"Update", "subresource": <string>"status", "time": <string>"2026-07-16T23:07:48Z", }, ], "name": <string>"default-tenant", }, "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "name": <string>"maas-default-gateway", "namespace": <string>"openshift-ingress", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", "status": <string>"True", "type": <string>"DependenciesAvailable", "lastTransitionTime": <string>"2026-07-16T23:07:26Z", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-16T23:07:26Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"True", "type": <string>"MaaSPrerequisitesAvailable", }, <map[string]interface {} | len:6>{ "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"False", "type": <string>"Degraded", "lastTransitionTime": <string>"2026-07-16T23:07:26Z", }, <map[string]interface {} | len:6>{ "status": <string>"True", "type": <string>"DeploymentsAvailable", "lastTransitionTime": <string>"2026-07-16T23:07:48Z", "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"DeploymentsReady", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-16T23:07:48Z", "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", "type": <string>"Ready", }, ], "phase": <string>"Active", }, }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (522.07s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/16 22:58:31 TEST FAILURE DETECTED: group 1 2026/07/16 22:58:31 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/16 22:58:31 === CLUSTER STATE === 2026/07/16 22:58:31 Node: ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 22:58:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 22:58:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 22:58:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 22:58:31 Condition Ready: True - kubelet is posting ready status 2026/07/16 22:58:31 Resources: 15500m CPU, 63871020Ki memory allocatable (of 16000m CPU, 65021996Ki memory capacity) 2026/07/16 22:58:31 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 22:58:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 22:58:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 22:58:31 Condition Ready: True - kubelet is posting ready status 2026/07/16 22:58:31 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/16 22:58:31 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 22:58:31 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 22:58:31 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 22:58:31 Condition Ready: True - kubelet is posting ready status 2026/07/16 22:58:31 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/16 22:58:31 === DEPLOYMENTS === 2026/07/16 22:58:31 Namespace: redhat-ods-applications 2026/07/16 22:58:31 Deployment feast-operator-controller-manager: 1/1 (READY) 2026/07/16 22:58:31 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/16 22:58:31 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/16 22:58:31 Namespace: kube-system 2026/07/16 22:58:31 === PODS === 2026/07/16 22:58:31 Namespace: redhat-ods-applications 2026/07/16 22:58:31 Pod maas-api-key-cleanup-29736435-89msd: Succeeded 2026/07/16 22:58:31 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/07/16 22:58:31 Namespace kube-system: no problematic pods found 2026/07/16 22:58:31 === LOGS (CURRENT) for container cleanup === 2026/07/16 22:58:31 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/07/16 22:58:31 === END LOGS === 2026/07/16 22:58:31 === OPERATOR STATUS === 2026/07/16 22:58:31 Operator deployment: 3/3 ready 2026/07/16 22:58:31 === DSCI STATUS === 2026/07/16 22:58:31 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/16 22:58:31 === DSC STATUS === 2026/07/16 22:58:31 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/16 22:58:31 === RECENT EVENTS === 2026/07/16 22:58:31 22:58:24 INFO Pod/dashboard-redirect-84fcf4d86-k8q6f: Killing - Stopping container nginx 2026/07/16 22:58:31 22:58:24 INFO Pod/dashboard-redirect-84fcf4d86-l9z29: Killing - Stopping container nginx 2026/07/16 22:58:31 22:57:55 INFO Pod/spark-operator-controller-5dd7d48f87-p6j78: Killing - Stopping container controller 2026/07/16 22:58:31 22:57:55 INFO Pod/spark-operator-webhook-5f8768dbfd-gsd6k: Killing - Stopping container webhook 2026/07/16 22:58:31 22:57:54 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:57:42 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-5dd7d48f87-p6j78_a7a49462-83cf-42e2-8778-dcef4dae17da became leader 2026/07/16 22:58:31 22:57:41 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 492ms (492ms including waiting). Image size: 185092464 bytes. 2026/07/16 22:58:31 22:57:41 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Created - Created container: mlflow-ui 2026/07/16 22:58:31 22:57:41 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 608ms (608ms including waiting). Image size: 185092464 bytes. 2026/07/16 22:58:31 22:57:41 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Started - Started container mlflow-ui 2026/07/16 22:58:31 22:57:41 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/16 22:58:31 22:57:40 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Started - Started container maas-ui 2026/07/16 22:58:31 22:57:40 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 468ms (468ms including waiting). Image size: 177808591 bytes. 2026/07/16 22:58:31 22:57:40 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Created - Created container: mlflow-ui 2026/07/16 22:58:31 22:57:40 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Started - Started container mlflow-ui 2026/07/16 22:58:31 22:57:40 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/16 22:58:31 22:57:40 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 414ms (414ms including waiting). Image size: 187802205 bytes. 2026/07/16 22:58:31 22:57:40 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 448ms (448ms including waiting). Image size: 177808591 bytes. 2026/07/16 22:58:31 22:57:40 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/16 22:58:31 22:57:40 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Created - Created container: maas-ui 2026/07/16 22:58:31 22:57:39 INFO Pod/spark-operator-webhook-5f8768dbfd-gsd6k: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-gsd6k to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 585ms (585ms including waiting). Image size: 187802205 bytes. 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 445ms (445ms including waiting). Image size: 241242949 bytes. 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Started - Started container model-registry-ui 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Created - Created container: model-registry-ui 2026/07/16 22:58:31 22:57:39 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-gsd6k 2026/07/16 22:58:31 22:57:39 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 0 to 1 2026/07/16 22:58:31 22:57:39 INFO Pod/spark-operator-webhook-5f8768dbfd-6hkbb: Killing - Stopping container webhook 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/16 22:58:31 22:57:39 INFO Pod/spark-operator-webhook-5f8768dbfd-gsd6k: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/16 22:58:31 22:57:39 INFO Pod/spark-operator-webhook-5f8768dbfd-gsd6k: Created - Created container: webhook 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Created - Created container: gen-ai-ui 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Started - Started container maas-ui 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Created - Created container: maas-ui 2026/07/16 22:58:31 22:57:39 INFO Pod/spark-operator-webhook-5f8768dbfd-flfcm: Killing - Stopping container webhook 2026/07/16 22:58:31 22:57:39 INFO Pod/spark-operator-webhook-5f8768dbfd-gsd6k: AddedInterface - Add eth0 [10.129.0.42/23] from ovn-kubernetes 2026/07/16 22:58:31 22:57:39 INFO Pod/spark-operator-webhook-5f8768dbfd-gsd6k: Started - Started container webhook 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Started - Started container gen-ai-ui 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Started - Started container gen-ai-ui 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Created - Created container: gen-ai-ui 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 592ms (592ms including waiting). Image size: 241242949 bytes. 2026/07/16 22:58:31 22:57:39 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Created - Created container: model-registry-ui 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Started - Started container kube-rbac-proxy 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Started - Started container rhods-dashboard 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Created - Created container: kube-rbac-proxy 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Started - Started container model-registry-ui 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Created - Created container: rhods-dashboard 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 634ms (634ms including waiting). Image size: 799822033 bytes. 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 520ms (520ms including waiting). Image size: 182438697 bytes. 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Created - Created container: rhods-dashboard 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Started - Started container rhods-dashboard 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Created - Created container: kube-rbac-proxy 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 433ms (433ms including waiting). Image size: 182438697 bytes. 2026/07/16 22:58:31 22:57:38 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Started - Started container kube-rbac-proxy 2026/07/16 22:58:31 22:57:37 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: AddedInterface - Add eth0 [10.130.0.79/23] from ovn-kubernetes 2026/07/16 22:58:31 22:57:37 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/16 22:58:31 22:57:37 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 568ms (568ms including waiting). Image size: 799822033 bytes. 2026/07/16 22:58:31 22:57:37 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/16 22:58:31 22:57:37 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: AddedInterface - Add eth0 [10.128.1.238/23] from ovn-kubernetes 2026/07/16 22:58:31 22:57:36 INFO Pod/rhods-dashboard-84cdfcd5d8-cczmd: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-84cdfcd5d8-cczmd to ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 22:58:31 22:57:36 INFO Pod/rhods-dashboard-84cdfcd5d8-66t7v: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-84cdfcd5d8-66t7v to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:57:36 INFO ReplicaSet/rhods-dashboard-84cdfcd5d8: SuccessfulCreate - Created pod: rhods-dashboard-84cdfcd5d8-66t7v 2026/07/16 22:58:31 22:57:36 INFO ReplicaSet/rhods-dashboard-84cdfcd5d8: SuccessfulCreate - Created pod: rhods-dashboard-84cdfcd5d8-cczmd 2026/07/16 22:58:31 22:57:36 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-84cdfcd5d8 from 0 to 2 2026/07/16 22:58:31 22:57:25 INFO Pod/spark-operator-controller-5dd7d48f87-p6j78: Started - Started container controller 2026/07/16 22:58:31 22:57:25 INFO Pod/spark-operator-controller-5dd7d48f87-p6j78: Created - Created container: controller 2026/07/16 22:58:31 22:57:24 INFO Pod/spark-operator-controller-5dd7d48f87-p6j78: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-p6j78 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:57:24 INFO Pod/spark-operator-controller-5dd7d48f87-p6j78: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/16 22:58:31 22:57:24 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 0 to 1 2026/07/16 22:58:31 22:57:24 INFO Pod/spark-operator-controller-5dd7d48f87-p6j78: AddedInterface - Add eth0 [10.130.0.78/23] from ovn-kubernetes 2026/07/16 22:58:31 22:57:24 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-p6j78 2026/07/16 22:58:31 22:57:24 INFO Pod/spark-operator-controller-5dd7d48f87-4hx5g: Killing - Stopping container controller 2026/07/16 22:58:31 22:57:24 INFO Pod/spark-operator-controller-5dd7d48f87-2t5xr: Killing - Stopping container controller 2026/07/16 22:58:31 22:57:13 WARN Endpoints/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242-94598275-ui-svc: FailedToUpdateEndpoint - Failed to update endpoint redhat-ods-applications/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242-94598275-ui-svc: Operation cannot be fulfilled on endpoints "scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242-94598275-ui-svc": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242-94598275-ui-svc, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 5b748e2b-c87c-4c2d-aabe-22c2a74e4eb2, UID in object meta: 2026/07/16 22:58:31 22:57:12 INFO SparkApplication/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297: SparkDriverCompleted - Driver scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297-driver completed 2026/07/16 22:58:31 22:57:10 INFO SparkApplication/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297: SparkExecutorCompleted - Executor spark-pi-e7a9a79f6d25ad10-exec-1 completed 2026/07/16 22:58:31 22:57:10 INFO Pod/spark-pi-e7a9a79f6d25ad10-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/16 22:58:31 22:56:58 INFO SparkApplication/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297: SparkExecutorRunning - Executor spark-pi-e7a9a79f6d25ad10-exec-1 is running 2026/07/16 22:58:31 22:56:57 INFO Pod/spark-pi-e7a9a79f6d25ad10-exec-1: Started - Started container spark-kubernetes-executor 2026/07/16 22:58:31 22:56:57 INFO Pod/spark-pi-e7a9a79f6d25ad10-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/16 22:58:31 22:56:57 INFO Pod/spark-pi-e7a9a79f6d25ad10-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/16 22:58:31 22:56:57 INFO Pod/spark-pi-e7a9a79f6d25ad10-exec-1: AddedInterface - Add eth0 [10.130.0.77/23] from ovn-kubernetes 2026/07/16 22:58:31 22:56:56 INFO Pod/spark-pi-e7a9a79f6d25ad10-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-e7a9a79f6d25ad10-exec-1 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:56:56 INFO SparkApplication/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297: SparkExecutorPending - Executor spark-pi-e7a9a79f6d25ad10-exec-1 is pending 2026/07/16 22:58:31 22:56:53 INFO Pod/maas-api-5b7784b768-vmqdr: Killing - Stopping container maas-api 2026/07/16 22:58:31 22:56:52 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:56:52 INFO Pod/odh-model-controller-bd4c5cbcb-58scg: Killing - Stopping container manager 2026/07/16 22:58:31 22:56:52 INFO Pod/maas-controller-75bbfd978-tlmxf: Killing - Stopping container manager 2026/07/16 22:58:31 22:56:52 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qgkhk: Killing - Stopping container manager 2026/07/16 22:58:31 22:56:52 INFO Pod/kserve-controller-manager-dd9f6cf9-tcn7m: Killing - Stopping container manager 2026/07/16 22:58:31 22:56:52 INFO Pod/model-serving-api-5679d6b4f-vf7m9: Killing - Stopping container server 2026/07/16 22:58:31 22:56:51 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:56:51 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:56:48 INFO SparkApplication/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297: SparkDriverRunning - Driver scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297-driver is running 2026/07/16 22:58:31 22:56:47 INFO Pod/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297-driver: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/16 22:58:31 22:56:47 INFO Pod/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297-driver: Created - Created container: spark-kubernetes-driver 2026/07/16 22:58:31 22:56:47 INFO Pod/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297-driver: Started - Started container spark-kubernetes-driver 2026/07/16 22:58:31 22:56:47 INFO SparkApplication/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297: SparkApplicationSubmitted - SparkApplication scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297 was submitted successfully 2026/07/16 22:58:31 22:56:47 INFO Pod/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297-driver: AddedInterface - Add eth0 [10.130.0.75/23] from ovn-kubernetes 2026/07/16 22:58:31 22:56:46 INFO Pod/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297-driver: Scheduled - Successfully assigned redhat-ods-applications/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297-driver to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:56:46 WARN Pod/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-861d419f6d258bb1-conf-map" not found 2026/07/16 22:58:31 22:56:46 WARN Pod/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297-6fdf759f6d25794d-driver-podspec-conf-map" not found 2026/07/16 22:58:31 22:56:36 INFO Pod/kuberay-operator-6586cdc875-llzct: Killing - Stopping container kuberay-operator 2026/07/16 22:58:31 22:56:34 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-qgkhk_2e3eec4d-9077-435a-a4b9-ed09d96a5b36 became leader 2026/07/16 22:58:31 22:56:20 INFO Pod/kuberay-operator-6586cdc875-llzct: AddedInterface - Add eth0 [10.129.0.41/23] from ovn-kubernetes 2026/07/16 22:58:31 22:56:20 INFO Pod/kuberay-operator-6586cdc875-llzct: Created - Created container: kuberay-operator 2026/07/16 22:58:31 22:56:20 INFO Pod/kuberay-operator-6586cdc875-llzct: Pulled - Container image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:0e1c5ae8ddd751b0c316f1f50d1f3c8ad4f9b595df1b8354d8361176e3362fa4" already present on machine 2026/07/16 22:58:31 22:56:20 INFO Pod/kuberay-operator-6586cdc875-llzct: Started - Started container kuberay-operator 2026/07/16 22:58:31 22:56:19 INFO Pod/kuberay-operator-6586cdc875-llzct: Scheduled - Successfully assigned redhat-ods-applications/kuberay-operator-6586cdc875-llzct to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:56:19 INFO ReplicaSet/kuberay-operator-6586cdc875: SuccessfulCreate - Created pod: kuberay-operator-6586cdc875-llzct 2026/07/16 22:58:31 22:56:19 INFO Deployment/kuberay-operator: ScalingReplicaSet - Scaled up replica set kuberay-operator-6586cdc875 from 0 to 1 2026/07/16 22:58:31 22:56:19 INFO Pod/kuberay-operator-6586cdc875-sclvq: Killing - Stopping container kuberay-operator 2026/07/16 22:58:31 22:56:19 INFO Pod/kuberay-operator-6586cdc875-fvrs7: Killing - Stopping container kuberay-operator 2026/07/16 22:58:31 22:56:19 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled down replica set rhods-dashboard-84cdfcd5d8 from 2 to 1 2026/07/16 22:58:31 22:56:19 INFO ReplicaSet/rhods-dashboard-84cdfcd5d8: SuccessfulDelete - Deleted pod: rhods-dashboard-84cdfcd5d8-qf4mw 2026/07/16 22:58:31 22:56:19 WARN Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Unhealthy - Readiness probe failed: Get "https://10.130.0.72:8444/healthz": dial tcp 10.130.0.72:8444: connect: connection refused 2026/07/16 22:58:31 22:56:16 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qgkhk: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 526ms (526ms including waiting). Image size: 236366599 bytes. 2026/07/16 22:58:31 22:56:16 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qgkhk: Started - Started container manager 2026/07/16 22:58:31 22:56:16 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qgkhk: Created - Created container: manager 2026/07/16 22:58:31 22:56:15 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qgkhk: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-qgkhk to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:56:15 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-qgkhk 2026/07/16 22:58:31 22:56:15 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qgkhk: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 22:58:31 22:56:15 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qgkhk: AddedInterface - Add eth0 [10.129.0.40/23] from ovn-kubernetes 2026/07/16 22:58:31 22:56:15 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-njc6r: Killing - Stopping container manager 2026/07/16 22:58:31 22:56:15 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmpmw: Killing - Stopping container manager 2026/07/16 22:58:31 22:56:15 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/16 22:58:31 22:56:03 INFO SparkApplication/spark-pi-d9cm2mrfh92s7b6ih5vg: SparkDriverCompleted - Driver spark-pi-d9cm2mrfh92s7b6ih5vg-driver completed 2026/07/16 22:58:31 22:56:00 INFO Pod/spark-pi-e9157f9f6d249e19-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/16 22:58:31 22:56:00 INFO SparkApplication/spark-pi-d9cm2mrfh92s7b6ih5vg: SparkExecutorCompleted - Executor spark-pi-e9157f9f6d249e19-exec-1 completed 2026/07/16 22:58:31 22:55:59 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-tcn7m_a4692551-67d9-4fc8-ad75-e188969bd5ae became leader 2026/07/16 22:58:31 22:55:48 INFO SparkApplication/spark-pi-d9cm2mrfh92s7b6ih5vg: SparkExecutorRunning - Executor spark-pi-e9157f9f6d249e19-exec-1 is running 2026/07/16 22:58:31 22:55:47 INFO Pod/spark-pi-e9157f9f6d249e19-exec-1: AddedInterface - Add eth0 [10.130.0.74/23] from ovn-kubernetes 2026/07/16 22:58:31 22:55:47 INFO Pod/spark-pi-e9157f9f6d249e19-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/16 22:58:31 22:55:47 INFO Pod/spark-pi-e9157f9f6d249e19-exec-1: Started - Started container spark-kubernetes-executor 2026/07/16 22:58:31 22:55:47 INFO Pod/spark-pi-e9157f9f6d249e19-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/16 22:58:31 22:55:47 INFO Lease/ray-operator-leader: LeaderElection - kuberay-operator-6586cdc875-sclvq_9ecf129a-66ad-442e-b484-d1a75e64d214 became leader 2026/07/16 22:58:31 22:55:46 INFO Pod/spark-pi-e9157f9f6d249e19-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-e9157f9f6d249e19-exec-1 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:55:46 INFO SparkApplication/spark-pi-d9cm2mrfh92s7b6ih5vg: SparkExecutorPending - Executor spark-pi-e9157f9f6d249e19-exec-1 is pending 2026/07/16 22:58:31 22:55:44 INFO Pod/kuberay-operator-6586cdc875-fvrs7: Scheduled - Successfully assigned redhat-ods-applications/kuberay-operator-6586cdc875-fvrs7 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:55:44 INFO Pod/kuberay-operator-6586cdc875-fvrs7: Started - Started container kuberay-operator 2026/07/16 22:58:31 22:55:44 INFO Pod/kuberay-operator-6586cdc875-fvrs7: AddedInterface - Add eth0 [10.129.0.39/23] from ovn-kubernetes 2026/07/16 22:58:31 22:55:44 INFO Deployment/kuberay-operator: ScalingReplicaSet - Scaled up replica set kuberay-operator-6586cdc875 from 1 to 2 2026/07/16 22:58:31 22:55:44 INFO ReplicaSet/kuberay-operator-6586cdc875: SuccessfulCreate - Created pod: kuberay-operator-6586cdc875-fvrs7 2026/07/16 22:58:31 22:55:44 INFO Pod/kuberay-operator-6586cdc875-fvrs7: Created - Created container: kuberay-operator 2026/07/16 22:58:31 22:55:44 INFO Pod/kuberay-operator-6586cdc875-fvrs7: Pulled - Container image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:0e1c5ae8ddd751b0c316f1f50d1f3c8ad4f9b595df1b8354d8361176e3362fa4" already present on machine 2026/07/16 22:58:31 22:55:43 INFO Pod/kserve-controller-manager-dd9f6cf9-tcn7m: Created - Created container: manager 2026/07/16 22:58:31 22:55:43 INFO Pod/kserve-controller-manager-dd9f6cf9-tcn7m: Started - Started container manager 2026/07/16 22:58:31 22:55:42 INFO Pod/kserve-controller-manager-dd9f6cf9-tcn7m: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 22:58:31 22:55:42 INFO Pod/kserve-controller-manager-dd9f6cf9-tcn7m: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 522ms (522ms including waiting). Image size: 238435613 bytes. 2026/07/16 22:58:31 22:55:42 INFO Pod/notebook-controller-deployment-647b7d4d45-xwv5d: Killing - Stopping container manager 2026/07/16 22:58:31 22:55:42 INFO Pod/kserve-controller-manager-dd9f6cf9-tcn7m: AddedInterface - Add eth0 [10.129.0.38/23] from ovn-kubernetes 2026/07/16 22:58:31 22:55:42 INFO Pod/odh-notebook-controller-manager-7784c757df-pjxzf: Killing - Stopping container manager 2026/07/16 22:58:31 22:55:41 INFO Pod/kserve-controller-manager-dd9f6cf9-tcn7m: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-tcn7m to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:55:41 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-tcn7m 2026/07/16 22:58:31 22:55:41 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/16 22:58:31 22:55:39 INFO Pod/kserve-controller-manager-dd9f6cf9-4vfp6: Killing - Stopping container manager 2026/07/16 22:58:31 22:55:39 INFO SparkApplication/spark-pi-d9cm2mrfh92s7b6ih5vg: SparkDriverRunning - Driver spark-pi-d9cm2mrfh92s7b6ih5vg-driver is running 2026/07/16 22:58:31 22:55:39 INFO Pod/kserve-controller-manager-dd9f6cf9-hhvgx: Killing - Stopping container manager 2026/07/16 22:58:31 22:55:38 INFO Pod/odh-notebook-controller-manager-7784c757df-pjxzf: Pulled - Successfully pulled image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:d70fede8c54fb2edc8691360ee77339cb81bb3de0a39f7adb863c8754eaff13d" in 939ms (939ms including waiting). Image size: 173529551 bytes. 2026/07/16 22:58:31 22:55:38 INFO Pod/spark-pi-d9cm2mrfh92s7b6ih5vg-driver: Created - Created container: spark-kubernetes-driver 2026/07/16 22:58:31 22:55:38 INFO Pod/spark-pi-d9cm2mrfh92s7b6ih5vg-driver: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/16 22:58:31 22:55:38 INFO Pod/odh-notebook-controller-manager-7784c757df-pjxzf: Started - Started container manager 2026/07/16 22:58:31 22:55:38 INFO Pod/odh-notebook-controller-manager-7784c757df-pjxzf: Created - Created container: manager 2026/07/16 22:58:31 22:55:38 INFO Pod/spark-pi-d9cm2mrfh92s7b6ih5vg-driver: AddedInterface - Add eth0 [10.130.0.73/23] from ovn-kubernetes 2026/07/16 22:58:31 22:55:38 INFO Pod/spark-pi-d9cm2mrfh92s7b6ih5vg-driver: Started - Started container spark-kubernetes-driver 2026/07/16 22:58:31 22:55:37 INFO Pod/odh-notebook-controller-manager-7784c757df-pjxzf: Scheduled - Successfully assigned redhat-ods-applications/odh-notebook-controller-manager-7784c757df-pjxzf to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:55:37 INFO Pod/spark-pi-d9cm2mrfh92s7b6ih5vg-driver: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-d9cm2mrfh92s7b6ih5vg-driver to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:55:37 INFO SparkApplication/spark-pi-d9cm2mrfh92s7b6ih5vg: SparkApplicationSubmitted - SparkApplication spark-pi-d9cm2mrfh92s7b6ih5vg was submitted successfully 2026/07/16 22:58:31 22:55:37 INFO ReplicaSet/odh-notebook-controller-manager-7784c757df: SuccessfulCreate - Created pod: odh-notebook-controller-manager-7784c757df-pjxzf 2026/07/16 22:58:31 22:55:37 INFO Pod/odh-notebook-controller-manager-7784c757df-pjxzf: Pulling - Pulling image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:d70fede8c54fb2edc8691360ee77339cb81bb3de0a39f7adb863c8754eaff13d" 2026/07/16 22:58:31 22:55:37 INFO Pod/odh-notebook-controller-manager-7784c757df-pjxzf: AddedInterface - Add eth0 [10.129.0.37/23] from ovn-kubernetes 2026/07/16 22:58:31 22:55:37 INFO Deployment/odh-notebook-controller-manager: ScalingReplicaSet - Scaled up replica set odh-notebook-controller-manager-7784c757df from 0 to 1 2026/07/16 22:58:31 22:55:37 WARN Pod/spark-pi-d9cm2mrfh92s7b6ih5vg-driver: FailedMount - MountVolume.SetUp failed for volume "pod-template-volume" : configmap "spark-pi-d9cm2mrfh92s7b6ih5vg-bfd2579f6d246919-driver-podspec-conf-map" not found 2026/07/16 22:58:31 22:55:37 WARN Pod/spark-pi-d9cm2mrfh92s7b6ih5vg-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-c6e6fc9f6d247bd6-conf-map" not found 2026/07/16 22:58:31 22:55:36 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 598ms (598ms including waiting). Image size: 185092464 bytes. 2026/07/16 22:58:31 22:55:36 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 766ms (766ms including waiting). Image size: 185092464 bytes. 2026/07/16 22:58:31 22:55:35 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Created - Created container: mlflow-ui 2026/07/16 22:58:31 22:55:35 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 624ms (624ms including waiting). Image size: 177808591 bytes. 2026/07/16 22:58:31 22:55:35 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Created - Created container: mlflow-ui 2026/07/16 22:58:31 22:55:35 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Started - Started container mlflow-ui 2026/07/16 22:58:31 22:55:35 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/16 22:58:31 22:55:35 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/16 22:58:31 22:55:35 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Started - Started container mlflow-ui 2026/07/16 22:58:31 22:55:35 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 514ms (514ms including waiting). Image size: 177808591 bytes. 2026/07/16 22:58:31 22:55:35 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Created - Created container: maas-ui 2026/07/16 22:58:31 22:55:35 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/16 22:58:31 22:55:35 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Started - Started container maas-ui 2026/07/16 22:58:31 22:55:34 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/16 22:58:31 22:55:34 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Created - Created container: maas-ui 2026/07/16 22:58:31 22:55:34 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 586ms (586ms including waiting). Image size: 241242949 bytes. 2026/07/16 22:58:31 22:55:34 INFO Pod/odh-notebook-controller-manager-7784c757df-5h26v: Killing - Stopping container manager 2026/07/16 22:58:31 22:55:34 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/16 22:58:31 22:55:34 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Started - Started container maas-ui 2026/07/16 22:58:31 22:55:34 INFO Pod/odh-notebook-controller-manager-7784c757df-gdvx7: Killing - Stopping container manager 2026/07/16 22:58:31 22:55:34 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 589ms (589ms including waiting). Image size: 187802205 bytes. 2026/07/16 22:58:31 22:55:34 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Started - Started container gen-ai-ui 2026/07/16 22:58:31 22:55:34 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Created - Created container: gen-ai-ui 2026/07/16 22:58:31 22:55:34 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 512ms (512ms including waiting). Image size: 241242949 bytes. 2026/07/16 22:58:31 22:55:34 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Created - Created container: gen-ai-ui 2026/07/16 22:58:31 22:55:34 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Started - Started container gen-ai-ui 2026/07/16 22:58:31 22:55:34 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/16 22:58:31 22:55:34 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 487ms (487ms including waiting). Image size: 187802205 bytes. 2026/07/16 22:58:31 22:55:33 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/16 22:58:31 22:55:33 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/16 22:58:31 22:55:33 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Started - Started container model-registry-ui 2026/07/16 22:58:31 22:55:33 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Created - Created container: model-registry-ui 2026/07/16 22:58:31 22:55:33 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 646ms (646ms including waiting). Image size: 182438697 bytes. 2026/07/16 22:58:31 22:55:33 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/16 22:58:31 22:55:33 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 512ms (512ms including waiting). Image size: 182438697 bytes. 2026/07/16 22:58:31 22:55:33 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Created - Created container: model-registry-ui 2026/07/16 22:58:31 22:55:33 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Started - Started container model-registry-ui 2026/07/16 22:58:31 22:55:32 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Created - Created container: rhods-dashboard 2026/07/16 22:58:31 22:55:32 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/16 22:58:31 22:55:32 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Created - Created container: kube-rbac-proxy 2026/07/16 22:58:31 22:55:32 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Started - Started container kube-rbac-proxy 2026/07/16 22:58:31 22:55:32 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Started - Started container rhods-dashboard 2026/07/16 22:58:31 22:55:32 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/16 22:58:31 22:55:32 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Created - Created container: kube-rbac-proxy 2026/07/16 22:58:31 22:55:32 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 742ms (742ms including waiting). Image size: 799822033 bytes. 2026/07/16 22:58:31 22:55:32 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Created - Created container: rhods-dashboard 2026/07/16 22:58:31 22:55:32 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Started - Started container kube-rbac-proxy 2026/07/16 22:58:31 22:55:32 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/16 22:58:31 22:55:32 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 694ms (694ms including waiting). Image size: 799822033 bytes. 2026/07/16 22:58:31 22:55:32 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Started - Started container rhods-dashboard 2026/07/16 22:58:31 22:55:31 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-84cdfcd5d8-qf4mw to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:55:31 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/16 22:58:31 22:55:31 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: AddedInterface - Add eth0 [10.130.0.72/23] from ovn-kubernetes 2026/07/16 22:58:31 22:55:31 INFO Pod/rhods-dashboard-84cdfcd5d8-qf4mw: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/16 22:58:31 22:55:31 INFO ReplicaSet/rhods-dashboard-84cdfcd5d8: SuccessfulCreate - Created pod: rhods-dashboard-84cdfcd5d8-qf4mw 2026/07/16 22:58:31 22:55:31 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: AddedInterface - Add eth0 [10.128.1.237/23] from ovn-kubernetes 2026/07/16 22:58:31 22:55:30 INFO Pod/rhods-dashboard-84cdfcd5d8-45nfs: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-84cdfcd5d8-45nfs to ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 22:58:31 22:55:30 INFO ReplicaSet/rhods-dashboard-84cdfcd5d8: SuccessfulCreate - Created pod: rhods-dashboard-84cdfcd5d8-45nfs 2026/07/16 22:58:31 22:55:30 WARN Pod/rhods-dashboard-84cdfcd5d8-45nfs: FailedMount - MountVolume.SetUp failed for volume "modules-sa-token" : secret "odh-dashboard-modules-token" not found 2026/07/16 22:58:31 22:55:29 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-84cdfcd5d8 from 0 to 2 2026/07/16 22:58:31 22:55:29 WARN ReplicaSet/rhods-dashboard-84cdfcd5d8: FailedCreate - Error creating: pods "rhods-dashboard-84cdfcd5d8-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 2026/07/16 22:58:31 22:55:28 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bd4c5cbcb-58scg_3117ee93-0f06-4c2b-b22a-a1a629f09bd9 became leader 2026/07/16 22:58:31 22:55:28 INFO Pod/dashboard-redirect-84fcf4d86-l9z29: Started - Started container nginx 2026/07/16 22:58:31 22:55:27 INFO Pod/dashboard-redirect-84fcf4d86-l9z29: Scheduled - Successfully assigned redhat-ods-applications/dashboard-redirect-84fcf4d86-l9z29 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:55:27 INFO Pod/dashboard-redirect-84fcf4d86-k8q6f: Scheduled - Successfully assigned redhat-ods-applications/dashboard-redirect-84fcf4d86-k8q6f to ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 22:58:31 22:55:27 INFO ReplicaSet/dashboard-redirect-84fcf4d86: SuccessfulCreate - Created pod: dashboard-redirect-84fcf4d86-k8q6f 2026/07/16 22:58:31 22:55:27 INFO ReplicaSet/dashboard-redirect-84fcf4d86: SuccessfulCreate - Created pod: dashboard-redirect-84fcf4d86-l9z29 2026/07/16 22:58:31 22:55:27 INFO Pod/dashboard-redirect-84fcf4d86-l9z29: AddedInterface - Add eth0 [10.130.0.70/23] from ovn-kubernetes 2026/07/16 22:58:31 22:55:27 INFO Pod/dashboard-redirect-84fcf4d86-l9z29: Pulled - Container image "registry.redhat.io/ubi9/nginx-126@sha256:f95e7f86407d2bdef5960eae9795f90df0bf33249eafa1de227368d64244d0a8" already present on machine 2026/07/16 22:58:31 22:55:27 INFO Pod/dashboard-redirect-84fcf4d86-k8q6f: AddedInterface - Add eth0 [10.128.1.236/23] from ovn-kubernetes 2026/07/16 22:58:31 22:55:27 INFO Pod/dashboard-redirect-84fcf4d86-k8q6f: Pulled - Container image "registry.redhat.io/ubi9/nginx-126@sha256:f95e7f86407d2bdef5960eae9795f90df0bf33249eafa1de227368d64244d0a8" already present on machine 2026/07/16 22:58:31 22:55:27 INFO Pod/dashboard-redirect-84fcf4d86-k8q6f: Started - Started container nginx 2026/07/16 22:58:31 22:55:27 INFO Pod/notebook-controller-deployment-647b7d4d45-xwv5d: Pulled - Successfully pulled image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:d181e8d8cf8607fecde5b06876f74b9e445e9f2288593ab0b75c57648388fd80" in 551ms (551ms including waiting). Image size: 168353804 bytes. 2026/07/16 22:58:31 22:55:27 INFO Pod/notebook-controller-deployment-647b7d4d45-xwv5d: Created - Created container: manager 2026/07/16 22:58:31 22:55:27 INFO Pod/notebook-controller-deployment-647b7d4d45-xwv5d: Started - Started container manager 2026/07/16 22:58:31 22:55:27 INFO Pod/dashboard-redirect-84fcf4d86-l9z29: Created - Created container: nginx 2026/07/16 22:58:31 22:55:27 INFO Deployment/dashboard-redirect: ScalingReplicaSet - Scaled up replica set dashboard-redirect-84fcf4d86 from 0 to 2 2026/07/16 22:58:31 22:55:27 INFO Pod/kuberay-operator-6586cdc875-sclvq: Started - Started container kuberay-operator 2026/07/16 22:58:31 22:55:27 INFO Pod/kuberay-operator-6586cdc875-sclvq: Created - Created container: kuberay-operator 2026/07/16 22:58:31 22:55:27 INFO Pod/dashboard-redirect-84fcf4d86-k8q6f: Created - Created container: nginx 2026/07/16 22:58:31 22:55:26 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:55:26 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:55:26 INFO Pod/notebook-controller-deployment-647b7d4d45-xwv5d: AddedInterface - Add eth0 [10.130.0.69/23] from ovn-kubernetes 2026/07/16 22:58:31 22:55:26 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-9wdrm: Killing - Stopping container manager 2026/07/16 22:58:31 22:55:26 INFO Pod/kuberay-operator-6586cdc875-sclvq: AddedInterface - Add eth0 [10.130.0.68/23] from ovn-kubernetes 2026/07/16 22:58:31 22:55:26 INFO Pod/kuberay-operator-6586cdc875-sclvq: Pulled - Container image "quay.io/rhoai/odh-kuberay-operator-controller-rhel9@sha256:0e1c5ae8ddd751b0c316f1f50d1f3c8ad4f9b595df1b8354d8361176e3362fa4" already present on machine 2026/07/16 22:58:31 22:55:26 INFO Pod/notebook-controller-deployment-647b7d4d45-xwv5d: Pulling - Pulling image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:d181e8d8cf8607fecde5b06876f74b9e445e9f2288593ab0b75c57648388fd80" 2026/07/16 22:58:31 22:55:25 INFO Pod/notebook-controller-deployment-647b7d4d45-xwv5d: Scheduled - Successfully assigned redhat-ods-applications/notebook-controller-deployment-647b7d4d45-xwv5d to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:55:25 INFO Pod/kuberay-operator-6586cdc875-sclvq: Scheduled - Successfully assigned redhat-ods-applications/kuberay-operator-6586cdc875-sclvq to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:55:25 INFO ReplicaSet/kuberay-operator-6586cdc875: SuccessfulCreate - Created pod: kuberay-operator-6586cdc875-sclvq 2026/07/16 22:58:31 22:55:25 INFO Deployment/kuberay-operator: ScalingReplicaSet - Scaled up replica set kuberay-operator-6586cdc875 from 0 to 1 2026/07/16 22:58:31 22:55:25 INFO ReplicaSet/notebook-controller-deployment-647b7d4d45: SuccessfulCreate - Created pod: notebook-controller-deployment-647b7d4d45-xwv5d 2026/07/16 22:58:31 22:55:25 INFO Deployment/notebook-controller-deployment: ScalingReplicaSet - Scaled up replica set notebook-controller-deployment-647b7d4d45 from 0 to 1 2026/07/16 22:58:31 22:55:25 WARN Pod/kuberay-operator-6586cdc875-sclvq: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kuberay-webhook-server-cert" not found 2026/07/16 22:58:31 22:55:19 INFO Pod/notebook-controller-deployment-647b7d4d45-bv7ng: Killing - Stopping container manager 2026/07/16 22:58:31 22:55:19 INFO Pod/notebook-controller-deployment-647b7d4d45-9tvlz: Killing - Stopping container manager 2026/07/16 22:58:31 22:55:19 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-84cdfcd5d8 from 0 to 2 2026/07/16 22:58:31 22:55:19 WARN ReplicaSet/rhods-dashboard-84cdfcd5d8: FailedCreate - Error creating: pods "rhods-dashboard-84cdfcd5d8-" is forbidden: error looking up service account redhat-ods-applications/rhods-dashboard: serviceaccount "rhods-dashboard" not found 2026/07/16 22:58:31 22:55:16 INFO Pod/dashboard-redirect-84fcf4d86-dt29q: Killing - Stopping container nginx 2026/07/16 22:58:31 22:55:16 INFO Pod/dashboard-redirect-84fcf4d86-bkf9f: Killing - Stopping container nginx 2026/07/16 22:58:31 22:55:14 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-b5fd84b9c-9wdrm_08bbd9e1-de14-4723-a286-6fe5857dff3f became leader 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-br82z: Killing - Stopping container rhods-dashboard 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-8l2mg: Killing - Stopping container kube-rbac-proxy 2026/07/16 22:58:31 22:55:13 WARN Pod/rhods-dashboard-84cdfcd5d8-br82z: Unhealthy - Readiness probe failed: Get "https://10.128.1.201:8444/healthz": dial tcp 10.128.1.201:8444: connect: connection refused 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-br82z: Killing - Stopping container maas-ui 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-br82z: Killing - Stopping container autorag-ui 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-8l2mg: Killing - Stopping container rhods-dashboard 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-8l2mg: Killing - Stopping container eval-hub-ui 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-br82z: Killing - Stopping container eval-hub-ui 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-8l2mg: Killing - Stopping container mlflow-ui 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-8l2mg: Killing - Stopping container gen-ai-ui 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-br82z: Killing - Stopping container kube-rbac-proxy 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-8l2mg: Killing - Stopping container model-registry-ui 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-br82z: Killing - Stopping container model-registry-ui 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-br82z: Killing - Stopping container mlflow-ui 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-br82z: Killing - Stopping container gen-ai-ui 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-br82z: Killing - Stopping container automl-ui 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-8l2mg: Killing - Stopping container maas-ui 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-8l2mg: Killing - Stopping container autorag-ui 2026/07/16 22:58:31 22:55:13 WARN Pod/rhods-dashboard-84cdfcd5d8-8l2mg: Unhealthy - Readiness probe failed: Get "https://10.130.0.42:8444/healthz": dial tcp 10.130.0.42:8444: connect: connection refused 2026/07/16 22:58:31 22:55:13 INFO Pod/rhods-dashboard-84cdfcd5d8-8l2mg: Killing - Stopping container automl-ui 2026/07/16 22:58:31 22:55:11 WARN Pod/odh-notebook-controller-manager-7784c757df-gdvx7: FailedMount - MountVolume.SetUp failed for volume "webhook-cert" : secret "odh-notebook-controller-webhook-cert" not found 2026/07/16 22:58:31 22:55:10 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:55:10 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:55:10 INFO Pod/kubeflow-training-operator-54786944d6-5mx4z: Killing - Stopping container training-operator 2026/07/16 22:58:31 22:55:09 INFO Pod/odh-model-controller-65fdb9c5c-ldvdx: Killing - Stopping container manager 2026/07/16 22:58:31 22:55:09 INFO ReplicaSet/odh-model-controller-65fdb9c5c: SuccessfulDelete - Deleted pod: odh-model-controller-65fdb9c5c-ldvdx 2026/07/16 22:58:31 22:55:09 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled down replica set odh-model-controller-65fdb9c5c from 1 to 0 2026/07/16 22:58:31 22:55:06 INFO Pod/kuberay-operator-6586cdc875-t9v7d: Killing - Stopping container kuberay-operator 2026/07/16 22:58:31 22:55:03 INFO Pod/spark-operator-webhook-5f8768dbfd-6hkbb: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-6hkbb to ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 22:58:31 22:55:03 INFO Pod/spark-operator-webhook-5f8768dbfd-6hkbb: Created - Created container: webhook 2026/07/16 22:58:31 22:55:03 INFO Pod/spark-operator-webhook-5f8768dbfd-6hkbb: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/16 22:58:31 22:55:03 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 1 to 2 2026/07/16 22:58:31 22:55:03 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-6hkbb 2026/07/16 22:58:31 22:55:03 INFO Pod/spark-operator-webhook-5f8768dbfd-6hkbb: AddedInterface - Add eth0 [10.128.1.235/23] from ovn-kubernetes 2026/07/16 22:58:31 22:55:03 INFO Pod/spark-operator-webhook-5f8768dbfd-6hkbb: Started - Started container webhook 2026/07/16 22:58:31 22:55:00 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:55:00 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:55:00 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-xxbzx: Killing - Stopping container manager 2026/07/16 22:58:31 22:55:00 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988mxvj8: Killing - Stopping container manager 2026/07/16 22:58:31 22:54:58 INFO Pod/odh-model-controller-bd4c5cbcb-58scg: Started - Started container manager 2026/07/16 22:58:31 22:54:58 INFO Pod/odh-model-controller-bd4c5cbcb-58scg: Created - Created container: manager 2026/07/16 22:58:31 22:54:57 INFO Pod/odh-model-controller-bd4c5cbcb-58scg: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 22:58:31 22:54:57 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-9wdrm: Started - Started container manager 2026/07/16 22:58:31 22:54:57 INFO Pod/odh-model-controller-bd4c5cbcb-58scg: AddedInterface - Add eth0 [10.129.0.36/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:57 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-9wdrm: AddedInterface - Add eth0 [10.128.1.234/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:57 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-9wdrm: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/16 22:58:31 22:54:57 INFO Pod/odh-model-controller-bd4c5cbcb-58scg: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 615ms (615ms including waiting). Image size: 238959091 bytes. 2026/07/16 22:58:31 22:54:57 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-9wdrm: Created - Created container: manager 2026/07/16 22:58:31 22:54:56 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-9wdrm: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-b5fd84b9c-9wdrm to ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:56 INFO Pod/odh-model-controller-bd4c5cbcb-58scg: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bd4c5cbcb-58scg to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:56 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-b5fd84b9c from 0 to 1 2026/07/16 22:58:31 22:54:56 INFO ReplicaSet/kubeflow-trainer-controller-manager-b5fd84b9c: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-b5fd84b9c-9wdrm 2026/07/16 22:58:31 22:54:56 INFO ReplicaSet/odh-model-controller-bd4c5cbcb: SuccessfulCreate - Created pod: odh-model-controller-bd4c5cbcb-58scg 2026/07/16 22:58:31 22:54:56 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bd4c5cbcb from 0 to 1 2026/07/16 22:58:31 22:54:56 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-vv5bp: Killing - Stopping container manager 2026/07/16 22:58:31 22:54:55 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-gcnrs: Killing - Stopping container manager 2026/07/16 22:58:31 22:54:55 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-d8nz4: Killing - Stopping container manager 2026/07/16 22:58:31 22:54:47 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-5dd7d48f87-2t5xr_a5088b6b-5774-4ae2-b2d3-1e34f3686f66 became leader 2026/07/16 22:58:31 22:54:45 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-njc6r: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 453ms (453ms including waiting). Image size: 236366599 bytes. 2026/07/16 22:58:31 22:54:45 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-njc6r: Created - Created container: manager 2026/07/16 22:58:31 22:54:45 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-njc6r: Started - Started container manager 2026/07/16 22:58:31 22:54:44 INFO Pod/kubeflow-training-operator-54786944d6-5mx4z: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-training-operator-54786944d6-5mx4z to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:44 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-njc6r: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-njc6r to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:44 INFO Pod/kubeflow-training-operator-54786944d6-5mx4z: AddedInterface - Add eth0 [10.129.0.35/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:44 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988mxvj8: Created - Created container: manager 2026/07/16 22:58:31 22:54:44 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-xxbzx: Created - Created container: manager 2026/07/16 22:58:31 22:54:44 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-xxbzx: Pulled - Successfully pulled image "quay.io/rhoai/odh-ogx-k8s-operator-rhel9@sha256:a4ba89c801a3fd9799d2c1fc37b53e66e295d49475bbb211c3d85274a6d6b9dc" in 507ms (507ms including waiting). Image size: 195397995 bytes. 2026/07/16 22:58:31 22:54:44 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-njc6r 2026/07/16 22:58:31 22:54:44 WARN PodDisruptionBudget/ogx-k8s-operator-controller-manager-pdb: CalculateExpectedPodCountFailed - Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-6f9798dc69-fcc4r" 2026/07/16 22:58:31 22:54:44 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-xxbzx: Started - Started container manager 2026/07/16 22:58:31 22:54:44 INFO Pod/kubeflow-training-operator-54786944d6-5mx4z: Pulled - Container image "quay.io/rhoai/odh-training-operator-rhel9@sha256:765c218f8a02f24b4569bb6624087a1ccc843ec73d803eb5408d7d93a64e15b7" already present on machine 2026/07/16 22:58:31 22:54:44 INFO Pod/kubeflow-training-operator-54786944d6-5mx4z: Created - Created container: training-operator 2026/07/16 22:58:31 22:54:44 INFO Lease/spark-operator-webhook-lock: LeaderElection - spark-operator-webhook-5f8768dbfd-flfcm_006446ee-8924-47cb-b725-cc00494eb42f became leader 2026/07/16 22:58:31 22:54:44 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988mxvj8: Started - Started container manager 2026/07/16 22:58:31 22:54:44 INFO Pod/kubeflow-training-operator-54786944d6-5mx4z: Started - Started container training-operator 2026/07/16 22:58:31 22:54:44 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-njc6r: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 22:58:31 22:54:44 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 1 to 2 2026/07/16 22:58:31 22:54:44 INFO Deployment/kubeflow-training-operator: ScalingReplicaSet - Scaled up replica set kubeflow-training-operator-54786944d6 from 0 to 1 2026/07/16 22:58:31 22:54:44 INFO ReplicaSet/kubeflow-training-operator-54786944d6: SuccessfulCreate - Created pod: kubeflow-training-operator-54786944d6-5mx4z 2026/07/16 22:58:31 22:54:44 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-njc6r: AddedInterface - Add eth0 [10.129.0.34/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:43 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988mxvj8: Scheduled - Successfully assigned redhat-ods-applications/data-science-pipelines-operator-controller-manager-7bf7988mxvj8 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:43 INFO Pod/spark-operator-controller-5dd7d48f87-4hx5g: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-4hx5g to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:43 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988mxvj8: Pulled - Container image "quay.io/rhoai/odh-data-science-pipelines-operator-controller-rhel9@sha256:d9eaf8dab3521c7b97f1ed5d48481a4306122bb7bc4b751a965ebc9479dc2d93" already present on machine 2026/07/16 22:58:31 22:54:43 INFO Pod/spark-operator-controller-5dd7d48f87-4hx5g: Started - Started container controller 2026/07/16 22:58:31 22:54:43 INFO Pod/spark-operator-controller-5dd7d48f87-4hx5g: Created - Created container: controller 2026/07/16 22:58:31 22:54:43 INFO Pod/spark-operator-controller-5dd7d48f87-4hx5g: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/16 22:58:31 22:54:43 INFO Pod/spark-operator-controller-5dd7d48f87-4hx5g: AddedInterface - Add eth0 [10.130.0.67/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:43 INFO Pod/kubeflow-training-operator-54786944d6-wp8jq: Killing - Stopping container training-operator 2026/07/16 22:58:31 22:54:43 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-fcc4r: Killing - Stopping container manager 2026/07/16 22:58:31 22:54:43 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-xxbzx: AddedInterface - Add eth0 [10.129.0.32/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:43 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-xxbzx: Pulling - Pulling image "quay.io/rhoai/odh-ogx-k8s-operator-rhel9@sha256:a4ba89c801a3fd9799d2c1fc37b53e66e295d49475bbb211c3d85274a6d6b9dc" 2026/07/16 22:58:31 22:54:43 INFO Deployment/data-science-pipelines-operator-controller-manager: ScalingReplicaSet - Scaled up replica set data-science-pipelines-operator-controller-manager-7bf7988f59 from 0 to 1 2026/07/16 22:58:31 22:54:43 INFO Pod/kubeflow-training-operator-54786944d6-n5pln: Killing - Stopping container training-operator 2026/07/16 22:58:31 22:54:43 WARN PodDisruptionBudget/ogx-k8s-operator-controller-manager-pdb: CalculateExpectedPodCountFailed - Failed to calculate the number of expected pods: found no controllers for pod "ogx-k8s-operator-controller-manager-6f9798dc69-mt84k" 2026/07/16 22:58:31 22:54:43 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988f4t42: Killing - Stopping container manager 2026/07/16 22:58:31 22:54:43 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988mxvj8: AddedInterface - Add eth0 [10.129.0.33/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:43 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-4hx5g 2026/07/16 22:58:31 22:54:43 INFO Pod/data-science-pipelines-operator-controller-manager-7bf79888g85j: Killing - Stopping container manager 2026/07/16 22:58:31 22:54:43 INFO ReplicaSet/data-science-pipelines-operator-controller-manager-7bf7988f59: SuccessfulCreate - Created pod: data-science-pipelines-operator-controller-manager-7bf7988mxvj8 2026/07/16 22:58:31 22:54:42 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-xxbzx: Scheduled - Successfully assigned redhat-ods-applications/ogx-k8s-operator-controller-manager-6f9798dc69-xxbzx to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:42 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-mt84k: Killing - Stopping container manager 2026/07/16 22:58:31 22:54:42 INFO ReplicaSet/ogx-k8s-operator-controller-manager-6f9798dc69: SuccessfulCreate - Created pod: ogx-k8s-operator-controller-manager-6f9798dc69-xxbzx 2026/07/16 22:58:31 22:54:42 INFO Deployment/ogx-k8s-operator-controller-manager: ScalingReplicaSet - Scaled up replica set ogx-k8s-operator-controller-manager-6f9798dc69 from 0 to 1 2026/07/16 22:58:31 22:54:42 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 1 to 2 2026/07/16 22:58:31 22:54:39 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-vv5bp: Pulled - Container image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" already present on machine 2026/07/16 22:58:31 22:54:39 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-vv5bp: Created - Created container: manager 2026/07/16 22:58:31 22:54:39 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-vv5bp: Started - Started container manager 2026/07/16 22:58:31 22:54:38 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-vv5bp: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-74db7cdccd-vv5bp to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:38 INFO ReplicaSet/model-registry-operator-controller-manager-74db7cdccd: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-74db7cdccd-vv5bp 2026/07/16 22:58:31 22:54:38 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-j5dw9: Killing - Stopping container manager 2026/07/16 22:58:31 22:54:38 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-vv5bp: AddedInterface - Add eth0 [10.129.0.31/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:38 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-74db7cdccd from 0 to 1 2026/07/16 22:58:31 22:54:38 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-xvjc9: Killing - Stopping container manager 2026/07/16 22:58:31 22:54:36 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-jmpmw_c0d76f78-b815-4d69-87b9-52eee6546d35 became leader 2026/07/16 22:58:31 22:54:36 INFO Pod/maas-api-5b7784b768-vmqdr: Started - Started container maas-api 2026/07/16 22:58:31 22:54:36 INFO Pod/maas-api-5b7784b768-vmqdr: Created - Created container: maas-api 2026/07/16 22:58:31 22:54:36 INFO Pod/maas-api-5b7784b768-vmqdr: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 530ms (530ms including waiting). Image size: 175018396 bytes. 2026/07/16 22:58:31 22:54:35 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:54:35 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:54:35 INFO Pod/maas-api-5b7784b768-vmqdr: Scheduled - Successfully assigned redhat-ods-applications/maas-api-5b7784b768-vmqdr to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:35 WARN ReplicaSet/maas-api-5b7784b768: FailedCreate - Error creating: pods "maas-api-5b7784b768-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/16 22:58:31 22:54:35 INFO Pod/maas-api-5b7784b768-vmqdr: AddedInterface - Add eth0 [10.129.0.28/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:35 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-5b7784b768 from 0 to 1 2026/07/16 22:58:31 22:54:35 INFO ReplicaSet/maas-api-5b7784b768: SuccessfulCreate - Created pod: maas-api-5b7784b768-vmqdr 2026/07/16 22:58:31 22:54:35 INFO Pod/maas-api-5b7784b768-vmqdr: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/16 22:58:31 22:54:34 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-4vfp6_6ec215fb-1d2f-4e7b-924a-b4bb0ae6f935 became leader 2026/07/16 22:58:31 22:54:29 INFO Pod/spark-operator-webhook-5f8768dbfd-flfcm: Started - Started container webhook 2026/07/16 22:58:31 22:54:29 INFO Pod/spark-operator-webhook-5f8768dbfd-flfcm: AddedInterface - Add eth0 [10.129.0.27/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:29 INFO Pod/spark-operator-webhook-5f8768dbfd-flfcm: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/16 22:58:31 22:54:29 INFO Pod/spark-operator-webhook-5f8768dbfd-flfcm: Created - Created container: webhook 2026/07/16 22:58:31 22:54:28 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:54:28 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:54:28 INFO Pod/spark-operator-webhook-5f8768dbfd-flfcm: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-5f8768dbfd-flfcm to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:28 INFO Pod/spark-operator-controller-5dd7d48f87-2t5xr: Created - Created container: controller 2026/07/16 22:58:31 22:54:28 INFO Pod/spark-operator-controller-5dd7d48f87-2t5xr: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/16 22:58:31 22:54:28 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-5f8768dbfd from 0 to 1 2026/07/16 22:58:31 22:54:28 INFO ReplicaSet/spark-operator-webhook-5f8768dbfd: SuccessfulCreate - Created pod: spark-operator-webhook-5f8768dbfd-flfcm 2026/07/16 22:58:31 22:54:28 INFO Pod/spark-operator-controller-5dd7d48f87-2t5xr: Started - Started container controller 2026/07/16 22:58:31 22:54:28 INFO Pod/spark-operator-controller-5dd7d48f87-2t5xr: AddedInterface - Add eth0 [10.129.0.26/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:27 INFO Pod/spark-operator-controller-5dd7d48f87-2t5xr: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-5dd7d48f87-2t5xr to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:27 INFO ReplicaSet/spark-operator-controller-5dd7d48f87: SuccessfulCreate - Created pod: spark-operator-controller-5dd7d48f87-2t5xr 2026/07/16 22:58:31 22:54:27 WARN ReplicaSet/spark-operator-controller-5dd7d48f87: FailedCreate - Error creating: pods "spark-operator-controller-5dd7d48f87-" is forbidden: error looking up service account redhat-ods-applications/spark-operator-controller: serviceaccount "spark-operator-controller" not found 2026/07/16 22:58:31 22:54:27 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-5dd7d48f87 from 0 to 1 2026/07/16 22:58:31 22:54:25 INFO Pod/kserve-controller-manager-dd9f6cf9-hhvgx: Created - Created container: manager 2026/07/16 22:58:31 22:54:25 INFO Pod/kserve-controller-manager-dd9f6cf9-hhvgx: Started - Started container manager 2026/07/16 22:58:31 22:54:25 INFO Pod/kserve-controller-manager-dd9f6cf9-hhvgx: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 556ms (556ms including waiting). Image size: 238435613 bytes. 2026/07/16 22:58:31 22:54:24 INFO Pod/kserve-controller-manager-dd9f6cf9-hhvgx: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-hhvgx to ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:24 INFO Pod/kserve-controller-manager-dd9f6cf9-hhvgx: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 22:58:31 22:54:24 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 1 to 2 2026/07/16 22:58:31 22:54:24 INFO Pod/kserve-controller-manager-dd9f6cf9-hhvgx: AddedInterface - Add eth0 [10.128.1.233/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:24 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-hhvgx 2026/07/16 22:58:31 22:54:21 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-d8nz4: Started - Started container manager 2026/07/16 22:58:31 22:54:21 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-d8nz4: Created - Created container: manager 2026/07/16 22:58:31 22:54:20 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-d8nz4: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-b5fd84b9c-d8nz4 to ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:20 INFO ReplicaSet/kubeflow-trainer-controller-manager-b5fd84b9c: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-b5fd84b9c-d8nz4 2026/07/16 22:58:31 22:54:20 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-d8nz4: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/16 22:58:31 22:54:20 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-b5fd84b9c from 1 to 2 2026/07/16 22:58:31 22:54:20 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-d8nz4: AddedInterface - Add eth0 [10.128.1.232/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:17 INFO Pod/kserve-controller-manager-dd9f6cf9-4vfp6: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 391ms (391ms including waiting). Image size: 238435613 bytes. 2026/07/16 22:58:31 22:54:17 INFO Pod/kserve-controller-manager-dd9f6cf9-4vfp6: Started - Started container manager 2026/07/16 22:58:31 22:54:17 INFO Pod/kserve-controller-manager-dd9f6cf9-4vfp6: Created - Created container: manager 2026/07/16 22:58:31 22:54:16 WARN Pod/kserve-controller-manager-dd9f6cf9-4vfp6: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/16 22:58:31 22:54:16 INFO Pod/kserve-controller-manager-dd9f6cf9-4vfp6: AddedInterface - Add eth0 [10.129.0.25/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:16 INFO Pod/spark-operator-webhook-5f8768dbfd-c8hcf: Killing - Stopping container webhook 2026/07/16 22:58:31 22:54:16 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmpmw: AddedInterface - Add eth0 [10.128.1.231/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:16 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmpmw: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 22:58:31 22:54:16 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmpmw: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 421ms (421ms including waiting). Image size: 236366599 bytes. 2026/07/16 22:58:31 22:54:16 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmpmw: Created - Created container: manager 2026/07/16 22:58:31 22:54:16 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmpmw: Started - Started container manager 2026/07/16 22:58:31 22:54:16 INFO Pod/kserve-controller-manager-dd9f6cf9-4vfp6: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 22:58:31 22:54:15 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jmpmw: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-jmpmw to ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:15 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:54:15 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:54:15 INFO Pod/kserve-controller-manager-dd9f6cf9-4vfp6: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-4vfp6 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:15 INFO Pod/odh-notebook-controller-manager-7784c757df-5h26v: Pulled - Successfully pulled image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:d70fede8c54fb2edc8691360ee77339cb81bb3de0a39f7adb863c8754eaff13d" in 510ms (510ms including waiting). Image size: 173529551 bytes. 2026/07/16 22:58:31 22:54:15 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-jmpmw 2026/07/16 22:58:31 22:54:15 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/16 22:58:31 22:54:15 INFO Pod/spark-operator-controller-5dd7d48f87-7994n: Killing - Stopping container controller 2026/07/16 22:58:31 22:54:15 INFO Pod/odh-notebook-controller-manager-7784c757df-5h26v: Created - Created container: manager 2026/07/16 22:58:31 22:54:15 INFO Pod/odh-notebook-controller-manager-7784c757df-5h26v: Started - Started container manager 2026/07/16 22:58:31 22:54:15 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/16 22:58:31 22:54:15 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-4vfp6 2026/07/16 22:58:31 22:54:14 INFO Pod/odh-notebook-controller-manager-7784c757df-5h26v: AddedInterface - Add eth0 [10.130.0.66/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:14 INFO Pod/odh-notebook-controller-manager-7784c757df-5h26v: Pulling - Pulling image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:d70fede8c54fb2edc8691360ee77339cb81bb3de0a39f7adb863c8754eaff13d" 2026/07/16 22:58:31 22:54:14 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-b5fd84b9c-gcnrs_e031810e-a0af-4f35-9b6d-ba808746b157 became leader 2026/07/16 22:58:31 22:54:13 INFO Pod/odh-notebook-controller-manager-7784c757df-5h26v: Scheduled - Successfully assigned redhat-ods-applications/odh-notebook-controller-manager-7784c757df-5h26v to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:13 INFO Pod/kubeflow-training-operator-54786944d6-wp8jq: Created - Created container: training-operator 2026/07/16 22:58:31 22:54:13 INFO Pod/kubeflow-training-operator-54786944d6-wp8jq: AddedInterface - Add eth0 [10.128.1.230/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:13 INFO Pod/kubeflow-training-operator-54786944d6-wp8jq: Pulled - Container image "quay.io/rhoai/odh-training-operator-rhel9@sha256:765c218f8a02f24b4569bb6624087a1ccc843ec73d803eb5408d7d93a64e15b7" already present on machine 2026/07/16 22:58:31 22:54:13 INFO Pod/kubeflow-training-operator-54786944d6-wp8jq: Started - Started container training-operator 2026/07/16 22:58:31 22:54:13 INFO Deployment/odh-notebook-controller-manager: ScalingReplicaSet - Scaled up replica set odh-notebook-controller-manager-7784c757df from 1 to 2 2026/07/16 22:58:31 22:54:13 INFO ReplicaSet/odh-notebook-controller-manager-7784c757df: SuccessfulCreate - Created pod: odh-notebook-controller-manager-7784c757df-5h26v 2026/07/16 22:58:31 22:54:12 INFO Pod/kubeflow-training-operator-54786944d6-wp8jq: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-training-operator-54786944d6-wp8jq to ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:12 INFO ReplicaSet/kubeflow-training-operator-54786944d6: SuccessfulCreate - Created pod: kubeflow-training-operator-54786944d6-wp8jq 2026/07/16 22:58:31 22:54:12 INFO Pod/ai-gateway-operator-64fcd4bb69-hxh4d: Pulled - Successfully pulled image "quay.io/opendatahub/odh-ai-gateway-operator:odh-stable" in 6.979s (6.979s including waiting). Image size: 205242175 bytes. 2026/07/16 22:58:31 22:54:12 INFO Pod/ai-gateway-operator-64fcd4bb69-hxh4d: Created - Created container: copy-manifests 2026/07/16 22:58:31 22:54:12 INFO Pod/ai-gateway-operator-64fcd4bb69-hxh4d: Started - Started container copy-manifests 2026/07/16 22:58:31 22:54:12 INFO Deployment/kubeflow-training-operator: ScalingReplicaSet - Scaled up replica set kubeflow-training-operator-54786944d6 from 1 to 2 2026/07/16 22:58:31 22:54:07 INFO Pod/ai-gateway-operator-86947886bc-5tssj: Killing - Stopping container manager 2026/07/16 22:58:31 22:54:06 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:54:06 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:54:05 INFO Pod/ai-gateway-operator-64fcd4bb69-hxh4d: Pulling - Pulling image "quay.io/opendatahub/odh-ai-gateway-operator:odh-stable" 2026/07/16 22:58:31 22:54:05 INFO Pod/ai-gateway-operator-64fcd4bb69-hxh4d: AddedInterface - Add eth0 [10.129.0.24/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:04 INFO Pod/ai-gateway-operator-64fcd4bb69-hxh4d: Scheduled - Successfully assigned redhat-ods-applications/ai-gateway-operator-64fcd4bb69-hxh4d to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:04 INFO ReplicaSet/ai-gateway-operator-64fcd4bb69: SuccessfulCreate - Created pod: ai-gateway-operator-64fcd4bb69-hxh4d 2026/07/16 22:58:31 22:54:04 INFO Deployment/ai-gateway-operator: ScalingReplicaSet - Scaled up replica set ai-gateway-operator-64fcd4bb69 from 0 to 1 2026/07/16 22:58:31 22:54:03 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-xvjc9: Pulled - Container image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" already present on machine 2026/07/16 22:58:31 22:54:03 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-xvjc9: Created - Created container: manager 2026/07/16 22:58:31 22:54:03 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-fcc4r: AddedInterface - Add eth0 [10.130.0.63/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:03 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-xvjc9: Started - Started container manager 2026/07/16 22:58:31 22:54:03 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-75bbfd978-tlmxf_6f73bd48-8959-4847-80c8-39315b85b801 became leader 2026/07/16 22:58:31 22:54:03 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-xvjc9: AddedInterface - Add eth0 [10.130.0.64/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:03 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-fcc4r: Pulling - Pulling image "quay.io/rhoai/odh-ogx-k8s-operator-rhel9@sha256:a4ba89c801a3fd9799d2c1fc37b53e66e295d49475bbb211c3d85274a6d6b9dc" 2026/07/16 22:58:31 22:54:03 INFO Pod/data-science-pipelines-operator-controller-manager-7bf79888g85j: AddedInterface - Add eth0 [10.130.0.65/23] from ovn-kubernetes 2026/07/16 22:58:31 22:54:03 INFO Pod/data-science-pipelines-operator-controller-manager-7bf79888g85j: Pulled - Container image "quay.io/rhoai/odh-data-science-pipelines-operator-controller-rhel9@sha256:d9eaf8dab3521c7b97f1ed5d48481a4306122bb7bc4b751a965ebc9479dc2d93" already present on machine 2026/07/16 22:58:31 22:54:03 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-fcc4r: Created - Created container: manager 2026/07/16 22:58:31 22:54:03 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-fcc4r: Started - Started container manager 2026/07/16 22:58:31 22:54:03 INFO Pod/data-science-pipelines-operator-controller-manager-7bf79888g85j: Created - Created container: manager 2026/07/16 22:58:31 22:54:03 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-fcc4r: Pulled - Successfully pulled image "quay.io/rhoai/odh-ogx-k8s-operator-rhel9@sha256:a4ba89c801a3fd9799d2c1fc37b53e66e295d49475bbb211c3d85274a6d6b9dc" in 613ms (613ms including waiting). Image size: 195397995 bytes. 2026/07/16 22:58:31 22:54:03 INFO Pod/data-science-pipelines-operator-controller-manager-7bf79888g85j: Started - Started container manager 2026/07/16 22:58:31 22:54:02 INFO Pod/data-science-pipelines-operator-controller-manager-7bf79888g85j: Scheduled - Successfully assigned redhat-ods-applications/data-science-pipelines-operator-controller-manager-7bf79888g85j to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:02 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-xvjc9: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-74db7cdccd-xvjc9 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:02 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-fcc4r: Scheduled - Successfully assigned redhat-ods-applications/ogx-k8s-operator-controller-manager-6f9798dc69-fcc4r to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:54:02 INFO Deployment/data-science-pipelines-operator-controller-manager: ScalingReplicaSet - Scaled up replica set data-science-pipelines-operator-controller-manager-7bf7988f59 from 1 to 2 2026/07/16 22:58:31 22:54:02 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-74db7cdccd from 1 to 2 2026/07/16 22:58:31 22:54:02 INFO Deployment/ogx-k8s-operator-controller-manager: ScalingReplicaSet - Scaled up replica set ogx-k8s-operator-controller-manager-6f9798dc69 from 1 to 2 2026/07/16 22:58:31 22:54:02 INFO ReplicaSet/data-science-pipelines-operator-controller-manager-7bf7988f59: SuccessfulCreate - Created pod: data-science-pipelines-operator-controller-manager-7bf79888g85j 2026/07/16 22:58:31 22:54:02 INFO ReplicaSet/ogx-k8s-operator-controller-manager-6f9798dc69: SuccessfulCreate - Created pod: ogx-k8s-operator-controller-manager-6f9798dc69-fcc4r 2026/07/16 22:58:31 22:54:02 INFO ReplicaSet/model-registry-operator-controller-manager-74db7cdccd: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-74db7cdccd-xvjc9 2026/07/16 22:58:31 22:54:02 INFO Lease/54e06e98.ogx.io: LeaderElection - ogx-k8s-operator-controller-manager-6f9798dc69-mt84k_9011d2c4-3fa7-4f5f-a8e0-763db5fdf1ff became leader 2026/07/16 22:58:31 22:53:59 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-65fdb9c5c-ldvdx_fa5faf39-199e-4fbd-acf7-45f1b2456b72 became leader 2026/07/16 22:58:31 22:53:59 INFO Lease/66483dbd.feast.dev: LeaderElection - feast-operator-controller-manager-7b7cd769f7-l7lj7_c9049747-3238-4d24-afcf-7426301d05e0 became leader 2026/07/16 22:58:31 22:53:58 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-gcnrs: Created - Created container: manager 2026/07/16 22:58:31 22:53:58 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-74db7cdccd-j5dw9_81533755-83d1-45dd-8eb8-fbd802139e9c became leader 2026/07/16 22:58:31 22:53:58 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-gcnrs: Started - Started container manager 2026/07/16 22:58:31 22:53:58 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-gcnrs: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/16 22:58:31 22:53:58 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-gcnrs: AddedInterface - Add eth0 [10.129.0.22/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:57 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-gcnrs: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-b5fd84b9c-gcnrs to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:57 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-b5fd84b9c from 0 to 1 2026/07/16 22:58:31 22:53:57 INFO ReplicaSet/kubeflow-trainer-controller-manager-b5fd84b9c: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-b5fd84b9c-gcnrs 2026/07/16 22:58:31 22:53:55 INFO Pod/notebook-controller-deployment-647b7d4d45-9tvlz: Pulled - Successfully pulled image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:d181e8d8cf8607fecde5b06876f74b9e445e9f2288593ab0b75c57648388fd80" in 688ms (688ms including waiting). Image size: 168353804 bytes. 2026/07/16 22:58:31 22:53:55 INFO Pod/notebook-controller-deployment-647b7d4d45-9tvlz: Started - Started container manager 2026/07/16 22:58:31 22:53:55 INFO Pod/notebook-controller-deployment-647b7d4d45-9tvlz: Created - Created container: manager 2026/07/16 22:58:31 22:53:54 INFO Pod/notebook-controller-deployment-647b7d4d45-9tvlz: Pulling - Pulling image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:d181e8d8cf8607fecde5b06876f74b9e445e9f2288593ab0b75c57648388fd80" 2026/07/16 22:58:31 22:53:54 INFO Pod/notebook-controller-deployment-647b7d4d45-9tvlz: AddedInterface - Add eth0 [10.130.0.62/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:53 INFO Pod/notebook-controller-deployment-647b7d4d45-9tvlz: Scheduled - Successfully assigned redhat-ods-applications/notebook-controller-deployment-647b7d4d45-9tvlz to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:53 INFO Deployment/notebook-controller-deployment: ScalingReplicaSet - Scaled up replica set notebook-controller-deployment-647b7d4d45 from 1 to 2 2026/07/16 22:58:31 22:53:53 INFO ReplicaSet/notebook-controller-deployment-647b7d4d45: SuccessfulCreate - Created pod: notebook-controller-deployment-647b7d4d45-9tvlz 2026/07/16 22:58:31 22:53:50 INFO Pod/notebook-controller-deployment-647b7d4d45-bv7ng: Pulled - Successfully pulled image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:d181e8d8cf8607fecde5b06876f74b9e445e9f2288593ab0b75c57648388fd80" in 580ms (580ms including waiting). Image size: 168353804 bytes. 2026/07/16 22:58:31 22:53:50 INFO Pod/notebook-controller-deployment-647b7d4d45-bv7ng: Created - Created container: manager 2026/07/16 22:58:31 22:53:50 INFO Pod/notebook-controller-deployment-647b7d4d45-bv7ng: Started - Started container manager 2026/07/16 22:58:31 22:53:50 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qtsrw: Killing - Stopping container manager 2026/07/16 22:58:31 22:53:49 INFO Pod/odh-notebook-controller-manager-7784c757df-gdvx7: Pulled - Successfully pulled image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:d70fede8c54fb2edc8691360ee77339cb81bb3de0a39f7adb863c8754eaff13d" in 775ms (775ms including waiting). Image size: 173529551 bytes. 2026/07/16 22:58:31 22:53:49 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qtsrw: Started - Started container manager 2026/07/16 22:58:31 22:53:49 INFO Pod/kserve-controller-manager-dd9f6cf9-n5t42: Started - Started container manager 2026/07/16 22:58:31 22:53:49 INFO Pod/odh-notebook-controller-manager-7784c757df-gdvx7: Started - Started container manager 2026/07/16 22:58:31 22:53:49 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qtsrw: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 1.16s (1.16s including waiting). Image size: 236366599 bytes. 2026/07/16 22:58:31 22:53:49 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qtsrw: Created - Created container: manager 2026/07/16 22:58:31 22:53:49 INFO Pod/kserve-controller-manager-dd9f6cf9-n5t42: Created - Created container: manager 2026/07/16 22:58:31 22:53:49 INFO Pod/notebook-controller-deployment-647b7d4d45-bv7ng: AddedInterface - Add eth0 [10.129.0.21/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:49 INFO Pod/kserve-controller-manager-dd9f6cf9-n5t42: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 581ms (581ms including waiting). Image size: 238435613 bytes. 2026/07/16 22:58:31 22:53:49 INFO Pod/notebook-controller-deployment-647b7d4d45-bv7ng: Pulling - Pulling image "quay.io/rhoai/odh-kf-notebook-controller-rhel9@sha256:d181e8d8cf8607fecde5b06876f74b9e445e9f2288593ab0b75c57648388fd80" 2026/07/16 22:58:31 22:53:49 INFO Pod/odh-notebook-controller-manager-7784c757df-gdvx7: Created - Created container: manager 2026/07/16 22:58:31 22:53:48 INFO Pod/notebook-controller-deployment-647b7d4d45-bv7ng: Scheduled - Successfully assigned redhat-ods-applications/notebook-controller-deployment-647b7d4d45-bv7ng to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:48 INFO Pod/odh-notebook-controller-manager-7784c757df-gdvx7: Scheduled - Successfully assigned redhat-ods-applications/odh-notebook-controller-manager-7784c757df-gdvx7 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:48 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qtsrw: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-qtsrw to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:48 INFO Pod/kserve-controller-manager-dd9f6cf9-n5t42: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-n5t42 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:48 INFO Pod/kserve-controller-manager-dd9f6cf9-n5t42: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 22:58:31 22:53:48 INFO Pod/odh-notebook-controller-manager-7784c757df-gdvx7: AddedInterface - Add eth0 [10.129.0.20/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:48 INFO Lease/ai-gateway-lock: LeaderElection - ai-gateway-operator-86947886bc-5tssj_8e88d12c-e123-43e2-90b9-451c511b5834 became leader 2026/07/16 22:58:31 22:53:48 INFO Deployment/odh-notebook-controller-manager: ScalingReplicaSet - Scaled up replica set odh-notebook-controller-manager-7784c757df from 0 to 1 2026/07/16 22:58:31 22:53:48 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qtsrw: AddedInterface - Add eth0 [10.129.0.19/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:48 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-qtsrw: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 22:58:31 22:53:48 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-n5t42 2026/07/16 22:58:31 22:53:48 INFO ReplicaSet/odh-notebook-controller-manager-7784c757df: SuccessfulCreate - Created pod: odh-notebook-controller-manager-7784c757df-gdvx7 2026/07/16 22:58:31 22:53:48 INFO Pod/odh-notebook-controller-manager-7784c757df-gdvx7: Pulling - Pulling image "quay.io/rhoai/odh-notebook-controller-rhel9@sha256:d70fede8c54fb2edc8691360ee77339cb81bb3de0a39f7adb863c8754eaff13d" 2026/07/16 22:58:31 22:53:48 INFO Pod/kserve-controller-manager-dd9f6cf9-n5t42: AddedInterface - Add eth0 [10.129.0.18/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:48 INFO ReplicaSet/notebook-controller-deployment-647b7d4d45: SuccessfulCreate - Created pod: notebook-controller-deployment-647b7d4d45-bv7ng 2026/07/16 22:58:31 22:53:48 INFO Deployment/notebook-controller-deployment: ScalingReplicaSet - Scaled up replica set notebook-controller-deployment-647b7d4d45 from 0 to 1 2026/07/16 22:58:31 22:53:48 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/16 22:58:31 22:53:48 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/16 22:58:31 22:53:48 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-qtsrw 2026/07/16 22:58:31 22:53:45 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:53:45 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:53:45 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-hhzl5: Killing - Stopping container manager 2026/07/16 22:58:31 22:53:45 INFO Pod/maas-controller-75bbfd978-tlmxf: Started - Started container manager 2026/07/16 22:58:31 22:53:45 INFO Pod/kubeflow-training-operator-54786944d6-n5pln: Created - Created container: training-operator 2026/07/16 22:58:31 22:53:45 INFO Pod/kubeflow-training-operator-54786944d6-n5pln: Pulled - Container image "quay.io/rhoai/odh-training-operator-rhel9@sha256:765c218f8a02f24b4569bb6624087a1ccc843ec73d803eb5408d7d93a64e15b7" already present on machine 2026/07/16 22:58:31 22:53:45 INFO Pod/maas-controller-75bbfd978-tlmxf: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 699ms (699ms including waiting). Image size: 212866077 bytes. 2026/07/16 22:58:31 22:53:45 INFO Pod/kubeflow-training-operator-54786944d6-n5pln: Started - Started container training-operator 2026/07/16 22:58:31 22:53:45 INFO Pod/maas-controller-75bbfd978-tlmxf: Created - Created container: manager 2026/07/16 22:58:31 22:53:45 INFO Pod/kubeflow-training-operator-54786944d6-n5pln: AddedInterface - Add eth0 [10.129.0.17/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:44 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:53:44 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:53:44 INFO Pod/kubeflow-training-operator-54786944d6-n5pln: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-training-operator-54786944d6-n5pln to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:44 INFO Pod/maas-controller-75bbfd978-tlmxf: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-75bbfd978-tlmxf to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:44 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:53:44 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:53:44 INFO Pod/model-serving-api-5679d6b4f-vf7m9: AddedInterface - Add eth0 [10.129.0.15/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:44 INFO Pod/maas-controller-75bbfd978-tlmxf: AddedInterface - Add eth0 [10.129.0.16/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:44 INFO Pod/model-serving-api-5679d6b4f-vf7m9: Started - Started container server 2026/07/16 22:58:31 22:53:44 INFO ReplicaSet/kubeflow-training-operator-54786944d6: SuccessfulCreate - Created pod: kubeflow-training-operator-54786944d6-n5pln 2026/07/16 22:58:31 22:53:44 INFO Pod/maas-controller-75bbfd978-tlmxf: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/16 22:58:31 22:53:44 INFO ReplicaSet/maas-controller-75bbfd978: SuccessfulCreate - Created pod: maas-controller-75bbfd978-tlmxf 2026/07/16 22:58:31 22:53:44 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-mt84k: Started - Started container manager 2026/07/16 22:58:31 22:53:44 INFO Pod/model-serving-api-5679d6b4f-vf7m9: Created - Created container: server 2026/07/16 22:58:31 22:53:44 INFO Deployment/kubeflow-training-operator: ScalingReplicaSet - Scaled up replica set kubeflow-training-operator-54786944d6 from 0 to 1 2026/07/16 22:58:31 22:53:44 INFO Pod/model-serving-api-5679d6b4f-vf7m9: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 22:58:31 22:53:44 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-mt84k: Created - Created container: manager 2026/07/16 22:58:31 22:53:44 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-mt84k: Pulled - Successfully pulled image "quay.io/rhoai/odh-ogx-k8s-operator-rhel9@sha256:a4ba89c801a3fd9799d2c1fc37b53e66e295d49475bbb211c3d85274a6d6b9dc" in 639ms (640ms including waiting). Image size: 195397995 bytes. 2026/07/16 22:58:31 22:53:44 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-75bbfd978 from 0 to 1 2026/07/16 22:58:31 22:53:43 INFO Pod/model-serving-api-5679d6b4f-vf7m9: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-vf7m9 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:43 INFO Pod/feast-operator-controller-manager-7b7cd769f7-l7lj7: Started - Started container manager 2026/07/16 22:58:31 22:53:43 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-mt84k: Pulling - Pulling image "quay.io/rhoai/odh-ogx-k8s-operator-rhel9@sha256:a4ba89c801a3fd9799d2c1fc37b53e66e295d49475bbb211c3d85274a6d6b9dc" 2026/07/16 22:58:31 22:53:43 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/16 22:58:31 22:53:43 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-vf7m9 2026/07/16 22:58:31 22:53:43 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 22:58:31 22:53:43 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-mt84k: AddedInterface - Add eth0 [10.129.0.13/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:43 INFO Pod/odh-model-controller-65fdb9c5c-ldvdx: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 779ms (779ms including waiting). Image size: 238959091 bytes. 2026/07/16 22:58:31 22:53:43 INFO Pod/feast-operator-controller-manager-7b7cd769f7-l7lj7: AddedInterface - Add eth0 [10.129.0.14/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:43 INFO Pod/feast-operator-controller-manager-7b7cd769f7-l7lj7: Pulled - Container image "quay.io/rhoai/odh-feast-operator-rhel9@sha256:41d9733e38100e5412194dccb5d0c8a44db376fdb2c77011628bb38f0f650719" already present on machine 2026/07/16 22:58:31 22:53:43 INFO Pod/odh-model-controller-65fdb9c5c-ldvdx: Created - Created container: manager 2026/07/16 22:58:31 22:53:43 INFO Pod/odh-model-controller-65fdb9c5c-ldvdx: Started - Started container manager 2026/07/16 22:58:31 22:53:43 INFO Pod/feast-operator-controller-manager-7b7cd769f7-l7lj7: Created - Created container: manager 2026/07/16 22:58:31 22:53:42 INFO Pod/feast-operator-controller-manager-7b7cd769f7-l7lj7: Scheduled - Successfully assigned redhat-ods-applications/feast-operator-controller-manager-7b7cd769f7-l7lj7 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:42 INFO Pod/odh-model-controller-65fdb9c5c-ldvdx: AddedInterface - Add eth0 [10.129.0.11/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:42 INFO Deployment/feast-operator-controller-manager: ScalingReplicaSet - Scaled up replica set feast-operator-controller-manager-7b7cd769f7 from 0 to 1 2026/07/16 22:58:31 22:53:42 INFO ReplicaSet/feast-operator-controller-manager-7b7cd769f7: SuccessfulCreate - Created pod: feast-operator-controller-manager-7b7cd769f7-l7lj7 2026/07/16 22:58:31 22:53:42 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-j5dw9: AddedInterface - Add eth0 [10.129.0.10/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:42 INFO Pod/odh-model-controller-65fdb9c5c-ldvdx: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 22:58:31 22:53:42 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-j5dw9: Created - Created container: manager 2026/07/16 22:58:31 22:53:42 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-j5dw9: Started - Started container manager 2026/07/16 22:58:31 22:53:42 WARN Pod/ogx-k8s-operator-controller-manager-6f9798dc69-mt84k: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "ogx-k8s-operator-webhook-cert" not found 2026/07/16 22:58:31 22:53:42 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-j5dw9: Pulled - Container image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" already present on machine 2026/07/16 22:58:31 22:53:41 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-mt84k: Scheduled - Successfully assigned redhat-ods-applications/ogx-k8s-operator-controller-manager-6f9798dc69-mt84k to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:41 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:53:41 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:53:41 INFO ReplicaSet/ogx-k8s-operator-controller-manager-6f9798dc69: SuccessfulCreate - Created pod: ogx-k8s-operator-controller-manager-6f9798dc69-mt84k 2026/07/16 22:58:31 22:53:41 INFO Deployment/ogx-k8s-operator-controller-manager: ScalingReplicaSet - Scaled up replica set ogx-k8s-operator-controller-manager-6f9798dc69 from 0 to 1 2026/07/16 22:58:31 22:53:41 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988f4t42: Started - Started container manager 2026/07/16 22:58:31 22:53:41 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988f4t42: Created - Created container: manager 2026/07/16 22:58:31 22:53:41 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988f4t42: Pulled - Container image "quay.io/rhoai/odh-data-science-pipelines-operator-controller-rhel9@sha256:d9eaf8dab3521c7b97f1ed5d48481a4306122bb7bc4b751a965ebc9479dc2d93" already present on machine 2026/07/16 22:58:31 22:53:41 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988f4t42: AddedInterface - Add eth0 [10.129.0.12/23] from ovn-kubernetes 2026/07/16 22:58:31 22:53:41 WARN Pod/odh-model-controller-65fdb9c5c-ldvdx: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/16 22:58:31 22:53:40 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:53:40 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988f4t42: Scheduled - Successfully assigned redhat-ods-applications/data-science-pipelines-operator-controller-manager-7bf7988f4t42 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:40 INFO Pod/odh-model-controller-65fdb9c5c-ldvdx: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-65fdb9c5c-ldvdx to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:40 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-j5dw9: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-74db7cdccd-j5dw9 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 22:58:31 22:53:40 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-74db7cdccd from 0 to 1 2026/07/16 22:58:31 22:53:40 INFO Deployment/data-science-pipelines-operator-controller-manager: ScalingReplicaSet - Scaled up replica set data-science-pipelines-operator-controller-manager-7bf7988f59 from 0 to 1 2026/07/16 22:58:31 22:53:40 INFO ReplicaSet/odh-model-controller-65fdb9c5c: SuccessfulCreate - Created pod: odh-model-controller-65fdb9c5c-ldvdx 2026/07/16 22:58:31 22:53:40 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-65fdb9c5c from 0 to 1 2026/07/16 22:58:31 22:53:40 INFO ReplicaSet/model-registry-operator-controller-manager-74db7cdccd: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-74db7cdccd-j5dw9 2026/07/16 22:58:31 22:53:40 WARN Pod/data-science-pipelines-operator-controller-manager-7bf7988f4t42: FailedMount - MountVolume.SetUp failed for volume "config" : configmap "data-science-pipelines-operator-dspo-config" not found 2026/07/16 22:58:31 22:53:40 WARN Pod/model-registry-operator-controller-manager-74db7cdccd-j5dw9: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/07/16 22:58:31 22:53:40 INFO ReplicaSet/data-science-pipelines-operator-controller-manager-7bf7988f59: SuccessfulCreate - Created pod: data-science-pipelines-operator-controller-manager-7bf7988f4t42 2026/07/16 22:58:31 22:53:35 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:53:35 INFO Pod/kubeflow-training-operator-54786944d6-mhhx4: Killing - Stopping container training-operator 2026/07/16 22:58:31 22:53:35 INFO Pod/model-serving-api-5679d6b4f-d594b: Killing - Stopping container server 2026/07/16 22:58:31 22:53:35 WARN Pod/maas-api-8f8fd6cd4-kwpjv: Unhealthy - Readiness probe failed: Get "https://10.129.1.219:8443/health": dial tcp 10.129.1.219:8443: connect: connection refused 2026/07/16 22:58:31 22:53:35 INFO Pod/odh-model-controller-65fdb9c5c-qsmv4: Killing - Stopping container manager 2026/07/16 22:58:31 22:53:35 INFO Pod/notebook-controller-deployment-647b7d4d45-fs4zz: Killing - Stopping container manager 2026/07/16 22:58:31 22:53:35 INFO Pod/maas-controller-75bbfd978-p8gbx: Killing - Stopping container manager 2026/07/16 22:58:31 22:53:35 INFO Pod/ogx-k8s-operator-controller-manager-6f9798dc69-j89gx: Killing - Stopping container manager 2026/07/16 22:58:31 22:53:35 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-lcft7: Killing - Stopping container manager 2026/07/16 22:58:31 22:53:35 INFO Pod/maas-api-8f8fd6cd4-kwpjv: Killing - Stopping container maas-api 2026/07/16 22:58:31 22:53:35 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-4tqnw: Killing - Stopping container manager 2026/07/16 22:58:31 22:53:35 INFO Pod/odh-notebook-controller-manager-7784c757df-hs92w: Killing - Stopping container manager 2026/07/16 22:58:31 22:53:35 INFO Pod/kserve-controller-manager-dd9f6cf9-qnjks: Killing - Stopping container manager 2026/07/16 22:58:31 22:53:34 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:53:34 INFO Pod/data-science-pipelines-operator-controller-manager-7bf7988w7km9: Killing - Stopping container manager 2026/07/16 22:58:31 22:53:34 INFO Pod/feast-operator-controller-manager-7b7cd769f7-m8h4v: Killing - Stopping container manager 2026/07/16 22:58:31 22:53:33 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 22:53:32 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 22:58:31 === RESOURCE QUOTAS === 2026/07/16 22:58:31 No resource quotas found in monitored namespaces 2026/07/16 22:58:31 Classification for group 1: infrastructure/storage (code=1006, confidence=medium) 2026/07/16 22:58:31 Evidence: event Pod/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297-driver: FailedMount - MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-861d419f6d258bb1-conf-map" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/scheduled-spark-pi-d9cm30rfh92s7b6ih600-1784242593000652297-driver: FailedMount - MountVolume.SetUp failed for volume \"spark-conf-volume-driver\" : configmap \"spark-drv-861d419f6d258bb1-conf-map\" not found"],"confidence":"medium"} 2026/07/16 22:58:31 Diagnostics complete for failed test: group 1 controller_test.go:753: Stopping: group 1 test failed. 2026/07/16 22:58:31 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/16 22:58:31 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/16 23:02:14 Circuit breaker: success recorded, resetting failure counter (was 1) 2026/07/16 23:15:05 TEST FAILURE DETECTED: group 4 2026/07/16 23:15:05 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/16 23:15:06 === CLUSTER STATE === 2026/07/16 23:15:06 Node: ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 23:15:06 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 23:15:06 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 23:15:06 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 23:15:06 Condition Ready: True - kubelet is posting ready status 2026/07/16 23:15:06 Resources: 15500m CPU, 63871020Ki memory allocatable (of 16000m CPU, 65021996Ki memory capacity) 2026/07/16 23:15:06 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 23:15:06 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 23:15:06 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 23:15:06 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 23:15:06 Condition Ready: True - kubelet is posting ready status 2026/07/16 23:15:06 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/16 23:15:06 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 23:15:06 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 23:15:06 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 23:15:06 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 23:15:06 Condition Ready: True - kubelet is posting ready status 2026/07/16 23:15:06 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/16 23:15:06 === DEPLOYMENTS === 2026/07/16 23:15:06 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/16 23:15:06 === PODS === 2026/07/16 23:15:06 Failed to collect pod data: redhat-ods-applications/maas-controller-75bbfd978-jk2xw: container manager not ready 2026/07/16 23:15:06 === OPERATOR STATUS === 2026/07/16 23:15:06 Operator deployment: 3/3 ready 2026/07/16 23:15:06 === DSCI STATUS === 2026/07/16 23:15:06 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/16 23:15:06 === DSC STATUS === 2026/07/16 23:15:06 DSC default-dsc: 2026/07/16 23:15:06 Ready: True - 2026/07/16 23:15:06 ProvisioningSucceeded: True - 2026/07/16 23:15:06 ComponentsReady: True - 2026/07/16 23:15:06 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/16 23:15:06 DashboardReady: False - Component ManagementState is set to Removed 2026/07/16 23:15:06 FeastOperatorReady: True - 2026/07/16 23:15:06 KserveLLMInferenceServiceDependencies: True - 2026/07/16 23:15:06 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/16 23:15:06 KserveReady: True - 2026/07/16 23:15:06 KueueReady: False - Component ManagementState is set to Removed 2026/07/16 23:15:06 LLM-D-WVADependencies: True - 2026/07/16 23:15:06 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 23:15:06 ModelControllerReady: True - 2026/07/16 23:15:06 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/16 23:15:06 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/16 23:15:06 OGXReady: False - Component ManagementState is set to Removed 2026/07/16 23:15:06 RayReady: False - Component ManagementState is set to Removed 2026/07/16 23:15:06 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 23:15:06 TrainerReady: False - Component ManagementState is set to Removed 2026/07/16 23:15:06 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 23:15:06 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/16 23:15:06 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/16 23:15:06 === RECENT EVENTS === 2026/07/16 23:15:06 23:15:04 INFO Job/maas-api-key-cleanup-29737395: Completed - Job completed 2026/07/16 23:15:06 23:15:04 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29737395, condition: Complete 2026/07/16 23:15:06 23:15:01 INFO Pod/maas-api-key-cleanup-29737395-vcbnv: Created - Created container: cleanup 2026/07/16 23:15:06 23:15:01 INFO Pod/maas-api-key-cleanup-29737395-vcbnv: Started - Started container cleanup 2026/07/16 23:15:06 23:15:00 INFO Pod/maas-api-key-cleanup-29737395-vcbnv: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29737395-vcbnv to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 23:15:06 23:15:00 INFO Job/maas-api-key-cleanup-29737395: SuccessfulCreate - Created pod: maas-api-key-cleanup-29737395-vcbnv 2026/07/16 23:15:06 23:15:00 INFO Pod/maas-api-key-cleanup-29737395-vcbnv: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/16 23:15:06 23:15:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29737395 2026/07/16 23:15:06 23:15:00 INFO Pod/maas-api-key-cleanup-29737395-vcbnv: AddedInterface - Add eth0 [10.130.0.109/23] from ovn-kubernetes 2026/07/16 23:15:06 23:14:07 INFO Pod/maas-controller-75bbfd978-jk2xw: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/16 23:15:06 23:14:07 INFO Pod/maas-controller-75bbfd978-jk2xw: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 706ms (706ms including waiting). Image size: 212866077 bytes. 2026/07/16 23:15:06 23:13:01 WARN Pod/maas-controller-75bbfd978-jk2xw: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-khqtf); attempting to pull the image may not succeed. 2026/07/16 23:15:06 23:11:12 INFO Pod/maas-controller-75bbfd978-jk2xw: Created - Created container: manager 2026/07/16 23:15:06 23:11:12 INFO Pod/maas-controller-75bbfd978-jk2xw: Started - Started container manager 2026/07/16 23:15:06 23:11:12 INFO Pod/maas-controller-75bbfd978-jk2xw: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 537ms (537ms including waiting). Image size: 212866077 bytes. 2026/07/16 23:15:06 === RESOURCE QUOTAS === 2026/07/16 23:15:06 No resource quotas found in monitored namespaces 2026/07/16 23:15:06 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/16 23:15:06 Evidence: container maas-controller-75bbfd978-jk2xw/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-75bbfd978-jk2xw_redhat-ods-applications(bd0e31ff-2c55-4635-9ce5-e03bb9b59155) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-75bbfd978-jk2xw/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-75bbfd978-jk2xw_redhat-ods-applications(bd0e31ff-2c55-4635-9ce5-e03bb9b59155)"],"confidence":"medium"} 2026/07/16 23:15:06 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/16 23:15:06 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/16 23:15:06 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/16 23:16:17 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1437.60s) === RUN TestOdhOperator 2026/07/16 22:53:10 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/16 23:17:18 TEST FAILURE DETECTED: components 2026/07/16 23:17:18 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/16 23:17:18 === CLUSTER STATE === 2026/07/16 23:17:18 Node: ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 23:17:18 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 23:17:18 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 23:17:18 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 23:17:18 Condition Ready: True - kubelet is posting ready status 2026/07/16 23:17:18 Resources: 15500m CPU, 63871020Ki memory allocatable (of 16000m CPU, 65021996Ki memory capacity) 2026/07/16 23:17:18 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 23:17:18 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 23:17:18 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 23:17:18 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 23:17:18 Condition Ready: True - kubelet is posting ready status 2026/07/16 23:17:18 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/16 23:17:18 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/16 23:17:18 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/16 23:17:18 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/16 23:17:18 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/16 23:17:18 Condition Ready: True - kubelet is posting ready status 2026/07/16 23:17:18 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/16 23:17:18 === DEPLOYMENTS === 2026/07/16 23:17:18 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/16 23:17:18 === PODS === 2026/07/16 23:17:18 Failed to collect pod data: redhat-ods-applications/maas-controller-75bbfd978-jk2xw: container manager not ready 2026/07/16 23:17:18 === OPERATOR STATUS === 2026/07/16 23:17:18 Operator deployment: 3/3 ready 2026/07/16 23:17:18 === DSCI STATUS === 2026/07/16 23:17:18 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/16 23:17:18 === DSC STATUS === 2026/07/16 23:17:18 DSC default-dsc: 2026/07/16 23:17:18 Ready: True - 2026/07/16 23:17:18 ProvisioningSucceeded: True - 2026/07/16 23:17:18 ComponentsReady: True - 2026/07/16 23:17:18 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/16 23:17:18 DashboardReady: False - Component ManagementState is set to Removed 2026/07/16 23:17:18 FeastOperatorReady: True - 2026/07/16 23:17:18 KserveLLMInferenceServiceDependencies: True - 2026/07/16 23:17:18 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/16 23:17:18 KserveReady: True - 2026/07/16 23:17:18 KueueReady: False - Component ManagementState is set to Removed 2026/07/16 23:17:18 LLM-D-WVADependencies: True - 2026/07/16 23:17:18 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 23:17:18 ModelControllerReady: True - 2026/07/16 23:17:18 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/16 23:17:18 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/16 23:17:18 OGXReady: False - Component ManagementState is set to Removed 2026/07/16 23:17:18 RayReady: False - Component ManagementState is set to Removed 2026/07/16 23:17:18 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 23:17:18 TrainerReady: True - 2026/07/16 23:17:18 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/16 23:17:18 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/16 23:17:18 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/16 23:17:18 === RECENT EVENTS === 2026/07/16 23:17:18 23:17:02 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-c5rlq_5898d75d-a853-4cb6-9393-392c8aa387db became leader 2026/07/16 23:17:18 23:17:01 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-bkxr6_94e253fb-e80c-4239-a109-ef7256e14634 became leader 2026/07/16 23:17:18 23:16:48 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bd4c5cbcb-lgwwr_50a27760-12da-4bb8-be1b-9460d9d4f898 became leader 2026/07/16 23:17:18 23:16:44 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-c5rlq: Started - Started container manager 2026/07/16 23:17:18 23:16:44 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-c5rlq: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 845ms (845ms including waiting). Image size: 236366599 bytes. 2026/07/16 23:17:18 23:16:44 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-c5rlq: Created - Created container: manager 2026/07/16 23:17:18 23:16:43 INFO Pod/kserve-controller-manager-dd9f6cf9-bkxr6: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 599ms (599ms including waiting). Image size: 238435613 bytes. 2026/07/16 23:17:18 23:16:43 INFO Pod/kserve-controller-manager-dd9f6cf9-bkxr6: Started - Started container manager 2026/07/16 23:17:18 23:16:43 INFO Pod/kserve-controller-manager-dd9f6cf9-bkxr6: Created - Created container: manager 2026/07/16 23:17:18 23:16:43 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-c5rlq: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 23:17:18 23:16:43 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-c5rlq: AddedInterface - Add eth0 [10.130.0.117/23] from ovn-kubernetes 2026/07/16 23:17:18 23:16:43 INFO Pod/kserve-controller-manager-dd9f6cf9-bkxr6: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 23:17:18 23:16:43 INFO Pod/kserve-controller-manager-dd9f6cf9-bkxr6: AddedInterface - Add eth0 [10.128.0.17/23] from ovn-kubernetes 2026/07/16 23:17:18 23:16:42 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-c5rlq: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-c5rlq to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 23:17:18 23:16:42 INFO Pod/kserve-controller-manager-dd9f6cf9-bkxr6: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-bkxr6 to ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 23:17:18 23:16:42 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/16 23:17:18 23:16:42 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/16 23:17:18 23:16:42 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-bkxr6 2026/07/16 23:17:18 23:16:42 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-c5rlq 2026/07/16 23:17:18 23:16:32 INFO Pod/model-serving-api-5679d6b4f-vtl5h: Created - Created container: server 2026/07/16 23:17:18 23:16:32 INFO Pod/model-serving-api-5679d6b4f-vtl5h: Started - Started container server 2026/07/16 23:17:18 23:16:31 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 23:17:18 23:16:31 INFO Pod/model-serving-api-5679d6b4f-vtl5h: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-vtl5h to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 23:17:18 23:16:31 INFO Pod/odh-model-controller-bd4c5cbcb-lgwwr: Started - Started container manager 2026/07/16 23:17:18 23:16:31 INFO Pod/odh-model-controller-bd4c5cbcb-lgwwr: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 803ms (803ms including waiting). Image size: 238959091 bytes. 2026/07/16 23:17:18 23:16:31 INFO Pod/model-serving-api-5679d6b4f-vtl5h: AddedInterface - Add eth0 [10.130.0.116/23] from ovn-kubernetes 2026/07/16 23:17:18 23:16:31 INFO Pod/model-serving-api-5679d6b4f-vtl5h: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 23:17:18 23:16:31 INFO Pod/odh-model-controller-bd4c5cbcb-lgwwr: Created - Created container: manager 2026/07/16 23:17:18 23:16:31 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 23:17:18 23:16:31 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-vtl5h 2026/07/16 23:17:18 23:16:30 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/16 23:17:18 23:16:30 INFO Pod/odh-model-controller-bd4c5cbcb-lgwwr: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 23:17:18 23:16:30 INFO Pod/odh-model-controller-bd4c5cbcb-lgwwr: AddedInterface - Add eth0 [10.130.0.115/23] from ovn-kubernetes 2026/07/16 23:17:18 23:16:29 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 23:17:18 23:16:29 INFO Pod/odh-model-controller-bd4c5cbcb-lgwwr: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bd4c5cbcb-lgwwr to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 23:17:18 23:16:29 WARN Pod/odh-model-controller-bd4c5cbcb-lgwwr: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/16 23:17:18 23:16:29 INFO ReplicaSet/odh-model-controller-bd4c5cbcb: SuccessfulCreate - Created pod: odh-model-controller-bd4c5cbcb-lgwwr 2026/07/16 23:17:18 23:16:29 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bd4c5cbcb from 0 to 1 2026/07/16 23:17:18 23:16:19 INFO Pod/odh-model-controller-bd4c5cbcb-npq6f: Killing - Stopping container manager 2026/07/16 23:17:18 23:16:19 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-7bc5k: Killing - Stopping container manager 2026/07/16 23:17:18 23:16:19 INFO Pod/kserve-controller-manager-dd9f6cf9-fgkn7: Killing - Stopping container manager 2026/07/16 23:17:18 23:16:19 INFO Pod/model-serving-api-5679d6b4f-p25tp: Killing - Stopping container server 2026/07/16 23:17:18 23:16:18 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 23:17:18 23:15:52 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-fgkn7_d0a83b62-6df7-4d7e-b6b2-04b2833e1adc became leader 2026/07/16 23:17:18 23:15:51 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-7bc5k_cd3c0b45-33ed-41c9-b323-955f9487f1ad became leader 2026/07/16 23:17:18 23:15:37 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bd4c5cbcb-npq6f_eab1b8ae-6d60-4994-a6c3-20fe62747a30 became leader 2026/07/16 23:17:18 23:15:35 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-7bc5k: Created - Created container: manager 2026/07/16 23:17:18 23:15:35 INFO Pod/kserve-controller-manager-dd9f6cf9-fgkn7: Started - Started container manager 2026/07/16 23:17:18 23:15:35 INFO Pod/kserve-controller-manager-dd9f6cf9-fgkn7: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 637ms (637ms including waiting). Image size: 238435613 bytes. 2026/07/16 23:17:18 23:15:35 INFO Pod/kserve-controller-manager-dd9f6cf9-fgkn7: Created - Created container: manager 2026/07/16 23:17:18 23:15:35 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-7bc5k: Started - Started container manager 2026/07/16 23:17:18 23:15:34 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-7bc5k: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/16 23:17:18 23:15:34 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-7bc5k: AddedInterface - Add eth0 [10.130.0.114/23] from ovn-kubernetes 2026/07/16 23:17:18 23:15:34 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-7bc5k: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 765ms (765ms including waiting). Image size: 236366599 bytes. 2026/07/16 23:17:18 23:15:34 INFO Pod/kserve-controller-manager-dd9f6cf9-fgkn7: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/16 23:17:18 23:15:34 INFO Pod/kserve-controller-manager-dd9f6cf9-fgkn7: AddedInterface - Add eth0 [10.128.0.16/23] from ovn-kubernetes 2026/07/16 23:17:18 23:15:33 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-7bc5k: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-7bc5k to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 23:17:18 23:15:33 INFO Pod/kserve-controller-manager-dd9f6cf9-fgkn7: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-fgkn7 to ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 23:17:18 23:15:33 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-fgkn7 2026/07/16 23:17:18 23:15:33 WARN Pod/kserve-controller-manager-dd9f6cf9-fgkn7: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/16 23:17:18 23:15:33 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/16 23:17:18 23:15:33 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/16 23:17:18 23:15:33 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-7bc5k 2026/07/16 23:17:18 23:15:27 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-b5fd84b9c-2hx97_e4958cd3-4c49-46b6-9734-0c73f4d9e4ee became leader 2026/07/16 23:17:18 23:15:21 INFO Pod/odh-model-controller-bd4c5cbcb-npq6f: Started - Started container manager 2026/07/16 23:17:18 23:15:21 INFO Pod/odh-model-controller-bd4c5cbcb-npq6f: Created - Created container: manager 2026/07/16 23:17:18 23:15:21 INFO Pod/odh-model-controller-bd4c5cbcb-npq6f: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 632ms (632ms including waiting). Image size: 238959091 bytes. 2026/07/16 23:17:18 23:15:20 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 23:17:18 23:15:20 INFO Pod/model-serving-api-5679d6b4f-p25tp: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-p25tp to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 23:17:18 23:15:20 INFO Pod/odh-model-controller-bd4c5cbcb-npq6f: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/16 23:17:18 23:15:20 INFO Pod/odh-model-controller-bd4c5cbcb-npq6f: AddedInterface - Add eth0 [10.130.0.112/23] from ovn-kubernetes 2026/07/16 23:17:18 23:15:20 INFO Pod/model-serving-api-5679d6b4f-p25tp: Started - Started container server 2026/07/16 23:17:18 23:15:20 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-p25tp 2026/07/16 23:17:18 23:15:20 INFO Pod/model-serving-api-5679d6b4f-p25tp: Created - Created container: server 2026/07/16 23:17:18 23:15:20 INFO Pod/model-serving-api-5679d6b4f-p25tp: AddedInterface - Add eth0 [10.130.0.113/23] from ovn-kubernetes 2026/07/16 23:17:18 23:15:20 INFO Pod/model-serving-api-5679d6b4f-p25tp: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/16 23:17:18 23:15:19 WARN Pod/odh-model-controller-bd4c5cbcb-npq6f: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/16 23:17:18 23:15:19 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/16 23:17:18 23:15:19 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/16 23:17:18 23:15:18 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/16 23:17:18 23:15:18 INFO Pod/odh-model-controller-bd4c5cbcb-npq6f: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bd4c5cbcb-npq6f to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 23:17:18 23:15:18 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bd4c5cbcb from 0 to 1 2026/07/16 23:17:18 23:15:18 INFO ReplicaSet/odh-model-controller-bd4c5cbcb: SuccessfulCreate - Created pod: odh-model-controller-bd4c5cbcb-npq6f 2026/07/16 23:17:18 23:15:13 WARN Pod/kserve-controller-manager-dd9f6cf9-xmhbv: Unhealthy - Readiness probe failed: Get "http://10.130.0.106:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/16 23:17:18 23:15:12 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-2hx97: Started - Started container manager 2026/07/16 23:17:18 23:15:12 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-2hx97: AddedInterface - Add eth0 [10.128.0.15/23] from ovn-kubernetes 2026/07/16 23:17:18 23:15:12 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-2hx97: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/16 23:17:18 23:15:12 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-2hx97: Created - Created container: manager 2026/07/16 23:17:18 23:15:11 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-2hx97: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-b5fd84b9c-2hx97 to ip-10-0-1-195.us-east-2.compute.internal 2026/07/16 23:17:18 23:15:11 INFO ReplicaSet/kubeflow-trainer-controller-manager-b5fd84b9c: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-b5fd84b9c-2hx97 2026/07/16 23:17:18 23:15:11 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-b5fd84b9c from 0 to 1 2026/07/16 23:17:18 23:15:08 INFO Pod/model-serving-api-5679d6b4f-5wqcr: Killing - Stopping container server 2026/07/16 23:17:18 23:15:08 INFO Pod/odh-model-controller-bd4c5cbcb-24tks: Killing - Stopping container manager 2026/07/16 23:17:18 23:15:07 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-lnlkv: Killing - Stopping container manager 2026/07/16 23:17:18 23:15:07 INFO Pod/kserve-controller-manager-dd9f6cf9-xmhbv: Killing - Stopping container manager 2026/07/16 23:17:18 23:15:04 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29737395, condition: Complete 2026/07/16 23:17:18 23:15:04 INFO Job/maas-api-key-cleanup-29737395: Completed - Job completed 2026/07/16 23:17:18 23:15:01 INFO Pod/maas-api-key-cleanup-29737395-vcbnv: Created - Created container: cleanup 2026/07/16 23:17:18 23:15:01 INFO Pod/maas-api-key-cleanup-29737395-vcbnv: Started - Started container cleanup 2026/07/16 23:17:18 23:15:00 INFO Pod/maas-api-key-cleanup-29737395-vcbnv: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29737395-vcbnv to ip-10-0-1-23.us-east-2.compute.internal 2026/07/16 23:17:18 23:15:00 INFO Pod/maas-api-key-cleanup-29737395-vcbnv: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/16 23:17:18 23:15:00 INFO Pod/maas-api-key-cleanup-29737395-vcbnv: AddedInterface - Add eth0 [10.130.0.109/23] from ovn-kubernetes 2026/07/16 23:17:18 23:15:00 INFO Job/maas-api-key-cleanup-29737395: SuccessfulCreate - Created pod: maas-api-key-cleanup-29737395-vcbnv 2026/07/16 23:17:18 23:15:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29737395 2026/07/16 23:17:18 23:14:07 INFO Pod/maas-controller-75bbfd978-jk2xw: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/16 23:17:18 23:14:07 INFO Pod/maas-controller-75bbfd978-jk2xw: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 706ms (706ms including waiting). Image size: 212866077 bytes. 2026/07/16 23:17:18 23:13:01 WARN Pod/maas-controller-75bbfd978-jk2xw: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-khqtf); attempting to pull the image may not succeed. 2026/07/16 23:17:18 === RESOURCE QUOTAS === 2026/07/16 23:17:18 No resource quotas found in monitored namespaces 2026/07/16 23:17:18 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/16 23:17:18 Evidence: container maas-controller-75bbfd978-jk2xw/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-75bbfd978-jk2xw_redhat-ods-applications(bd0e31ff-2c55-4635-9ce5-e03bb9b59155) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-75bbfd978-jk2xw/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-75bbfd978-jk2xw_redhat-ods-applications(bd0e31ff-2c55-4635-9ce5-e03bb9b59155)"],"confidence":"medium"} 2026/07/16 23:17:18 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/16 23:17:18 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/16 23:17:18 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/16 23:17:19 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1450.30s)