=== RUN TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition helper_test.go:150: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. helper_test.go:150: Cleaning up any existing Kueue test resources. helper_test.go:150: Cleaning up Kueue cluster-scoped resources helper_test.go:150: Will attempt to delete 2 Kueue resources helper_test.go:150: Attempting to delete ClusterQueue /default helper_test.go:150: Successfully processed deletion of ClusterQueue /default helper_test.go:150: Attempting to delete Kueue /cluster helper_test.go:150: Successfully processed deletion of Kueue /cluster helper_test.go:150: Creating test namespace (test-kueue-managed-d97u735b802c7b7kv9u0) with Kueue management annotation. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:150: Updating the management state of the component in the DataScienceCluster to Unmanaged (expecting NotReady initially due to missing operator). helper_test.go:150: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:150: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:150: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:150: Verifying the component in the DataScienceCluster transitions to Ready state. helper_test.go:150: Verifying Kueue configuration (cluster) is created with all expected frameworks. testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition (31.31s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:223: Timed out after 10.001s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'cluster' of kind 'Kueue' to match condition {(["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 []}. Expected <string>: map[apiVersion:kueue.openshift.io/v1 kind:Kueue metadata:map[annotations:map[platform.opendatahub.io/instance.generation:1 platform.opendatahub.io/instance.name:default-kueue platform.opendatahub.io/instance.uid:1d27cb2a-2e25-44a3-8ce2-75956162ee31 platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-09T16:45:00Z deletionGracePeriodSeconds:0 deletionTimestamp:2026-07-09T18:08:23Z finalizers:[kueue.openshift.io/finalizer] generation:2 labels:map[platform.opendatahub.io/part-of:kueue] managedFields:[map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:status:map[f:conditions:map[k:{"type":"Available"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"CertManagerAvailable"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Degraded"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Progressing"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]]] f:readyReplicas:map[]]] manager:kueue-operator operation:Apply subresource:status time:2026-07-09T16:46:07Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[.:map[] f:platform.opendatahub.io/part-of:map[]]] f:spec:map[.:map[] f:config:map[.:map[] f:integrations:map[.:map[] f:frameworks:map[.:map[] v:"Deployment":map[] v:"Pod":map[] v:"PyTorchJob":map[] v:"RayCluster":map[] v:"RayJob":map[] v:"StatefulSet":map[] v:"TrainJob":map[]]]] f:logLevel:map[] f:managementState:map[] f:operatorLogLevel:map[]]] manager:manager operation:Update time:2026-07-09T16:45:00Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"kueue.openshift.io/finalizer":map[]]]] manager:kueue-operator operation:Update time:2026-07-09T18:08:23Z]] name:cluster resourceVersion:208046152 uid:c0321245-93dd-449c-9ece-ec0c3b4c6b32] spec:map[config:map[integrations:map[frameworks:[Deployment Pod PyTorchJob RayCluster RayJob StatefulSet TrainJob]]] logLevel:Normal managementState:Managed operatorLogLevel:Normal] status:map[conditions:[map[lastTransitionTime:2026-07-09T16:45:41Z message:All 2 replicas are ready reason:AllReplicasReady status:True type:Available] map[lastTransitionTime:2026-07-09T16:45:41Z message:Deployment is up to date reason:AsExpected status:False type:Progressing] map[lastTransitionTime:2026-07-09T16:45:41Z message: reason:AsExpected status:False type:Degraded] map[lastTransitionTime:2026-07-09T16:45:25Z message:cert-manager is installed reason:CertManagerInstalled status:True type:CertManagerAvailable]] readyReplicas:2]] to match expression <string>: (["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (31.59s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:150: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:150: Disabling MaaS subcomponent (setting to Removed) helper_test.go:150: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.21s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-09T18:16:15Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.20.12"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.rosa.rh-nightly-pm.nuth.p3.openshiftapps.com, certSecret=2o8clk8b6cj832lmkd2oj78crnf7qn6d-primary-cert-bundle-secret controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.009s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "uid": <string>"c5816acb-badf-4d4f-8c39-027436c6732a", "creationTimestamp": <string>"2026-07-09T18:17:28Z", "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"431ca20c-6228-4472-8a59-bf588de68aff\"}": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-09T18:17:28Z", }, <map[string]interface {} | len:7>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"manager", "operation": <string>"Update", "subresource": <string>"status", "time": <string>"2026-07-09T18:17:53Z", }, ], "name": <string>"default-tenant", "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", "name": <string>"default", "uid": <string>"431ca20c-6228-4472-8a59-bf588de68aff", }, ], "resourceVersion": <string>"208059811", }, "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "name": <string>"maas-default-gateway", "namespace": <string>"openshift-ingress", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-09T18:17:31Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", "status": <string>"True", "type": <string>"DependenciesAvailable", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-09T18:17:31Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"True", "type": <string>"MaaSPrerequisitesAvailable", }, <map[string]interface {} | len:6>{ "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"False", "type": <string>"Degraded", "lastTransitionTime": <string>"2026-07-09T18:17:31Z", }, <map[string]interface {} | len:6>{ "status": <string>"True", "type": <string>"DeploymentsAvailable", "lastTransitionTime": <string>"2026-07-09T18:17:53Z", "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"DeploymentsReady", }, <map[string]interface {} | len:6>{ "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", "type": <string>"Ready", "lastTransitionTime": <string>"2026-07-09T18:17:53Z", }, ], "phase": <string>"Active", }, "apiVersion": <string>"maas.opendatahub.io/v1alpha1", }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (531.80s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/09 18:11:55 TEST FAILURE DETECTED: group 2 2026/07/09 18:11:55 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/09 18:11:55 === CLUSTER STATE === 2026/07/09 18:11:55 Node: ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 18:11:55 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 18:11:55 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 18:11:55 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 18:11:55 Condition Ready: True - kubelet is posting ready status 2026/07/09 18:11:55 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 18:11:55 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/07/09 18:11:55 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 18:11:55 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 18:11:55 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 18:11:55 Condition Ready: True - kubelet is posting ready status 2026/07/09 18:11:55 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 18:11:55 Node: ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:11:55 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 18:11:55 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 18:11:55 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 18:11:55 Condition Ready: True - kubelet is posting ready status 2026/07/09 18:11:55 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 18:11:55 === DEPLOYMENTS === 2026/07/09 18:11:55 Namespace: redhat-ods-applications 2026/07/09 18:11:55 Deployment maas-api: 1/1 (READY) 2026/07/09 18:11:55 Deployment maas-controller: 1/1 (READY) 2026/07/09 18:11:55 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/09 18:11:55 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/09 18:11:55 Namespace: kube-system 2026/07/09 18:11:55 === PODS === 2026/07/09 18:11:55 Namespace redhat-ods-applications: no problematic pods found 2026/07/09 18:11:55 Namespace: kube-system 2026/07/09 18:11:55 Pod ecr-credential-refresh-initial-k6gr5: Succeeded 2026/07/09 18:11:55 Container refresh: ready=false, terminated: Completed (exit 0) 2026/07/09 18:11:55 === LOGS (CURRENT) for container refresh === 2026/07/09 18:11:55 2026-04-23 02:42:29 - Discovering ECR registries from cluster configuration... 2026/07/09 18:11:55 2026-04-23 02:42:31 - No IDMS mirrors found 2026/07/09 18:11:55 2026-04-23 02:42:31 - Skipping - no ECR IDMS mirrors found (not a zero-egress cluster) 2026/07/09 18:11:55 === END LOGS === 2026/07/09 18:11:55 === OPERATOR STATUS === 2026/07/09 18:11:55 Operator deployment: 3/3 ready 2026/07/09 18:11:55 === DSCI STATUS === 2026/07/09 18:11:55 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/09 18:11:55 === DSC STATUS === 2026/07/09 18:11:55 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/09 18:11:55 === RECENT EVENTS === 2026/07/09 18:11:55 18:11:48 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-pd6ht: Killing - Stopping container manager 2026/07/09 18:11:55 18:11:48 INFO Pod/llmisvc-controller-manager-5d65df97bb-jm2rl: Killing - Stopping container manager 2026/07/09 18:11:55 18:11:48 INFO Pod/odh-model-controller-867855946f-ncr64: Killing - Stopping container manager 2026/07/09 18:11:55 18:11:48 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-fv7c8: Killing - Stopping container manager 2026/07/09 18:11:55 18:11:48 INFO Pod/model-serving-api-6688cd56bd-58rtv: Killing - Stopping container server 2026/07/09 18:11:55 18:11:47 INFO Pod/kserve-controller-manager-6c67dcb69c-6sp4b: Killing - Stopping container manager 2026/07/09 18:11:55 18:11:46 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/09 18:11:55 18:11:46 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/09 18:11:55 18:11:34 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-ncr64_b76f8b73-cfbd-4b01-a7c1-5b5d0d9c5bfa became leader 2026/07/09 18:11:55 18:11:30 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-pd6ht: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-pd6ht to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:11:55 18:11:30 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-pd6ht 2026/07/09 18:11:55 18:11:30 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-pd6ht: Started - Started container manager 2026/07/09 18:11:55 18:11:30 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-pd6ht: Created - Created container: manager 2026/07/09 18:11:55 18:11:30 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-pd6ht: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/09 18:11:55 18:11:30 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-pd6ht: AddedInterface - Add eth0 [10.130.11.52/23] from ovn-kubernetes 2026/07/09 18:11:55 18:11:30 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-pd6ht_24a6a40d-a842-4757-8e15-d329e08de60a became leader 2026/07/09 18:11:55 18:11:29 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-w6qc2: Killing - Stopping container manager 2026/07/09 18:11:55 18:11:29 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fr67d: Killing - Stopping container manager 2026/07/09 18:11:55 18:11:29 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/09 18:11:55 18:11:15 INFO Pod/odh-model-controller-867855946f-ncr64: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/09 18:11:55 18:11:15 INFO Pod/odh-model-controller-867855946f-ncr64: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 567ms (567ms including waiting). Image size: 238959091 bytes. 2026/07/09 18:11:55 18:11:15 INFO Pod/odh-model-controller-867855946f-ncr64: Created - Created container: manager 2026/07/09 18:11:55 18:11:15 INFO Pod/odh-model-controller-867855946f-ncr64: AddedInterface - Add eth0 [10.130.11.51/23] from ovn-kubernetes 2026/07/09 18:11:55 18:11:15 INFO Pod/odh-model-controller-867855946f-ncr64: Started - Started container manager 2026/07/09 18:11:55 18:11:14 INFO Pod/odh-model-controller-867855946f-ncr64: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-ncr64 to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:11:55 18:11:14 INFO Pod/odh-model-controller-867855946f-w7t9r: Killing - Stopping container manager 2026/07/09 18:11:55 18:11:14 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/09 18:11:55 18:11:14 INFO Pod/odh-model-controller-867855946f-m828t: Killing - Stopping container manager 2026/07/09 18:11:55 18:11:14 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-ncr64 2026/07/09 18:11:55 18:11:00 INFO Pod/model-serving-api-6688cd56bd-58rtv: AddedInterface - Add eth0 [10.130.11.50/23] from ovn-kubernetes 2026/07/09 18:11:55 18:11:00 INFO Pod/model-serving-api-6688cd56bd-58rtv: Started - Started container server 2026/07/09 18:11:55 18:11:00 INFO Pod/model-serving-api-6688cd56bd-58rtv: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/09 18:11:55 18:11:00 INFO Pod/model-serving-api-6688cd56bd-58rtv: Created - Created container: server 2026/07/09 18:11:55 18:10:59 INFO Pod/model-serving-api-6688cd56bd-58rtv: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-58rtv to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:11:55 18:10:59 WARN Pod/model-serving-api-6688cd56bd-nb756: Unhealthy - Readiness probe failed: Get "https://10.128.1.47:8443/readyz": dial tcp 10.128.1.47:8443: connect: connection refused 2026/07/09 18:11:55 18:10:59 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-58rtv 2026/07/09 18:11:55 18:10:59 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/09 18:11:55 18:10:59 INFO Pod/model-serving-api-6688cd56bd-vwgwv: Killing - Stopping container server 2026/07/09 18:11:55 18:10:59 INFO Pod/model-serving-api-6688cd56bd-nb756: Killing - Stopping container server 2026/07/09 18:11:55 18:10:01 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-w6qc2: Pulled - Successfully pulled image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" in 1.887s (1.887s including waiting). Image size: 195240093 bytes. 2026/07/09 18:11:55 18:10:01 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-w6qc2: Created - Created container: manager 2026/07/09 18:11:55 18:10:01 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-w6qc2: Started - Started container manager 2026/07/09 18:11:55 18:09:59 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-w6qc2: Pulling - Pulling image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" 2026/07/09 18:11:55 18:09:59 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-w6qc2: AddedInterface - Add eth0 [10.128.1.53/23] from ovn-kubernetes 2026/07/09 18:11:55 18:09:58 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-w6qc2: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-w6qc2 to ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 18:11:55 18:09:58 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 1 to 2 2026/07/09 18:11:55 18:09:58 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-w6qc2 2026/07/09 18:11:55 18:09:39 INFO Pod/odh-model-controller-867855946f-w7t9r: AddedInterface - Add eth0 [10.130.11.49/23] from ovn-kubernetes 2026/07/09 18:11:55 18:09:39 INFO Pod/odh-model-controller-867855946f-w7t9r: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 626ms (626ms including waiting). Image size: 238959091 bytes. 2026/07/09 18:11:55 18:09:39 INFO Pod/odh-model-controller-867855946f-w7t9r: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/09 18:11:55 18:09:39 INFO Pod/odh-model-controller-867855946f-w7t9r: Started - Started container manager 2026/07/09 18:11:55 18:09:39 INFO Pod/odh-model-controller-867855946f-w7t9r: Created - Created container: manager 2026/07/09 18:11:55 18:09:38 INFO Pod/odh-model-controller-867855946f-w7t9r: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-w7t9r to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:11:55 18:09:38 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 1 to 2 2026/07/09 18:11:55 18:09:38 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-w7t9r 2026/07/09 18:11:55 18:09:19 INFO Pod/model-serving-api-6688cd56bd-vwgwv: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/09 18:11:55 18:09:19 INFO Pod/model-serving-api-6688cd56bd-vwgwv: Started - Started container server 2026/07/09 18:11:55 18:09:19 INFO Pod/model-serving-api-6688cd56bd-vwgwv: Created - Created container: server 2026/07/09 18:11:55 18:09:18 INFO Pod/model-serving-api-6688cd56bd-vwgwv: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-vwgwv to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:11:55 18:09:18 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-vwgwv 2026/07/09 18:11:55 18:09:18 INFO Pod/model-serving-api-6688cd56bd-vwgwv: AddedInterface - Add eth0 [10.130.11.48/23] from ovn-kubernetes 2026/07/09 18:11:55 18:09:18 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 1 to 2 2026/07/09 18:11:55 18:09:10 INFO Pod/maas-api-84449bdd5d-8gqbm: Started - Started container maas-api 2026/07/09 18:11:55 18:09:10 INFO Pod/maas-api-84449bdd5d-8gqbm: Created - Created container: maas-api 2026/07/09 18:11:55 18:09:10 INFO Pod/maas-api-84449bdd5d-8gqbm: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 2.297s (2.297s including waiting). Image size: 175018396 bytes. 2026/07/09 18:11:55 18:09:08 INFO Pod/maas-api-84449bdd5d-8gqbm: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/09 18:11:55 18:09:08 INFO Pod/maas-api-84449bdd5d-8gqbm: AddedInterface - Add eth0 [10.128.1.51/23] from ovn-kubernetes 2026/07/09 18:11:55 18:09:07 INFO Pod/maas-api-84449bdd5d-8gqbm: Scheduled - Successfully assigned redhat-ods-applications/maas-api-84449bdd5d-8gqbm to ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 18:11:55 18:09:07 INFO ReplicaSet/maas-api-84449bdd5d: SuccessfulCreate - Created pod: maas-api-84449bdd5d-8gqbm 2026/07/09 18:11:55 18:09:07 WARN ReplicaSet/maas-api-84449bdd5d: FailedCreate - Error creating: pods "maas-api-84449bdd5d-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/09 18:11:55 18:09:07 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-84449bdd5d from 0 to 1 2026/07/09 18:11:55 18:09:04 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-6sp4b_08901aac-7b5d-462e-8fbd-1a03bef7b553 became leader 2026/07/09 18:11:55 18:09:03 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-jm2rl_56c2ca48-4ef8-4b14-b9d2-c0bd16a23d44 became leader 2026/07/09 18:11:55 18:08:47 INFO Pod/kserve-controller-manager-6c67dcb69c-6sp4b: Created - Created container: manager 2026/07/09 18:11:55 18:08:47 INFO Pod/kserve-controller-manager-6c67dcb69c-6sp4b: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 3.1s (3.1s including waiting). Image size: 238435613 bytes. 2026/07/09 18:11:55 18:08:47 INFO Pod/kserve-controller-manager-6c67dcb69c-6sp4b: Started - Started container manager 2026/07/09 18:11:55 18:08:45 INFO Pod/llmisvc-controller-manager-5d65df97bb-jm2rl: Started - Started container manager 2026/07/09 18:11:55 18:08:45 INFO Pod/llmisvc-controller-manager-5d65df97bb-jm2rl: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 1.806s (1.806s including waiting). Image size: 236366599 bytes. 2026/07/09 18:11:55 18:08:45 INFO Pod/llmisvc-controller-manager-5d65df97bb-jm2rl: Created - Created container: manager 2026/07/09 18:11:55 18:08:44 INFO Pod/kserve-controller-manager-6c67dcb69c-6sp4b: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/09 18:11:55 18:08:44 INFO Pod/kserve-controller-manager-6c67dcb69c-6sp4b: AddedInterface - Add eth0 [10.130.11.44/23] from ovn-kubernetes 2026/07/09 18:11:55 18:08:43 INFO Pod/llmisvc-controller-manager-5d65df97bb-jm2rl: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/09 18:11:55 18:08:43 INFO Pod/llmisvc-controller-manager-5d65df97bb-jm2rl: AddedInterface - Add eth0 [10.130.11.45/23] from ovn-kubernetes 2026/07/09 18:11:55 18:08:43 WARN Pod/llmisvc-controller-manager-5d65df97bb-jm2rl: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/09 18:11:55 18:08:43 WARN Pod/kserve-controller-manager-6c67dcb69c-6sp4b: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/09 18:11:55 18:08:42 INFO Pod/llmisvc-controller-manager-5d65df97bb-jm2rl: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-jm2rl to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:11:55 18:08:42 INFO Pod/kserve-controller-manager-6c67dcb69c-6sp4b: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-6sp4b to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:11:55 18:08:42 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/09 18:11:55 18:08:42 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-6sp4b 2026/07/09 18:11:55 18:08:42 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-jm2rl 2026/07/09 18:11:55 18:08:42 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/09 18:11:55 18:08:37 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fr67d: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/09 18:11:55 18:08:37 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fr67d: AddedInterface - Add eth0 [10.130.11.42/23] from ovn-kubernetes 2026/07/09 18:11:55 18:08:37 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fr67d: Started - Started container manager 2026/07/09 18:11:55 18:08:37 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fr67d: Created - Created container: manager 2026/07/09 18:11:55 18:08:37 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-5bd688895b-fr67d_ebd5b48a-1f17-4ddc-9f4f-e1d95fe93b2f became leader 2026/07/09 18:11:55 18:08:36 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85bdcf9988-9tjm7_099da152-d413-467b-a88b-4407a39c8697 became leader 2026/07/09 18:11:55 18:08:36 WARN Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fr67d: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 2026/07/09 18:11:55 18:08:36 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/workload-variant-autoscaler-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: bearerTokenSecret: [REDACTED] to get secret "workload-variant-autoscaler-controller-manager-token": secrets "workload-variant-autoscaler-controller-manager-token" not found 2026/07/09 18:11:55 18:08:35 INFO Pod/workload-variant-autoscaler-controller-manager-5bd688895b-fr67d: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-5bd688895b-fr67d to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:11:55 18:08:35 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: FailedCreate - Error creating: pods "workload-variant-autoscaler-controller-manager-5bd688895b-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 2026/07/09 18:11:55 18:08:35 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-66d8c86b58-fv7c8_c1dab641-78c0-49b8-ab1e-2a66e6167782 became leader 2026/07/09 18:11:55 18:08:35 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-5bd688895b from 0 to 1 2026/07/09 18:11:55 18:08:35 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-5bd688895b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-5bd688895b-fr67d 2026/07/09 18:11:55 18:08:31 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-867855946f-m828t_1f3b13a8-945b-4a3a-88b8-cdfb4eb3130c became leader 2026/07/09 18:11:55 18:08:18 INFO Pod/maas-controller-85bdcf9988-9tjm7: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 2.896s (2.896s including waiting). Image size: 212866077 bytes. 2026/07/09 18:11:55 18:08:18 INFO Pod/maas-controller-85bdcf9988-9tjm7: Started - Started container manager 2026/07/09 18:11:55 18:08:18 INFO Pod/maas-controller-85bdcf9988-9tjm7: Created - Created container: manager 2026/07/09 18:11:55 18:08:17 INFO Pod/model-serving-api-6688cd56bd-nb756: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-nb756 to ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 18:11:55 18:08:17 INFO Pod/model-serving-api-6688cd56bd-nb756: AddedInterface - Add eth0 [10.128.1.47/23] from ovn-kubernetes 2026/07/09 18:11:55 18:08:17 INFO Pod/model-serving-api-6688cd56bd-nb756: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/09 18:11:55 18:08:17 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-fv7c8: Created - Created container: manager 2026/07/09 18:11:55 18:08:17 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-fv7c8: Started - Started container manager 2026/07/09 18:11:55 18:08:17 INFO Pod/model-serving-api-6688cd56bd-nb756: Started - Started container server 2026/07/09 18:11:55 18:08:17 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-fv7c8: AddedInterface - Add eth0 [10.128.1.46/23] from ovn-kubernetes 2026/07/09 18:11:55 18:08:17 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-nb756 2026/07/09 18:11:55 18:08:17 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-fv7c8: Pulled - Container image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" already present on machine 2026/07/09 18:11:55 18:08:17 INFO Pod/model-serving-api-6688cd56bd-nb756: Created - Created container: server 2026/07/09 18:11:55 18:08:16 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/09 18:11:55 18:08:16 WARN Pod/model-registry-operator-controller-manager-66d8c86b58-fv7c8: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/07/09 18:11:55 18:08:16 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/09 18:11:55 18:08:16 INFO Pod/odh-model-controller-867855946f-m828t: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 550ms (550ms including waiting). Image size: 238959091 bytes. 2026/07/09 18:11:55 18:08:16 INFO Pod/odh-model-controller-867855946f-m828t: Created - Created container: manager 2026/07/09 18:11:55 18:08:16 INFO Pod/odh-model-controller-867855946f-m828t: Started - Started container manager 2026/07/09 18:11:55 18:08:15 INFO Pod/model-registry-operator-controller-manager-66d8c86b58-fv7c8: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-66d8c86b58-fv7c8 to ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 18:11:55 18:08:15 INFO Pod/maas-controller-85bdcf9988-9tjm7: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85bdcf9988-9tjm7 to ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 18:11:55 18:08:15 INFO ReplicaSet/model-registry-operator-controller-manager-66d8c86b58: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-66d8c86b58-fv7c8 2026/07/09 18:11:55 18:08:15 INFO Pod/odh-model-controller-867855946f-m828t: AddedInterface - Add eth0 [10.128.1.44/23] from ovn-kubernetes 2026/07/09 18:11:55 18:08:15 INFO Pod/maas-controller-85bdcf9988-9tjm7: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/09 18:11:55 18:08:15 INFO ReplicaSet/maas-controller-85bdcf9988: SuccessfulCreate - Created pod: maas-controller-85bdcf9988-9tjm7 2026/07/09 18:11:55 18:08:15 INFO Pod/maas-controller-85bdcf9988-9tjm7: AddedInterface - Add eth0 [10.128.1.45/23] from ovn-kubernetes 2026/07/09 18:11:55 18:08:15 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-66d8c86b58 from 0 to 1 2026/07/09 18:11:55 18:08:15 INFO Pod/odh-model-controller-867855946f-m828t: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/09 18:11:55 18:08:14 INFO Pod/odh-model-controller-867855946f-m828t: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-867855946f-m828t to ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 18:11:55 18:08:14 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-867855946f from 0 to 1 2026/07/09 18:11:55 18:08:14 INFO ReplicaSet/odh-model-controller-867855946f: SuccessfulCreate - Created pod: odh-model-controller-867855946f-m828t 2026/07/09 18:11:55 18:08:14 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85bdcf9988 from 0 to 1 2026/07/09 18:11:55 18:08:14 WARN Pod/odh-model-controller-867855946f-m828t: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/09 18:11:55 18:08:03 INFO Pod/spark-operator-webhook-9cb9bdf44-f8n4j: Killing - Stopping container webhook 2026/07/09 18:11:55 18:08:03 INFO Pod/spark-operator-controller-767899d44f-j595s: Killing - Stopping container controller 2026/07/09 18:11:55 18:07:49 INFO Lease/spark-operator-controller-lock: LeaderElection - spark-operator-controller-767899d44f-j595s_92fb5300-8cbf-4d95-beb1-859dcfa0a7c2 became leader 2026/07/09 18:11:55 18:07:47 INFO Pod/spark-operator-webhook-9cb9bdf44-f8n4j: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-webhook-9cb9bdf44-f8n4j to ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 18:11:55 18:07:47 INFO Pod/spark-operator-webhook-9cb9bdf44-xjwwm: Killing - Stopping container webhook 2026/07/09 18:11:55 18:07:47 INFO Pod/spark-operator-webhook-9cb9bdf44-f8n4j: AddedInterface - Add eth0 [10.128.1.43/23] from ovn-kubernetes 2026/07/09 18:11:55 18:07:47 INFO Pod/spark-operator-webhook-9cb9bdf44-f8n4j: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/09 18:11:55 18:07:47 INFO Pod/spark-operator-webhook-9cb9bdf44-f8n4j: Created - Created container: webhook 2026/07/09 18:11:55 18:07:47 INFO Pod/spark-operator-webhook-9cb9bdf44-f8n4j: Started - Started container webhook 2026/07/09 18:11:55 18:07:47 INFO Pod/spark-operator-webhook-9cb9bdf44-lx5lf: Killing - Stopping container webhook 2026/07/09 18:11:55 18:07:47 INFO ReplicaSet/spark-operator-webhook-9cb9bdf44: SuccessfulCreate - Created pod: spark-operator-webhook-9cb9bdf44-f8n4j 2026/07/09 18:11:55 18:07:47 INFO Deployment/spark-operator-webhook: ScalingReplicaSet - Scaled up replica set spark-operator-webhook-9cb9bdf44 from 0 to 1 2026/07/09 18:11:55 18:07:32 INFO Pod/spark-operator-controller-767899d44f-j595s: Scheduled - Successfully assigned redhat-ods-applications/spark-operator-controller-767899d44f-j595s to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:11:55 18:07:32 INFO Pod/spark-operator-controller-767899d44f-j595s: AddedInterface - Add eth0 [10.130.11.41/23] from ovn-kubernetes 2026/07/09 18:11:55 18:07:32 INFO Pod/spark-operator-controller-767899d44f-j595s: Pulled - Container image "quay.io/rhoai/odh-spark-operator-rhel9@sha256:df9ca97a588bcf27f65f6c8698a5288e5568bf8115f77987f5575b90b6b7cac5" already present on machine 2026/07/09 18:11:55 18:07:32 INFO Pod/spark-operator-controller-767899d44f-tc28s: Killing - Stopping container controller 2026/07/09 18:11:55 18:07:32 INFO ReplicaSet/spark-operator-controller-767899d44f: SuccessfulCreate - Created pod: spark-operator-controller-767899d44f-j595s 2026/07/09 18:11:55 18:07:32 INFO Deployment/spark-operator-controller: ScalingReplicaSet - Scaled up replica set spark-operator-controller-767899d44f from 0 to 1 2026/07/09 18:11:55 18:07:32 INFO Pod/spark-operator-controller-767899d44f-j595s: Started - Started container controller 2026/07/09 18:11:55 18:07:32 INFO Pod/spark-operator-controller-767899d44f-j595s: Created - Created container: controller 2026/07/09 18:11:55 18:07:32 INFO Pod/spark-operator-controller-767899d44f-t4xls: Killing - Stopping container controller 2026/07/09 18:11:55 18:07:21 INFO SparkApplication/scheduled-spark-pi-d97u64tb802c7b7kv9tg-1783620401000670715: SparkDriverCompleted - Driver scheduled-spark-pi-d97u64tb802c7b7kv9tg-1783620401000670715-driver completed 2026/07/09 18:11:55 18:07:19 INFO SparkApplication/scheduled-spark-pi-d97u64tb802c7b7kv9tg-1783620401000670715: SparkExecutorCompleted - Executor spark-pi-03f0bc9f480fc9c2-exec-1 completed 2026/07/09 18:11:55 18:07:19 INFO Pod/spark-pi-03f0bc9f480fc9c2-exec-1: Killing - Stopping container spark-kubernetes-executor 2026/07/09 18:11:55 18:07:06 INFO SparkApplication/scheduled-spark-pi-d97u64tb802c7b7kv9tg-1783620401000670715: SparkExecutorRunning - Executor spark-pi-03f0bc9f480fc9c2-exec-1 is running 2026/07/09 18:11:55 18:07:05 INFO Pod/spark-pi-03f0bc9f480fc9c2-exec-1: Started - Started container spark-kubernetes-executor 2026/07/09 18:11:55 18:07:05 INFO Pod/spark-pi-03f0bc9f480fc9c2-exec-1: Created - Created container: spark-kubernetes-executor 2026/07/09 18:11:55 18:07:05 INFO Pod/spark-pi-03f0bc9f480fc9c2-exec-1: AddedInterface - Add eth0 [10.130.11.40/23] from ovn-kubernetes 2026/07/09 18:11:55 18:07:05 INFO Pod/spark-pi-03f0bc9f480fc9c2-exec-1: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/09 18:11:55 18:07:04 INFO Pod/spark-pi-03f0bc9f480fc9c2-exec-1: Scheduled - Successfully assigned redhat-ods-applications/spark-pi-03f0bc9f480fc9c2-exec-1 to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:11:55 18:07:04 INFO SparkApplication/scheduled-spark-pi-d97u64tb802c7b7kv9tg-1783620401000670715: SparkExecutorPending - Executor spark-pi-03f0bc9f480fc9c2-exec-1 is pending 2026/07/09 18:11:55 18:06:56 INFO Pod/scheduled-spark-pi-d97u64tb802c7b7kv9tg-1783620401000670715-driver: AddedInterface - Add eth0 [10.128.1.42/23] from ovn-kubernetes 2026/07/09 18:11:55 18:06:56 INFO Pod/scheduled-spark-pi-d97u64tb802c7b7kv9tg-1783620401000670715-driver: Pulled - Container image "quay.io/opendatahub/data-processing:Spark-v4.0.1" already present on machine 2026/07/09 18:11:55 18:06:56 INFO Pod/scheduled-spark-pi-d97u64tb802c7b7kv9tg-1783620401000670715-driver: Created - Created container: spark-kubernetes-driver 2026/07/09 18:11:55 18:06:56 INFO Pod/scheduled-spark-pi-d97u64tb802c7b7kv9tg-1783620401000670715-driver: Started - Started container spark-kubernetes-driver 2026/07/09 18:11:55 18:06:56 INFO SparkApplication/scheduled-spark-pi-d97u64tb802c7b7kv9tg-1783620401000670715: SparkDriverRunning - Driver scheduled-spark-pi-d97u64tb802c7b7kv9tg-1783620401000670715-driver is running 2026/07/09 18:11:55 === RESOURCE QUOTAS === 2026/07/09 18:11:55 No resource quotas found in monitored namespaces 2026/07/09 18:11:55 Classification for group 2: infrastructure/storage (code=1006, confidence=medium) 2026/07/09 18:11:55 Evidence: event Pod/llmisvc-controller-manager-5d65df97bb-jm2rl: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/llmisvc-controller-manager-5d65df97bb-jm2rl: FailedMount - MountVolume.SetUp failed for volume \"cert\" : secret \"llmisvc-webhook-server-cert\" not found"],"confidence":"medium"} 2026/07/09 18:11:55 Diagnostics complete for failed test: group 2 controller_test.go:753: Stopping: group 2 test failed. 2026/07/09 18:11:55 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/09 18:11:55 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/09 18:16:15 Circuit breaker: success recorded, resetting failure counter (was 1) 2026/07/09 18:25:06 TEST FAILURE DETECTED: group 4 2026/07/09 18:25:06 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/09 18:25:07 === CLUSTER STATE === 2026/07/09 18:25:07 Node: ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 18:25:07 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 18:25:07 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 18:25:07 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 18:25:07 Condition Ready: True - kubelet is posting ready status 2026/07/09 18:25:07 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 18:25:07 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/07/09 18:25:07 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 18:25:07 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 18:25:07 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 18:25:07 Condition Ready: True - kubelet is posting ready status 2026/07/09 18:25:07 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 18:25:07 Node: ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:25:07 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 18:25:07 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 18:25:07 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 18:25:07 Condition Ready: True - kubelet is posting ready status 2026/07/09 18:25:07 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 18:25:07 === DEPLOYMENTS === 2026/07/09 18:25:07 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/09 18:25:07 === PODS === 2026/07/09 18:25:07 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-tpgc2: container manager not ready 2026/07/09 18:25:07 === OPERATOR STATUS === 2026/07/09 18:25:07 Operator deployment: 3/3 ready 2026/07/09 18:25:07 === DSCI STATUS === 2026/07/09 18:25:07 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/09 18:25:07 === DSC STATUS === 2026/07/09 18:25:07 DSC default-dsc: 2026/07/09 18:25:07 Ready: True - 2026/07/09 18:25:07 ProvisioningSucceeded: True - 2026/07/09 18:25:07 ComponentsReady: True - 2026/07/09 18:25:07 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/09 18:25:07 DashboardReady: False - Component ManagementState is set to Removed 2026/07/09 18:25:07 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 18:25:07 KserveLLMInferenceServiceDependencies: True - 2026/07/09 18:25:07 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/09 18:25:07 KserveReady: True - 2026/07/09 18:25:07 KueueReady: True - 2026/07/09 18:25:07 LLM-D-WVADependencies: True - 2026/07/09 18:25:07 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 18:25:07 ModelControllerReady: True - 2026/07/09 18:25:07 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/09 18:25:07 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/09 18:25:07 OGXReady: False - Component ManagementState is set to Removed 2026/07/09 18:25:07 RayReady: False - Component ManagementState is set to Removed 2026/07/09 18:25:07 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 18:25:07 TrainerReady: False - Component ManagementState is set to Removed 2026/07/09 18:25:07 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 18:25:07 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/09 18:25:07 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/09 18:25:07 === RECENT EVENTS === 2026/07/09 18:25:07 18:23:39 INFO Pod/maas-controller-85bdcf9988-tpgc2: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/09 18:25:07 18:23:39 INFO Pod/maas-controller-85bdcf9988-tpgc2: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 511ms (511ms including waiting). Image size: 212866077 bytes. 2026/07/09 18:25:07 18:23:16 WARN Pod/maas-controller-85bdcf9988-tpgc2: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-tcj5c); attempting to pull the image may not succeed. 2026/07/09 18:25:07 18:20:54 INFO Pod/maas-controller-85bdcf9988-tpgc2: Created - Created container: manager 2026/07/09 18:25:07 18:20:54 INFO Pod/maas-controller-85bdcf9988-tpgc2: Started - Started container manager 2026/07/09 18:25:07 18:20:54 INFO Pod/maas-controller-85bdcf9988-tpgc2: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 675ms (675ms including waiting). Image size: 212866077 bytes. 2026/07/09 18:25:07 === RESOURCE QUOTAS === 2026/07/09 18:25:07 No resource quotas found in monitored namespaces 2026/07/09 18:25:07 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/09 18:25:07 Evidence: container maas-controller-85bdcf9988-tpgc2/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-tpgc2_redhat-ods-applications(647e0e5b-10b2-410f-978b-2ca725bf6f87) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-tpgc2/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-tpgc2_redhat-ods-applications(647e0e5b-10b2-410f-978b-2ca725bf6f87)"],"confidence":"medium"} 2026/07/09 18:25:07 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/09 18:25:07 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/09 18:25:07 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/09 18:26:18 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1503.89s) === RUN TestOdhOperator 2026/07/09 18:02:04 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/09 18:27:19 TEST FAILURE DETECTED: components 2026/07/09 18:27:19 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/09 18:27:19 === CLUSTER STATE === 2026/07/09 18:27:19 Node: ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 18:27:19 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 18:27:19 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 18:27:19 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 18:27:19 Condition Ready: True - kubelet is posting ready status 2026/07/09 18:27:19 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 18:27:19 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/07/09 18:27:19 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 18:27:19 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 18:27:19 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 18:27:19 Condition Ready: True - kubelet is posting ready status 2026/07/09 18:27:19 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 18:27:19 Node: ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:27:19 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 18:27:19 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 18:27:19 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 18:27:19 Condition Ready: True - kubelet is posting ready status 2026/07/09 18:27:19 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/07/09 18:27:19 === DEPLOYMENTS === 2026/07/09 18:27:19 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/09 18:27:19 === PODS === 2026/07/09 18:27:19 Failed to collect pod data: redhat-ods-applications/maas-controller-85bdcf9988-tpgc2: container manager not ready 2026/07/09 18:27:19 === OPERATOR STATUS === 2026/07/09 18:27:19 Operator deployment: 3/3 ready 2026/07/09 18:27:19 === DSCI STATUS === 2026/07/09 18:27:19 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/09 18:27:19 === DSC STATUS === 2026/07/09 18:27:19 DSC default-dsc: 2026/07/09 18:27:19 Ready: True - 2026/07/09 18:27:19 ProvisioningSucceeded: True - 2026/07/09 18:27:19 ComponentsReady: True - 2026/07/09 18:27:19 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/09 18:27:19 DashboardReady: False - Component ManagementState is set to Removed 2026/07/09 18:27:19 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 18:27:19 KserveLLMInferenceServiceDependencies: True - 2026/07/09 18:27:19 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/09 18:27:19 KserveReady: True - 2026/07/09 18:27:19 KueueReady: True - 2026/07/09 18:27:19 LLM-D-WVADependencies: True - 2026/07/09 18:27:19 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 18:27:19 ModelControllerReady: True - 2026/07/09 18:27:19 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/09 18:27:19 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/09 18:27:19 OGXReady: False - Component ManagementState is set to Removed 2026/07/09 18:27:19 RayReady: False - Component ManagementState is set to Removed 2026/07/09 18:27:19 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 18:27:19 TrainerReady: True - 2026/07/09 18:27:19 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 18:27:19 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/09 18:27:19 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/09 18:27:19 === RECENT EVENTS === 2026/07/09 18:27:19 18:27:03 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-scgnw_7288e5d0-9a5b-4473-afc5-9f0a869c3a9d became leader 2026/07/09 18:27:19 18:27:02 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-lgtr7_53dcb6d2-e683-460b-bc1e-ac0ce435f72f became leader 2026/07/09 18:27:19 18:26:48 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-fmgkw_bb414f22-c246-483a-a6ec-9f4fee50b211 became leader 2026/07/09 18:27:19 18:26:45 INFO Pod/llmisvc-controller-manager-5d65df97bb-scgnw: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/09 18:27:19 18:26:45 INFO Pod/llmisvc-controller-manager-5d65df97bb-scgnw: Created - Created container: manager 2026/07/09 18:27:19 18:26:45 INFO Pod/kserve-controller-manager-6c67dcb69c-lgtr7: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 560ms (560ms including waiting). Image size: 238435613 bytes. 2026/07/09 18:27:19 18:26:45 INFO Pod/kserve-controller-manager-6c67dcb69c-lgtr7: Created - Created container: manager 2026/07/09 18:27:19 18:26:45 INFO Pod/kserve-controller-manager-6c67dcb69c-lgtr7: Started - Started container manager 2026/07/09 18:27:19 18:26:45 INFO Pod/llmisvc-controller-manager-5d65df97bb-scgnw: AddedInterface - Add eth0 [10.130.11.77/23] from ovn-kubernetes 2026/07/09 18:27:19 18:26:45 INFO Pod/llmisvc-controller-manager-5d65df97bb-scgnw: Started - Started container manager 2026/07/09 18:27:19 18:26:45 INFO Pod/kserve-controller-manager-6c67dcb69c-lgtr7: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/09 18:27:19 18:26:45 INFO Pod/llmisvc-controller-manager-5d65df97bb-scgnw: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 625ms (625ms including waiting). Image size: 236366599 bytes. 2026/07/09 18:27:19 18:26:44 INFO Pod/llmisvc-controller-manager-5d65df97bb-scgnw: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-scgnw to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:27:19 18:26:44 INFO Pod/kserve-controller-manager-6c67dcb69c-lgtr7: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-lgtr7 to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:27:19 18:26:44 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/09 18:27:19 18:26:44 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-scgnw 2026/07/09 18:27:19 18:26:44 INFO Pod/kserve-controller-manager-6c67dcb69c-lgtr7: AddedInterface - Add eth0 [10.130.11.76/23] from ovn-kubernetes 2026/07/09 18:27:19 18:26:44 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-lgtr7 2026/07/09 18:27:19 18:26:44 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/09 18:27:19 18:26:32 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/09 18:27:19 18:26:32 INFO Pod/model-serving-api-6688cd56bd-cwm7p: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/09 18:27:19 18:26:32 INFO Pod/model-serving-api-6688cd56bd-cwm7p: AddedInterface - Add eth0 [10.130.11.75/23] from ovn-kubernetes 2026/07/09 18:27:19 18:26:32 INFO Pod/model-serving-api-6688cd56bd-cwm7p: Created - Created container: server 2026/07/09 18:27:19 18:26:32 INFO Pod/model-serving-api-6688cd56bd-cwm7p: Started - Started container server 2026/07/09 18:27:19 18:26:31 INFO Pod/model-serving-api-6688cd56bd-cwm7p: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-cwm7p to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:27:19 18:26:31 INFO Pod/odh-model-controller-6849fc49d8-fmgkw: Created - Created container: manager 2026/07/09 18:27:19 18:26:31 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/09 18:27:19 18:26:31 INFO Pod/odh-model-controller-6849fc49d8-fmgkw: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 451ms (451ms including waiting). Image size: 238959091 bytes. 2026/07/09 18:27:19 18:26:31 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-cwm7p 2026/07/09 18:27:19 18:26:31 INFO Pod/odh-model-controller-6849fc49d8-fmgkw: AddedInterface - Add eth0 [10.130.11.74/23] from ovn-kubernetes 2026/07/09 18:27:19 18:26:31 INFO Pod/odh-model-controller-6849fc49d8-fmgkw: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/09 18:27:19 18:26:31 INFO Pod/odh-model-controller-6849fc49d8-fmgkw: Started - Started container manager 2026/07/09 18:27:19 18:26:31 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/09 18:27:19 18:26:30 INFO Pod/odh-model-controller-6849fc49d8-fmgkw: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-fmgkw to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:27:19 18:26:30 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/09 18:27:19 18:26:30 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-fmgkw 2026/07/09 18:27:19 18:26:20 INFO Pod/odh-model-controller-6849fc49d8-9dll6: Killing - Stopping container manager 2026/07/09 18:27:19 18:26:20 INFO Pod/llmisvc-controller-manager-5d65df97bb-lqr6x: Killing - Stopping container manager 2026/07/09 18:27:19 18:26:20 INFO Pod/kserve-controller-manager-6c67dcb69c-n2z6k: Killing - Stopping container manager 2026/07/09 18:27:19 18:26:20 INFO Pod/model-serving-api-6688cd56bd-vdbwx: Killing - Stopping container server 2026/07/09 18:27:19 18:26:20 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/09 18:27:19 18:25:52 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c67dcb69c-n2z6k_94e3f645-fbcd-46d4-84e9-97525d34e82b became leader 2026/07/09 18:27:19 18:25:52 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-5d65df97bb-lqr6x_0aa54fbc-2452-4362-ba2c-6aad0efce2f7 became leader 2026/07/09 18:27:19 18:25:36 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-6849fc49d8-9dll6_4e52edfe-7f8e-4d8e-89c6-8c5776f0ad67 became leader 2026/07/09 18:27:19 18:25:35 INFO Pod/kserve-controller-manager-6c67dcb69c-n2z6k: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 533ms (533ms including waiting). Image size: 238435613 bytes. 2026/07/09 18:27:19 18:25:35 INFO Pod/kserve-controller-manager-6c67dcb69c-n2z6k: Created - Created container: manager 2026/07/09 18:27:19 18:25:35 INFO Pod/kserve-controller-manager-6c67dcb69c-n2z6k: Started - Started container manager 2026/07/09 18:27:19 18:25:35 INFO Pod/llmisvc-controller-manager-5d65df97bb-lqr6x: Started - Started container manager 2026/07/09 18:27:19 18:25:35 INFO Pod/llmisvc-controller-manager-5d65df97bb-lqr6x: Created - Created container: manager 2026/07/09 18:27:19 18:25:35 INFO Pod/llmisvc-controller-manager-5d65df97bb-lqr6x: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 631ms (631ms including waiting). Image size: 236366599 bytes. 2026/07/09 18:27:19 18:25:34 INFO Pod/llmisvc-controller-manager-5d65df97bb-lqr6x: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-5d65df97bb-lqr6x to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:27:19 18:25:34 INFO Pod/llmisvc-controller-manager-5d65df97bb-lqr6x: AddedInterface - Add eth0 [10.130.11.73/23] from ovn-kubernetes 2026/07/09 18:27:19 18:25:34 INFO Pod/llmisvc-controller-manager-5d65df97bb-lqr6x: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/09 18:27:19 18:25:34 INFO ReplicaSet/llmisvc-controller-manager-5d65df97bb: SuccessfulCreate - Created pod: llmisvc-controller-manager-5d65df97bb-lqr6x 2026/07/09 18:27:19 18:25:34 INFO Pod/kserve-controller-manager-6c67dcb69c-n2z6k: AddedInterface - Add eth0 [10.130.11.72/23] from ovn-kubernetes 2026/07/09 18:27:19 18:25:34 INFO Pod/kserve-controller-manager-6c67dcb69c-n2z6k: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/09 18:27:19 18:25:33 INFO Pod/kserve-controller-manager-6c67dcb69c-n2z6k: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c67dcb69c-n2z6k to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:27:19 18:25:33 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c67dcb69c from 0 to 1 2026/07/09 18:27:19 18:25:33 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-5d65df97bb from 0 to 1 2026/07/09 18:27:19 18:25:33 INFO ReplicaSet/kserve-controller-manager-6c67dcb69c: SuccessfulCreate - Created pod: kserve-controller-manager-6c67dcb69c-n2z6k 2026/07/09 18:27:19 18:25:28 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7ccfdd66-5z7z2_e7272be0-9842-4ef0-a554-f478945424cd became leader 2026/07/09 18:27:19 18:25:21 INFO Pod/model-serving-api-6688cd56bd-vdbwx: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/09 18:27:19 18:25:21 INFO Pod/model-serving-api-6688cd56bd-vdbwx: Created - Created container: server 2026/07/09 18:27:19 18:25:21 INFO Pod/model-serving-api-6688cd56bd-vdbwx: Started - Started container server 2026/07/09 18:27:19 18:25:21 INFO Pod/model-serving-api-6688cd56bd-vdbwx: AddedInterface - Add eth0 [10.130.11.71/23] from ovn-kubernetes 2026/07/09 18:27:19 18:25:20 INFO Pod/model-serving-api-6688cd56bd-vdbwx: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-6688cd56bd-vdbwx to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:27:19 18:25:20 INFO Pod/odh-model-controller-6849fc49d8-9dll6: Started - Started container manager 2026/07/09 18:27:19 18:25:20 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-6688cd56bd from 0 to 1 2026/07/09 18:27:19 18:25:20 INFO Pod/odh-model-controller-6849fc49d8-9dll6: Created - Created container: manager 2026/07/09 18:27:19 18:25:20 INFO ReplicaSet/model-serving-api-6688cd56bd: SuccessfulCreate - Created pod: model-serving-api-6688cd56bd-vdbwx 2026/07/09 18:27:19 18:25:20 INFO Pod/odh-model-controller-6849fc49d8-9dll6: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 700ms (700ms including waiting). Image size: 238959091 bytes. 2026/07/09 18:27:19 18:25:20 WARN ReplicaSet/model-serving-api-6688cd56bd: FailedCreate - Error creating: pods "model-serving-api-6688cd56bd-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/09 18:27:19 18:25:20 INFO Pod/odh-model-controller-6849fc49d8-9dll6: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/09 18:27:19 18:25:20 INFO Pod/odh-model-controller-6849fc49d8-9dll6: AddedInterface - Add eth0 [10.130.11.70/23] from ovn-kubernetes 2026/07/09 18:27:19 18:25:19 WARN Pod/odh-model-controller-6849fc49d8-9dll6: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/09 18:27:19 18:25:18 INFO Pod/odh-model-controller-6849fc49d8-9dll6: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-6849fc49d8-9dll6 to ip-10-0-1-39.us-east-2.compute.internal 2026/07/09 18:27:19 18:25:18 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-6849fc49d8 from 0 to 1 2026/07/09 18:27:19 18:25:18 INFO ReplicaSet/odh-model-controller-6849fc49d8: SuccessfulCreate - Created pod: odh-model-controller-6849fc49d8-9dll6 2026/07/09 18:27:19 18:25:14 WARN Pod/kserve-controller-manager-6c67dcb69c-kctzc: Unhealthy - Readiness probe failed: Get "http://10.130.11.67:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026/07/09 18:27:19 18:25:12 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-5z7z2: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7ccfdd66-5z7z2 to ip-10-0-1-10.us-east-2.compute.internal 2026/07/09 18:27:19 18:25:12 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-5z7z2: Created - Created container: manager 2026/07/09 18:27:19 18:25:12 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-5z7z2: AddedInterface - Add eth0 [10.128.1.60/23] from ovn-kubernetes 2026/07/09 18:27:19 18:25:12 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-5z7z2: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/09 18:27:19 18:25:12 INFO Pod/kubeflow-trainer-controller-manager-7ccfdd66-5z7z2: Started - Started container manager 2026/07/09 18:27:19 18:25:12 INFO ReplicaSet/kubeflow-trainer-controller-manager-7ccfdd66: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7ccfdd66-5z7z2 2026/07/09 18:27:19 18:25:11 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7ccfdd66 from 0 to 1 2026/07/09 18:27:19 18:25:09 INFO Pod/model-serving-api-6688cd56bd-fp5mq: Killing - Stopping container server 2026/07/09 18:27:19 18:25:09 INFO Pod/llmisvc-controller-manager-5d65df97bb-xzs9k: Killing - Stopping container manager 2026/07/09 18:27:19 18:25:09 INFO Pod/odh-model-controller-6849fc49d8-7m8zf: Killing - Stopping container manager 2026/07/09 18:27:19 18:25:09 INFO Pod/kserve-controller-manager-6c67dcb69c-kctzc: Killing - Stopping container manager 2026/07/09 18:27:19 18:25:09 WARN Pod/odh-model-controller-6849fc49d8-7m8zf: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/09 18:27:19 18:25:09 WARN Pod/odh-model-controller-6849fc49d8-7m8zf: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (odh-model-controller-dockercfg-4bnrw); attempting to pull the image may not succeed. 2026/07/09 18:27:19 18:25:08 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/09 18:27:19 18:23:39 INFO Pod/maas-controller-85bdcf9988-tpgc2: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/09 18:27:19 18:23:39 INFO Pod/maas-controller-85bdcf9988-tpgc2: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 511ms (511ms including waiting). Image size: 212866077 bytes. 2026/07/09 18:27:19 18:23:16 WARN Pod/maas-controller-85bdcf9988-tpgc2: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-tcj5c); attempting to pull the image may not succeed. 2026/07/09 18:27:19 === RESOURCE QUOTAS === 2026/07/09 18:27:19 No resource quotas found in monitored namespaces 2026/07/09 18:27:19 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/09 18:27:19 Evidence: container maas-controller-85bdcf9988-tpgc2/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-tpgc2_redhat-ods-applications(647e0e5b-10b2-410f-978b-2ca725bf6f87) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-85bdcf9988-tpgc2/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-85bdcf9988-tpgc2_redhat-ods-applications(647e0e5b-10b2-410f-978b-2ca725bf6f87)"],"confidence":"medium"} 2026/07/09 18:27:19 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/09 18:27:19 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/09 18:27:19 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/09 18:27:21 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1517.12s)